Nov 02 02:46:59.173328 ip-10-0-133-163 systemd[1]: Starting Kubernetes Kubelet...
Nov 02 02:46:59.560256 ip-10-0-133-163 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 02:46:59.560256 ip-10-0-133-163 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 02 02:46:59.560256 ip-10-0-133-163 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 02:46:59.560256 ip-10-0-133-163 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 02:46:59.563434 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.563347    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 02:46:59.563434 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.563427    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 02:46:59.563434 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.563434    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 02:46:59.563434 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.563438    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 02:46:59.563675 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.563443    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 02:46:59.563675 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.563447    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 02:46:59.563675 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.563451    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 02:46:59.563675 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.563454    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 02:46:59.563675 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.563459    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 02:46:59.563675 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.563463    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 02:46:59.563675 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.563467    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 02:46:59.563675 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.563470    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 02:46:59.563675 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.563474    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 02:46:59.563675 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.563478    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 02:46:59.563675 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.563482    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 02:46:59.563675 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.563485    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 02:46:59.563675 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.563489    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 02:46:59.563675 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.563493    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 02:46:59.563675 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.563497    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 02:46:59.563675 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.563500    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 02:46:59.563675 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.563505    2577 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 02:46:59.563675 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.563509    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 02:46:59.563675 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.563513    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 02:46:59.564462 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.563517    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 02:46:59.564462 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.563521    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 02:46:59.564462 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.563525    2577 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 02:46:59.564462 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.563530    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 02:46:59.564462 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.563535    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 02:46:59.564462 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.563541    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 02:46:59.564462 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.563547    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 02:46:59.564462 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.563552    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 02:46:59.564462 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.563557    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 02:46:59.564462 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.563561    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 02:46:59.564462 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.563566    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 02:46:59.564462 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.563571    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 02:46:59.564462 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.563575    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 02:46:59.564462 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.563579    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 02:46:59.564462 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.563584    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 02:46:59.564462 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.563589    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 02:46:59.564462 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.563594    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 02:46:59.564462 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.563598    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 02:46:59.564462 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.563603    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 02:46:59.564462 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.563607    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 02:46:59.565288 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.563612    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 02:46:59.565288 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.563616    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 02:46:59.565288 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.563620    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 02:46:59.565288 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.563625    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 02:46:59.565288 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.563629    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 02:46:59.565288 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.563633    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 02:46:59.565288 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.563638    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 02:46:59.565288 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.563643    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 02:46:59.565288 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.563647    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 02:46:59.565288 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.563652    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 02:46:59.565288 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.563656    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 02:46:59.565288 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.563660    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 02:46:59.565288 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.563665    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 02:46:59.565288 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.563669    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 02:46:59.565288 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.563673    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 02:46:59.565288 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.563677    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 02:46:59.565288 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.563682    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 02:46:59.565288 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.563686    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 02:46:59.565288 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.563690    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 02:46:59.566118 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.563694    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 02:46:59.566118 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.563698    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 02:46:59.566118 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.563702    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 02:46:59.566118 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.563707    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 02:46:59.566118 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.563711    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 02:46:59.566118 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.563715    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 02:46:59.566118 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.563719    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 02:46:59.566118 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.563724    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 02:46:59.566118 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.563728    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 02:46:59.566118 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.563733    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 02:46:59.566118 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.563737    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 02:46:59.566118 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.563741    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 02:46:59.566118 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.563745    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 02:46:59.566118 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.563750    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 02:46:59.566118 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.563754    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 02:46:59.566118 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.563758    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 02:46:59.566118 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.563762    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 02:46:59.566118 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.563767    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 02:46:59.566118 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.563771    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 02:46:59.566916 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.563775    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 02:46:59.566916 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.563779    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 02:46:59.566916 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.563784    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 02:46:59.566916 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.563791    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 02:46:59.566916 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.563798    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 02:46:59.566916 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.563804    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 02:46:59.566916 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.563809    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 02:46:59.566916 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.563813    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 02:46:59.566916 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.563817    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 02:46:59.566916 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.563822    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 02:46:59.566916 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.563826    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 02:46:59.566916 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.563830    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 02:46:59.566916 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.563835    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 02:46:59.566916 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.563839    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 02:46:59.566916 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.563843    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 02:46:59.566916 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.563848    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 02:46:59.566916 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.563852    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 02:46:59.566916 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.563856    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 02:46:59.566916 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.563860    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 02:46:59.566916 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.563866    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 02:46:59.567777 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.563875    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 02:46:59.567777 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.563880    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 02:46:59.567777 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.563884    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 02:46:59.567777 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.563888    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 02:46:59.567777 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.563892    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 02:46:59.567777 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.565224    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 02:46:59.567777 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.565242    2577 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 02:46:59.567777 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.565248    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 02:46:59.567777 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.565253    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 02:46:59.567777 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.565258    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 02:46:59.567777 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.565263    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 02:46:59.567777 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.565267    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 02:46:59.567777 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.565271    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 02:46:59.567777 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.565275    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 02:46:59.567777 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.565280    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 02:46:59.567777 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.565284    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 02:46:59.567777 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.565289    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 02:46:59.567777 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.565293    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 02:46:59.567777 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.565299    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 02:46:59.567777 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.565304    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 02:46:59.568658 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.565308    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 02:46:59.568658 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.565313    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 02:46:59.568658 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.565317    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 02:46:59.568658 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.565321    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 02:46:59.568658 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.565325    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 02:46:59.568658 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.565330    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 02:46:59.568658 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.565334    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 02:46:59.568658 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.565338    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 02:46:59.568658 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.565343    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 02:46:59.568658 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.565347    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 02:46:59.568658 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.565351    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 02:46:59.568658 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.565355    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 02:46:59.568658 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.565361    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 02:46:59.568658 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.565366    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 02:46:59.568658 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.565370    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 02:46:59.568658 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.565374    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 02:46:59.568658 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.565378    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 02:46:59.568658 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.565382    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 02:46:59.568658 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.565386    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 02:46:59.569489 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.565391    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 02:46:59.569489 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.565395    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 02:46:59.569489 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.565399    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 02:46:59.569489 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.565403    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 02:46:59.569489 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.565407    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 02:46:59.569489 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.565411    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 02:46:59.569489 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.565415    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 02:46:59.569489 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.565420    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 02:46:59.569489 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.565424    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 02:46:59.569489 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.565429    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 02:46:59.569489 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.565433    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 02:46:59.569489 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.565437    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 02:46:59.569489 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.565442    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 02:46:59.569489 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.565447    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 02:46:59.569489 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.565451    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 02:46:59.569489 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.565455    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 02:46:59.569489 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.565459    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 02:46:59.569489 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.565463    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 02:46:59.569489 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.565467    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 02:46:59.570033 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.565471    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 02:46:59.570033 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.565478    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 02:46:59.570033 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.565484    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 02:46:59.570033 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.565489    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 02:46:59.570033 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.565494    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 02:46:59.570033 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.565499    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 02:46:59.570033 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.565504    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 02:46:59.570033 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.565508    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 02:46:59.570033 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.565513    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 02:46:59.570033 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.565517    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 02:46:59.570033 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.565521    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 02:46:59.570033 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.565528    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 02:46:59.570033 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.565533    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 02:46:59.570033 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.565537    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 02:46:59.570033 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.565541    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 02:46:59.570033 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.565546    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 02:46:59.570033 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.565550    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 02:46:59.570033 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.565555    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 02:46:59.570033 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.565559    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 02:46:59.570626 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.565563    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 02:46:59.570626 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.565567    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 02:46:59.570626 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.565571    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 02:46:59.570626 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.565576    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 02:46:59.570626 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.565581    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 02:46:59.570626 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.565585    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 02:46:59.570626 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.565589    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 02:46:59.570626 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.565594    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 02:46:59.570626 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.565598    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 02:46:59.570626 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.565601    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 02:46:59.570626 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.565605    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 02:46:59.570626 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.565609    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 02:46:59.570626 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.565613    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 02:46:59.570626 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.565617    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 02:46:59.570626 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.565621    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 02:46:59.570626 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.565626    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 02:46:59.570626 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.565631    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 02:46:59.570626 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.565635    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 02:46:59.570626 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.565641    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 02:46:59.571097 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.565646    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 02:46:59.571097 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.565650    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 02:46:59.571097 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.565655    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 02:46:59.571097 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.565659    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 02:46:59.571097 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.565664    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 02:46:59.571097 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.565668    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 02:46:59.571097 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.565672    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 02:46:59.571097 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.565676    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 02:46:59.571097 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.565681    2577 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 02:46:59.571097 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.565685    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 02:46:59.571097 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.565689    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 02:46:59.571097 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.565693    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 02:46:59.571097 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.565697    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 02:46:59.571097 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.565701    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 02:46:59.571097 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.565705    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 02:46:59.571097 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.566534    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 02:46:59.571097 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.566544    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 02:46:59.571097 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.566550    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 02:46:59.571097 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.566555    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 02:46:59.571657 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.566559    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 02:46:59.571657 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.566563    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 02:46:59.571657 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.566568    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 02:46:59.571657 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.566572    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 02:46:59.571657 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.566576    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 02:46:59.571657 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.566581    2577 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 02:46:59.571657 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.566585    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 02:46:59.571657 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.566590    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 02:46:59.571657 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.566594    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 02:46:59.571657 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.566599    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 02:46:59.571657 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.566604    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 02:46:59.571657 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.566608    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 02:46:59.571657 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.566612    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 02:46:59.571657 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.566616    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 02:46:59.571657 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.566620    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 02:46:59.571657 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.566625    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 02:46:59.571657 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.566629    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 02:46:59.571657 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.566633    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 02:46:59.571657 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.566637    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 02:46:59.571657 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.566641    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 02:46:59.572148 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.566645    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 02:46:59.572148 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.566650    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 02:46:59.572148 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.566653    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 02:46:59.572148 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.566657    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 02:46:59.572148 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.566662    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 02:46:59.572148 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.566666    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 02:46:59.572148 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.566670    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 02:46:59.572148 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.566675    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 02:46:59.572148 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.566679    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 02:46:59.572148 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.566683    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 02:46:59.572148 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.566687    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 02:46:59.572148 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.566692    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 02:46:59.572148 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.566696    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 02:46:59.572148 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.566700    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 02:46:59.572148 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.566704    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 02:46:59.572148 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.566708    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 02:46:59.572148 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.566712    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 02:46:59.572148 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.566717    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 02:46:59.572148 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.566721    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 02:46:59.572148 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.566727    2577 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 02:46:59.572622 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.566732    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 02:46:59.572622 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.566736    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 02:46:59.572622 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.566740    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 02:46:59.572622 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.566744    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 02:46:59.572622 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.566749    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 02:46:59.572622 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.566753    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 02:46:59.572622 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.566757    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 02:46:59.572622 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.566761    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 02:46:59.572622 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.566766    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 02:46:59.572622 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.566770    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 02:46:59.572622 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.566774    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 02:46:59.572622 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.566779    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 02:46:59.572622 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.566783    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 02:46:59.572622 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.566787    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 02:46:59.572622 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.566791    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 02:46:59.572622 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.566795    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 02:46:59.572622 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.566800    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 02:46:59.572622 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.566804    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 02:46:59.572622 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.566808    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 02:46:59.572622 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.566813    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 02:46:59.573138 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.566817    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 02:46:59.573138 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.566821    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 02:46:59.573138 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.566826    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 02:46:59.573138 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.566830    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 02:46:59.573138 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.566835    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 02:46:59.573138 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.566840    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 02:46:59.573138 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.566844    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 02:46:59.573138 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.566848    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 02:46:59.573138 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.566853    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 02:46:59.573138 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.566858    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 02:46:59.573138 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.566862    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 02:46:59.573138 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.566868    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 02:46:59.573138 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.566872    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 02:46:59.573138 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.566876    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 02:46:59.573138 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.566881    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 02:46:59.573138 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.566886    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 02:46:59.573138 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.566890    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 02:46:59.573138 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.566894    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 02:46:59.573138 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.566898    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 02:46:59.573138 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.566903    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 02:46:59.573612 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.566907    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 02:46:59.573612 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.566913    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 02:46:59.573612 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.566920    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 02:46:59.573612 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.566924    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 02:46:59.573612 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.566929    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 02:46:59.573612 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.566932    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 02:46:59.573612 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.566937    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 02:46:59.573612 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.566941    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 02:46:59.573612 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.566945    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 02:46:59.573612 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.566950    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 02:46:59.573612 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.566956    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 02:46:59.573612 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.566965    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 02:46:59.573612 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.566970    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 02:46:59.573612 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.566974    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 02:46:59.573612 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.566978    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 02:46:59.573612 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.566982    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 02:46:59.573612 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.566986    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 02:46:59.573612 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.566991    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 02:46:59.574090 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.566996    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 02:46:59.574090 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.567000    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 02:46:59.574090 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.567004    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 02:46:59.574090 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.567008    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 02:46:59.574090 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567156    2577 flags.go:64] FLAG: --address="0.0.0.0"
Nov 02 02:46:59.574090 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567175    2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 02 02:46:59.574090 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567186    2577 flags.go:64] FLAG: --anonymous-auth="true"
Nov 02 02:46:59.574090 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567195    2577 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 02 02:46:59.574090 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567202    2577 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 02 02:46:59.574090 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567207    2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 02 02:46:59.574090 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567213    2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 02 02:46:59.574090 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567221    2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 02 02:46:59.574090 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567226    2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 02 02:46:59.574090 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567231    2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 02 02:46:59.574090 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567237    2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 02 02:46:59.574090 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567242    2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 02 02:46:59.574090 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567247    2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 02 02:46:59.574090 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567252    2577 flags.go:64] FLAG: --cgroup-root=""
Nov 02 02:46:59.574090 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567257    2577 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 02 02:46:59.574090 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567262    2577 flags.go:64] FLAG: --client-ca-file=""
Nov 02 02:46:59.574090 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567267    2577 flags.go:64] FLAG: --cloud-provider="external"
Nov 02 02:46:59.574090 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567272    2577 flags.go:64] FLAG: --cluster-dns="[]"
Nov 02 02:46:59.574090 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567278    2577 flags.go:64] FLAG: --cluster-domain=""
Nov 02 02:46:59.574090 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567282    2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 02 02:46:59.574671 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567288    2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 02 02:46:59.574671 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567292    2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 02 02:46:59.574671 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567298    2577 flags.go:64] FLAG: --container-log-max-files="5"
Nov 02 02:46:59.574671 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567304    2577 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 02 02:46:59.574671 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567310    2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 02 02:46:59.574671 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567314    2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 02 02:46:59.574671 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567320    2577 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 02 02:46:59.574671 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567325    2577 flags.go:64] FLAG: --contention-profiling="false"
Nov 02 02:46:59.574671 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567330    2577 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 02 02:46:59.574671 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567336    2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 02 02:46:59.574671 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567341    2577 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 02 02:46:59.574671 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567346    2577 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 02 02:46:59.574671 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567352    2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 02 02:46:59.574671 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567359    2577 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 02 02:46:59.574671 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567364    2577 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 02 02:46:59.574671 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567370    2577 flags.go:64] FLAG: --enable-load-reader="false"
Nov 02 02:46:59.574671 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567375    2577 flags.go:64] FLAG: --enable-server="true"
Nov 02 02:46:59.574671 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567380    2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 02 02:46:59.574671 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567388    2577 flags.go:64] FLAG: --event-burst="100"
Nov 02 02:46:59.574671 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567393    2577 flags.go:64] FLAG: --event-qps="50"
Nov 02 02:46:59.574671 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567398    2577 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 02 02:46:59.574671 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567403    2577 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 02 02:46:59.574671 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567407    2577 flags.go:64] FLAG: --eviction-hard=""
Nov 02 02:46:59.574671 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567414    2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 02 02:46:59.575325 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567419    2577 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 02 02:46:59.575325 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567425    2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 02 02:46:59.575325 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567430    2577 flags.go:64] FLAG: --eviction-soft=""
Nov 02 02:46:59.575325 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567435    2577 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 02 02:46:59.575325 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567439    2577 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 02 02:46:59.575325 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567444    2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 02 02:46:59.575325 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567449    2577 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 02 02:46:59.575325 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567453    2577 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 02 02:46:59.575325 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567458    2577 flags.go:64] FLAG: --fail-swap-on="true"
Nov 02 02:46:59.575325 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567463    2577 flags.go:64] FLAG: --feature-gates=""
Nov 02 02:46:59.575325 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567470    2577 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 02 02:46:59.575325 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567475    2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 02 02:46:59.575325 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567480    2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 02 02:46:59.575325 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567485    2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 02 02:46:59.575325 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567490    2577 flags.go:64] FLAG: --healthz-port="10248"
Nov 02 02:46:59.575325 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567495    2577 flags.go:64] FLAG: --help="false"
Nov 02 02:46:59.575325 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567499    2577 flags.go:64] FLAG: --hostname-override="ip-10-0-133-163.us-west-1.compute.internal"
Nov 02 02:46:59.575325 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567507    2577 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 02 02:46:59.575325 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567512    2577 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 02 02:46:59.575325 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567517    2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 02 02:46:59.575325 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567523    2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 02 02:46:59.575325 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567530    2577 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 02 02:46:59.575325 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567535    2577 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 02 02:46:59.575864 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567541    2577 flags.go:64] FLAG: --image-service-endpoint=""
Nov 02 02:46:59.575864 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567546    2577 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 02 02:46:59.575864 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567550    2577 flags.go:64] FLAG: --kube-api-burst="100"
Nov 02 02:46:59.575864 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567555    2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 02 02:46:59.575864 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567560    2577 flags.go:64] FLAG: --kube-api-qps="50"
Nov 02 02:46:59.575864 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567565    2577 flags.go:64] FLAG: --kube-reserved=""
Nov 02 02:46:59.575864 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567570    2577 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 02 02:46:59.575864 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567574    2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 02 02:46:59.575864 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567579    2577 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 02 02:46:59.575864 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567584    2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 02 02:46:59.575864 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567589    2577 flags.go:64] FLAG: --lock-file=""
Nov 02 02:46:59.575864 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567594    2577 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 02 02:46:59.575864 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567599    2577 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 02 02:46:59.575864 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567604    2577 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 02 02:46:59.575864 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567618    2577 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 02 02:46:59.575864 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567623    2577 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 02 02:46:59.575864 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567628    2577 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 02 02:46:59.575864 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567633    2577 flags.go:64] FLAG: --logging-format="text"
Nov 02 02:46:59.575864 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567638    2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 02 02:46:59.575864 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567643    2577 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 02 02:46:59.575864 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567648    2577 flags.go:64] FLAG: --manifest-url=""
Nov 02 02:46:59.575864 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567653    2577 flags.go:64] FLAG: --manifest-url-header=""
Nov 02 02:46:59.575864 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567659    2577 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 02 02:46:59.575864 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567664    2577 flags.go:64] FLAG: --max-open-files="1000000"
Nov 02 02:46:59.575864 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567671    2577 flags.go:64] FLAG: --max-pods="110"
Nov 02 02:46:59.576659 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567676    2577 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 02 02:46:59.576659 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567681    2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 02 02:46:59.576659 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567686    2577 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 02 02:46:59.576659 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567691    2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 02 02:46:59.576659 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567696    2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 02 02:46:59.576659 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567705    2577 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 02 02:46:59.576659 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567710    2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 02 02:46:59.576659 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567722    2577 flags.go:64] FLAG: --node-status-max-images="50"
Nov 02 02:46:59.576659 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567727    2577 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 02 02:46:59.576659 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567732    2577 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 02 02:46:59.576659 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567737    2577 flags.go:64] FLAG: --pod-cidr=""
Nov 02 02:46:59.576659 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567741    2577 flags.go:64] FLAG: --pod-manifest-path=""
Nov 02 02:46:59.576659 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567746    2577 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 02 02:46:59.576659 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567751    2577 flags.go:64] FLAG: --pods-per-core="0"
Nov 02 02:46:59.576659 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567756    2577 flags.go:64] FLAG: --port="10250"
Nov 02 02:46:59.576659 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567761    2577 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 02 02:46:59.576659 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567765    2577 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0b4f9600c3b715dbd"
Nov 02 02:46:59.576659 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567770    2577 flags.go:64] FLAG: --qos-reserved=""
Nov 02 02:46:59.576659 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567775    2577 flags.go:64] FLAG: --read-only-port="10255"
Nov 02 02:46:59.576659 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567779    2577 flags.go:64] FLAG: --register-node="true"
Nov 02 02:46:59.576659 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567784    2577 flags.go:64] FLAG: --register-with-taints=""
Nov 02 02:46:59.576659 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567790    2577 flags.go:64] FLAG: --registry-burst="10"
Nov 02 02:46:59.576659 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567794    2577 flags.go:64] FLAG: --registry-qps="5"
Nov 02 02:46:59.576659 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567798    2577 flags.go:64] FLAG: --reserved-cpus=""
Nov 02 02:46:59.576659 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567803    2577 flags.go:64] FLAG: --reserved-memory=""
Nov 02 02:46:59.576659 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567809    2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 02 02:46:59.577578 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567814    2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 02 02:46:59.577578 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567819    2577 flags.go:64] FLAG: --rotate-certificates="false"
Nov 02 02:46:59.577578 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567824    2577 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 02 02:46:59.577578 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567829    2577 flags.go:64] FLAG: --runonce="false"
Nov 02 02:46:59.577578 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567834    2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 02 02:46:59.577578 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567839    2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 02 02:46:59.577578 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567844    2577 flags.go:64] FLAG: --seccomp-default="false"
Nov 02 02:46:59.577578 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567849    2577 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 02 02:46:59.577578 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567854    2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 02 02:46:59.577578 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567860    2577 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 02 02:46:59.577578 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567865    2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 02 02:46:59.577578 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567872    2577 flags.go:64] FLAG: --storage-driver-password="root"
Nov 02 02:46:59.577578 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567876    2577 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 02 02:46:59.577578 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567882    2577 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 02 02:46:59.577578 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567887    2577 flags.go:64] FLAG: --storage-driver-user="root"
Nov 02 02:46:59.577578 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567892    2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 02 02:46:59.577578 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567897    2577 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 02 02:46:59.577578 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567902    2577 flags.go:64] FLAG: --system-cgroups=""
Nov 02 02:46:59.577578 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567907    2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 02 02:46:59.577578 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567915    2577 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 02 02:46:59.577578 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567920    2577 flags.go:64] FLAG: --tls-cert-file=""
Nov 02 02:46:59.577578 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567924    2577 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 02 02:46:59.577578 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567930    2577 flags.go:64] FLAG: --tls-min-version=""
Nov 02 02:46:59.577578 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567935    2577 flags.go:64] FLAG: --tls-private-key-file=""
Nov 02 02:46:59.577578 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567939    2577 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 02 02:46:59.578239 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567944    2577 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 02 02:46:59.578239 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567949    2577 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 02 02:46:59.578239 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567953    2577 flags.go:64] FLAG: --v="2"
Nov 02 02:46:59.578239 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567960    2577 flags.go:64] FLAG: --version="false"
Nov 02 02:46:59.578239 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567967    2577 flags.go:64] FLAG: --vmodule=""
Nov 02 02:46:59.578239 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567973    2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 02 02:46:59.578239 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.567979    2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 02 02:46:59.578239 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568236    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 02:46:59.578239 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568246    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 02:46:59.578239 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568251    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 02:46:59.578239 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568257    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 02:46:59.578239 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568262    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 02:46:59.578239 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568268    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 02:46:59.578239 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568273    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 02:46:59.578239 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568278    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 02:46:59.578239 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568282    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 02:46:59.578239 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568286    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 02:46:59.578239 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568292    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 02:46:59.578239 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568298    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 02:46:59.578239 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568302    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 02:46:59.578239 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568308    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 02:46:59.578729 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568313    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 02:46:59.578729 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568317    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 02:46:59.578729 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568321    2577 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 02:46:59.578729 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568326    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 02:46:59.578729 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568330    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 02:46:59.578729 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568335    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 02:46:59.578729 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568339    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 02:46:59.578729 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568343    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 02:46:59.578729 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568347    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 02:46:59.578729 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568352    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 02:46:59.578729 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568356    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 02:46:59.578729 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568361    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 02:46:59.578729 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568365    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 02:46:59.578729 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568370    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 02:46:59.578729 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568374    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 02:46:59.578729 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568378    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 02:46:59.578729 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568382    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 02:46:59.578729 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568387    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 02:46:59.578729 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568391    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 02:46:59.578729 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568397    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 02:46:59.579227 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568402    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 02:46:59.579227 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568408    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 02:46:59.579227 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568413    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 02:46:59.579227 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568420    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 02:46:59.579227 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568424    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 02:46:59.579227 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568429    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 02:46:59.579227 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568433    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 02:46:59.579227 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568437    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 02:46:59.579227 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568441    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 02:46:59.579227 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568452    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 02:46:59.579227 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568456    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 02:46:59.579227 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568462    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 02:46:59.579227 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568467    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 02:46:59.579227 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568471    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 02:46:59.579227 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568475    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 02:46:59.579227 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568480    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 02:46:59.579227 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568484    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 02:46:59.579227 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568488    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 02:46:59.579227 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568493    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 02:46:59.579659 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568497    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 02:46:59.579659 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568501    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 02:46:59.579659 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568506    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 02:46:59.579659 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568510    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 02:46:59.579659 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568514    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 02:46:59.579659 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568518    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 02:46:59.579659 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568522    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 02:46:59.579659 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568527    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 02:46:59.579659 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568531    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 02:46:59.579659 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568535    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 02:46:59.579659 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568540    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 02:46:59.579659 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568544    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 02:46:59.579659 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568548    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 02:46:59.579659 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568553    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 02:46:59.579659 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568557    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 02:46:59.579659 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568561    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 02:46:59.579659 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568565    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 02:46:59.579659 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568569    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 02:46:59.579659 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568573    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 02:46:59.580130 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568577    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 02:46:59.580130 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568582    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 02:46:59.580130 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568586    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 02:46:59.580130 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568592    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 02:46:59.580130 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568598    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 02:46:59.580130 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568604    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 02:46:59.580130 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568608    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 02:46:59.580130 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568612    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 02:46:59.580130 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568616    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 02:46:59.580130 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568620    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 02:46:59.580130 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568624    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 02:46:59.580130 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568629    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 02:46:59.580130 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568633    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 02:46:59.580130 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568638    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 02:46:59.580130 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568642    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 02:46:59.580130 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568646    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 02:46:59.580130 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568650    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 02:46:59.580130 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568654    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 02:46:59.580130 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568659    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 02:46:59.580598 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568663    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 02:46:59.580598 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568666    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 02:46:59.580598 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568671    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 02:46:59.580598 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568675    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 02:46:59.580598 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568679    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 02:46:59.580598 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568683    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 02:46:59.580598 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568687    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 02:46:59.580598 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568691    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 02:46:59.580598 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568695    2577 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 02:46:59.580598 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568700    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 02:46:59.580598 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568704    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 02:46:59.580598 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568708    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 02:46:59.580598 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568712    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 02:46:59.580598 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568717    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 02:46:59.580598 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568721    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 02:46:59.580598 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.568762    2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 02:46:59.580972 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568817    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 02:46:59.580972 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568826    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 02:46:59.580972 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568833    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 02:46:59.580972 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568838    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 02:46:59.580972 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568843    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 02:46:59.580972 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568847    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 02:46:59.580972 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568852    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 02:46:59.580972 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568856    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 02:46:59.580972 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568860    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 02:46:59.580972 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568865    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 02:46:59.580972 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568869    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 02:46:59.580972 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568873    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 02:46:59.580972 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568878    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 02:46:59.580972 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568882    2577 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 02:46:59.580972 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568886    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 02:46:59.580972 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568890    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 02:46:59.580972 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568894    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 02:46:59.580972 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568899    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 02:46:59.580972 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568903    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 02:46:59.580972 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568908    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 02:46:59.581581 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568914    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 02:46:59.581581 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568921    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 02:46:59.581581 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568927    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 02:46:59.581581 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568931    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 02:46:59.581581 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568937    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 02:46:59.581581 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568942    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 02:46:59.581581 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568948    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 02:46:59.581581 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568952    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 02:46:59.581581 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568956    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 02:46:59.581581 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568961    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 02:46:59.581581 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568965    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 02:46:59.581581 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568969    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 02:46:59.581581 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568973    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 02:46:59.581581 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568979    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 02:46:59.581581 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568985    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 02:46:59.581581 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568990    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 02:46:59.581581 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568994    2577 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 02:46:59.581581 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.568999    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 02:46:59.581581 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.569003    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 02:46:59.582195 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.569007    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 02:46:59.582195 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.569026    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 02:46:59.582195 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.569031    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 02:46:59.582195 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.569035    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 02:46:59.582195 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.569040    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 02:46:59.582195 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.569044    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 02:46:59.582195 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.569048    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 02:46:59.582195 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.569052    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 02:46:59.582195 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.569056    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 02:46:59.582195 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.569061    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 02:46:59.582195 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.569065    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 02:46:59.582195 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.569069    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 02:46:59.582195 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.569074    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 02:46:59.582195 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.569078    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 02:46:59.582195 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.569084    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 02:46:59.582195 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.569090    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 02:46:59.582195 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.569095    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 02:46:59.582195 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.569101    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 02:46:59.582195 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.569106    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 02:46:59.582195 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.569110    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 02:46:59.582685 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.569114    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 02:46:59.582685 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.569119    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 02:46:59.582685 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.569123    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 02:46:59.582685 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.569127    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 02:46:59.582685 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.569133    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 02:46:59.582685 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.569137    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 02:46:59.582685 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.569144    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 02:46:59.582685 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.569151    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 02:46:59.582685 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.569155    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 02:46:59.582685 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.569160    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 02:46:59.582685 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.569164    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 02:46:59.582685 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.569168    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 02:46:59.582685 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.569172    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 02:46:59.582685 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.569177    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 02:46:59.582685 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.569181    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 02:46:59.582685 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.569185    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 02:46:59.582685 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.569190    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 02:46:59.582685 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.569194    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 02:46:59.582685 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.569198    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 02:46:59.583154 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.569202    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 02:46:59.583154 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.569207    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 02:46:59.583154 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.569211    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 02:46:59.583154 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.569215    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 02:46:59.583154 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.569219    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 02:46:59.583154 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.569223    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 02:46:59.583154 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.569228    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 02:46:59.583154 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.569238    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 02:46:59.583154 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.569242    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 02:46:59.583154 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.569247    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 02:46:59.583154 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.569251    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 02:46:59.583154 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.569255    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 02:46:59.583154 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.569259    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 02:46:59.583154 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.569264    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 02:46:59.583154 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.569268    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 02:46:59.583154 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.569272    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 02:46:59.583154 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.569277    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 02:46:59.583154 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.569281    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 02:46:59.583154 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.569285    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 02:46:59.583618 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.569291    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 02:46:59.583618 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.569297    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 02:46:59.583618 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.569301    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 02:46:59.583618 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.569306    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 02:46:59.583618 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.569311    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 02:46:59.583618 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.569316    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 02:46:59.583618 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.569320    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 02:46:59.583618 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.569326    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 02:46:59.583618 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.569330    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 02:46:59.583618 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.575638    2577 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 02 02:46:59.583618 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.575656    2577 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 02 02:46:59.583618 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.575669    2577 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 02 02:46:59.583618 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.575674    2577 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 02 02:46:59.583618 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.575738    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 02:46:59.583618 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.575744    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 02:46:59.583618 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.575748    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 02:46:59.583618 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.575751    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 02:46:59.583618 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.575755    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 02:46:59.583618 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.575758    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 02:46:59.583618 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.575761    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 02:46:59.584334 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.575764    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 02:46:59.584334 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.575767    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 02:46:59.584334 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.575770    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 02:46:59.584334 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.575773    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 02:46:59.584334 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.575776    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 02:46:59.584334 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.575779    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 02:46:59.584334 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.575781    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 02:46:59.584334 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.575784    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 02:46:59.584334 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.575787    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 02:46:59.584334 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.575790    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 02:46:59.584334 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.575793    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 02:46:59.584334 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.575796    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 02:46:59.584334 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.575799    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 02:46:59.584334 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.575801    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 02:46:59.584334 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.575804    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 02:46:59.584334 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.575807    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 02:46:59.584334 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.575810    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 02:46:59.584334 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.575813    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 02:46:59.584334 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.575816    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 02:46:59.584334 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.575819    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 02:46:59.585096 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.575822    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 02:46:59.585096 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.575825    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 02:46:59.585096 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.575829    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 02:46:59.585096 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.575832    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 02:46:59.585096 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.575834    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 02:46:59.585096 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.575837    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 02:46:59.585096 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.575839    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 02:46:59.585096 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.575842    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 02:46:59.585096 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.575845    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 02:46:59.585096 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.575847    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 02:46:59.585096 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.575850    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 02:46:59.585096 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.575853    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 02:46:59.585096 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.575856    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 02:46:59.585096 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.575858    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 02:46:59.585096 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.575862    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 02:46:59.585096 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.575867    2577 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 02:46:59.585096 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.575870    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 02:46:59.585096 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.575873    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 02:46:59.585096 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.575876    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 02:46:59.585572 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.575879    2577 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 02:46:59.585572 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.575882    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 02:46:59.585572 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.575885    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 02:46:59.585572 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.575888    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 02:46:59.585572 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.575891    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 02:46:59.585572 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.575894    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 02:46:59.585572 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.575896    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 02:46:59.585572 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.575899    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 02:46:59.585572 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.575902    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 02:46:59.585572 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.575905    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 02:46:59.585572 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.575907    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 02:46:59.585572 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.575910    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 02:46:59.585572 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.575912    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 02:46:59.585572 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.575916    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 02:46:59.585572 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.575919    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 02:46:59.585572 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.575921    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 02:46:59.585572 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.575925    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 02:46:59.585572 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.575927    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 02:46:59.585572 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.575930    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 02:46:59.585572 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.575933    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 02:46:59.586155 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.575935    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 02:46:59.586155 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.575938    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 02:46:59.586155 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.575941    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 02:46:59.586155 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.575944    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 02:46:59.586155 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.575946    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 02:46:59.586155 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.575949    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 02:46:59.586155 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.575952    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 02:46:59.586155 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.575954    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 02:46:59.586155 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.575957    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 02:46:59.586155 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.575960    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 02:46:59.586155 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.575962    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 02:46:59.586155 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.575965    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 02:46:59.586155 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.575968    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 02:46:59.586155 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.575972    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 02:46:59.586155 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.575977    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 02:46:59.586155 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.575980    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 02:46:59.586155 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.575983    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 02:46:59.586155 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.575987    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 02:46:59.586155 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.575990    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 02:46:59.586738 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.575993    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 02:46:59.586738 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.575995    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 02:46:59.586738 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.575998    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 02:46:59.586738 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.576001    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 02:46:59.586738 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.576004    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 02:46:59.586738 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.576006    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 02:46:59.586738 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.576009    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 02:46:59.586738 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.576027    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 02:46:59.586738 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.576031    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 02:46:59.586738 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.576034    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 02:46:59.586738 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.576038    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 02:46:59.586738 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.576041    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 02:46:59.586738 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.576044    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 02:46:59.586738 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.576046    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 02:46:59.586738 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.576049    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 02:46:59.586738 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.576052    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 02:46:59.586738 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.576054    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 02:46:59.586738 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.576057    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 02:46:59.586738 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.576060    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 02:46:59.586738 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.576062    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 02:46:59.587286 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.576065    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 02:46:59.587286 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.576094    2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 02:46:59.587286 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577345    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 02:46:59.587286 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577362    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 02:46:59.587286 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577366    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 02:46:59.587286 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577370    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 02:46:59.587286 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577374    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 02:46:59.587286 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577377    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 02:46:59.587286 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577380    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 02:46:59.587286 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577384    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 02:46:59.587286 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577386    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 02:46:59.587286 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577389    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 02:46:59.587286 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577392    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 02:46:59.587286 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577395    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 02:46:59.587286 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577397    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 02:46:59.587657 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577400    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 02:46:59.587657 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577402    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 02:46:59.587657 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577405    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 02:46:59.587657 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577408    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 02:46:59.587657 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577411    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 02:46:59.587657 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577414    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 02:46:59.587657 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577416    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 02:46:59.587657 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577419    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 02:46:59.587657 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577422    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 02:46:59.587657 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577424    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 02:46:59.587657 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577428    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 02:46:59.587657 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577437    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 02:46:59.587657 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577441    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 02:46:59.587657 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577444    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 02:46:59.587657 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577447    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 02:46:59.587657 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577450    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 02:46:59.587657 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577452    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 02:46:59.587657 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577455    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 02:46:59.587657 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577458    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 02:46:59.588125 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577461    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 02:46:59.588125 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577464    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 02:46:59.588125 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577467    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 02:46:59.588125 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577469    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 02:46:59.588125 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577472    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 02:46:59.588125 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577475    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 02:46:59.588125 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577477    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 02:46:59.588125 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577480    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 02:46:59.588125 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577483    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 02:46:59.588125 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577486    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 02:46:59.588125 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577489    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 02:46:59.588125 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577491    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 02:46:59.588125 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577494    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 02:46:59.588125 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577496    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 02:46:59.588125 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577499    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 02:46:59.588125 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577503    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 02:46:59.588125 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577508    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 02:46:59.588125 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577512    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 02:46:59.588125 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577514    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 02:46:59.588125 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577517    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 02:46:59.588597 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577520    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 02:46:59.588597 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577523    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 02:46:59.588597 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577525    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 02:46:59.588597 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577529    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 02:46:59.588597 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577531    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 02:46:59.588597 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577534    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 02:46:59.588597 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577537    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 02:46:59.588597 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577540    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 02:46:59.588597 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577542    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 02:46:59.588597 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577546    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 02:46:59.588597 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577550    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 02:46:59.588597 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577553    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 02:46:59.588597 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577556    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 02:46:59.588597 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577559    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 02:46:59.588597 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577562    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 02:46:59.588597 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577565    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 02:46:59.588597 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577568    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 02:46:59.588597 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577571    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 02:46:59.588597 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577574    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 02:46:59.588597 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577576    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 02:46:59.589075 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577579    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 02:46:59.589075 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577582    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 02:46:59.589075 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577584    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 02:46:59.589075 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577587    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 02:46:59.589075 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577590    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 02:46:59.589075 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577593    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 02:46:59.589075 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577596    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 02:46:59.589075 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577599    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 02:46:59.589075 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577602    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 02:46:59.589075 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577605    2577 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 02:46:59.589075 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577607    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 02:46:59.589075 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577611    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 02:46:59.589075 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577614    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 02:46:59.589075 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577617    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 02:46:59.589075 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577619    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 02:46:59.589075 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577622    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 02:46:59.589075 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577624    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 02:46:59.589075 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577628    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 02:46:59.589075 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577630    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 02:46:59.589516 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577633    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 02:46:59.589516 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577635    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 02:46:59.589516 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577638    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 02:46:59.589516 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577641    2577 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 02:46:59.589516 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577643    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 02:46:59.589516 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577646    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 02:46:59.589516 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577649    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 02:46:59.589516 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577652    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 02:46:59.589516 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577654    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 02:46:59.589516 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577657    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 02:46:59.589516 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577660    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 02:46:59.589516 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577663    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 02:46:59.589516 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577665    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 02:46:59.589516 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577668    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 02:46:59.589516 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577671    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 02:46:59.589516 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.577698    2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 02:46:59.589883 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577752    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 02:46:59.589883 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577757    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 02:46:59.589883 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577760    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 02:46:59.589883 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577763    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 02:46:59.589883 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577766    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 02:46:59.589883 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577769    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 02:46:59.589883 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577772    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 02:46:59.589883 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577774    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 02:46:59.589883 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577777    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 02:46:59.589883 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577780    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 02:46:59.589883 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577795    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 02:46:59.589883 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577800    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 02:46:59.589883 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577803    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 02:46:59.589883 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577805    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 02:46:59.589883 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577809    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 02:46:59.589883 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577812    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 02:46:59.589883 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577815    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 02:46:59.589883 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577817    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 02:46:59.589883 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577820    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 02:46:59.590415 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577823    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 02:46:59.590415 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577826    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 02:46:59.590415 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577828    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 02:46:59.590415 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577831    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 02:46:59.590415 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577834    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 02:46:59.590415 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577836    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 02:46:59.590415 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577839    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 02:46:59.590415 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577842    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 02:46:59.590415 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577844    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 02:46:59.590415 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577847    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 02:46:59.590415 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577850    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 02:46:59.590415 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577852    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 02:46:59.590415 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577855    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 02:46:59.590415 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577857    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 02:46:59.590415 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577860    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 02:46:59.590415 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577863    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 02:46:59.590415 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577865    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 02:46:59.590415 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577868    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 02:46:59.590415 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577871    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 02:46:59.590858 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577873    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 02:46:59.590858 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577876    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 02:46:59.590858 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577878    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 02:46:59.590858 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577881    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 02:46:59.590858 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577884    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 02:46:59.590858 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577886    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 02:46:59.590858 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577889    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 02:46:59.590858 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577892    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 02:46:59.590858 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577895    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 02:46:59.590858 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577897    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 02:46:59.590858 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577900    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 02:46:59.590858 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577903    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 02:46:59.590858 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577905    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 02:46:59.590858 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577908    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 02:46:59.590858 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577910    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 02:46:59.590858 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577913    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 02:46:59.590858 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577916    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 02:46:59.590858 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577918    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 02:46:59.590858 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577921    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 02:46:59.590858 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577924    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 02:46:59.591357 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577927    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 02:46:59.591357 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577932    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 02:46:59.591357 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577935    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 02:46:59.591357 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577937    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 02:46:59.591357 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577940    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 02:46:59.591357 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577943    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 02:46:59.591357 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577945    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 02:46:59.591357 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577948    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 02:46:59.591357 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577951    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 02:46:59.591357 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577953    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 02:46:59.591357 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577956    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 02:46:59.591357 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577958    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 02:46:59.591357 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577961    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 02:46:59.591357 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577964    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 02:46:59.591357 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577968    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 02:46:59.591357 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577972    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 02:46:59.591357 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577975    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 02:46:59.591357 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577979    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 02:46:59.591357 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577982    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 02:46:59.591790 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577984    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 02:46:59.591790 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577987    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 02:46:59.591790 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577990    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 02:46:59.591790 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577992    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 02:46:59.591790 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577995    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 02:46:59.591790 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.577997    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 02:46:59.591790 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.578000    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 02:46:59.591790 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.578002    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 02:46:59.591790 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.578005    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 02:46:59.591790 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.578007    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 02:46:59.591790 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.578027    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 02:46:59.591790 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.578031    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 02:46:59.591790 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.578033    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 02:46:59.591790 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.578037    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 02:46:59.591790 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.578039    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 02:46:59.591790 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.578042    2577 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 02:46:59.591790 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.578044    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 02:46:59.591790 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.578047    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 02:46:59.591790 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.578050    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 02:46:59.591790 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.578053    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 02:46:59.592262 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.578055    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 02:46:59.592262 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.578058    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 02:46:59.592262 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.578061    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 02:46:59.592262 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.578063    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 02:46:59.592262 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.578066    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 02:46:59.592262 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.578068    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 02:46:59.592262 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.578071    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 02:46:59.592262 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.578075    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 02:46:59.592262 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:46:59.578078    2577 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 02:46:59.592262 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.578135    2577 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 02 02:46:59.592262 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.580005    2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Nov 02 02:46:59.592262 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.580932    2577 server.go:1014] "Starting client certificate rotation"
Nov 02 02:46:59.592262 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.581042    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 02 02:46:59.592262 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.581662    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 02 02:46:59.592262 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.582960    2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 02:46:59.592262 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.586005    2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 02:46:59.597379 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.597364    2577 log.go:25] "Validated CRI v1 runtime API"
Nov 02 02:46:59.600977 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.600958    2577 log.go:25] "Validated CRI v1 image API"
Nov 02 02:46:59.602271 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.602243    2577 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 02 02:46:59.604549 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.604531    2577 fs.go:135] Filesystem UUIDs: map[6b1b24d1-2508-478f-959d-02ff53ad6425:/dev/nvme0n1p3 74b93dce-6152-4166-9174-c1be2384bf9a:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Nov 02 02:46:59.604603 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.604549    2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Nov 02 02:46:59.612034 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.611905    2577 manager.go:217] Machine: {Timestamp:2026-11-02 02:46:59.610286237 +0000 UTC m=+0.325354368 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3103607 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ab0e6370e0a3b506477cf0d2b752e SystemUUID:ec2ab0e6-370e-0a3b-5064-77cf0d2b752e BootID:9465d3cb-d6fa-443e-a97e-2a8a74633ee2 Filesystems:[{Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:3d:cb:48:41:27 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:3d:cb:48:41:27 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:b6:74:6d:1c:58:eb Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 02 02:46:59.612034 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.612026    2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 02 02:46:59.612186 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.612104    2577 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 02 02:46:59.614726 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.614693    2577 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 02 02:46:59.614897 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.614727    2577 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-163.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 02 02:46:59.614989 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.614901    2577 topology_manager.go:143] "Creating topology manager with none policy"
Nov 02 02:46:59.614989 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.614914    2577 container_manager_linux.go:311] "Creating device plugin manager"
Nov 02 02:46:59.614989 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.614968    2577 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 02:46:59.615112 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.614994    2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 02:46:59.615715 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.615702    2577 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 02 02:46:59.619748 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.619733    2577 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 02 02:46:59.619879 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.619868    2577 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 02 02:46:59.620507 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.620496    2577 kubelet.go:493] "Attempting to sync node with API server"
Nov 02 02:46:59.620560 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.620514    2577 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 02 02:46:59.620560 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.620527    2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 02 02:46:59.620560 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.620542    2577 kubelet.go:405] "Adding apiserver pod source"
Nov 02 02:46:59.620560 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.620553    2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 02 02:46:59.620916 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.620879    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 02:46:59.624460 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.624445    2577 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 02 02:46:59.628797 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.628784    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 02:46:59.629526 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.629514    2577 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 02 02:46:59.629566 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.629544    2577 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 02 02:46:59.629695 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.629676    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 02 02:46:59.629781 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.629701    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 02 02:46:59.629781 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.629711    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 02 02:46:59.629781 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.629718    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 02 02:46:59.629781 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.629726    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 02 02:46:59.629781 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.629735    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 02 02:46:59.630240 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.630230    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 02 02:46:59.630289 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.630244    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 02 02:46:59.630289 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.630255    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 02 02:46:59.630386 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.630290    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 02 02:46:59.630386 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.630309    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 02 02:46:59.630386 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.630323    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 02 02:46:59.630857 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.630847    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 02 02:46:59.630912 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.630860    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 02 02:46:59.634406 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.634390    2577 server.go:1263] "Started kubelet"
Nov 02 02:46:59.634482 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.634454    2577 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 02 02:46:59.634548 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.634494    2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 02 02:46:59.634588 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.634579    2577 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 02 02:46:59.635164 ip-10-0-133-163 systemd[1]: Started Kubernetes Kubelet.
Nov 02 02:46:59.635871 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.635852    2577 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 02 02:46:59.636763 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.636747    2577 server.go:317] "Adding debug handlers to kubelet server"
Nov 02 02:46:59.640370 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.640347    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 02 02:46:59.640727 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.640708    2577 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 02 02:46:59.641382 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:46:59.641353    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-163.us-west-1.compute.internal\" not found"
Nov 02 02:46:59.642065 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.642046    2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 02 02:46:59.642143 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.642055    2577 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 02 02:46:59.642143 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.642081    2577 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 02 02:46:59.642250 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.642162    2577 factory.go:153] Registering CRI-O factory
Nov 02 02:46:59.642250 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.642196    2577 reconstruct.go:97] "Volume reconstruction finished"
Nov 02 02:46:59.642250 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.642202    2577 factory.go:223] Registration of the crio container factory successfully
Nov 02 02:46:59.642380 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.642256    2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 02 02:46:59.642380 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.642266    2577 factory.go:55] Registering systemd factory
Nov 02 02:46:59.642380 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.642274    2577 factory.go:223] Registration of the systemd container factory successfully
Nov 02 02:46:59.642380 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.642290    2577 factory.go:103] Registering Raw factory
Nov 02 02:46:59.642380 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.642302    2577 manager.go:1199] Started watching for new ooms in manager
Nov 02 02:46:59.642557 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.642204    2577 reconciler.go:29] "Reconciler: start to sync state"
Nov 02 02:46:59.642685 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:46:59.642650    2577 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Nov 02 02:46:59.645150 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.644400    2577 manager.go:319] Starting recovery of all containers
Nov 02 02:46:59.656167 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.656145    2577 manager.go:324] Recovery completed
Nov 02 02:46:59.660977 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.660962    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 02:46:59.666266 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.666237    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-163.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 02 02:46:59.666341 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.666278    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-163.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 02:46:59.666341 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.666293    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-163.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 02 02:46:59.666829 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.666813    2577 cpu_manager.go:226] "Starting" policy="none"
Nov 02 02:46:59.666891 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.666829    2577 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 02 02:46:59.666891 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.666850    2577 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 02 02:46:59.666891 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.666882    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 02:46:59.667645 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.667627    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-163.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 02 02:46:59.667712 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.667657    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-163.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 02:46:59.667712 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.667671    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-163.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 02 02:46:59.669248 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.669228    2577 policy_none.go:50] "Start"
Nov 02 02:46:59.669303 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.669255    2577 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 02 02:46:59.669303 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.669269    2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 02 02:46:59.671310 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.671299    2577 policy_none.go:44] "Start"
Nov 02 02:46:59.722390 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.709702    2577 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 02 02:46:59.722390 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.710798    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 02 02:46:59.722390 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.712054    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 02 02:46:59.722390 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.712083    2577 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 02 02:46:59.722390 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.712104    2577 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 02 02:46:59.722390 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:46:59.712150    2577 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 02 02:46:59.722390 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.715115    2577 manager.go:341] "Starting Device Plugin manager"
Nov 02 02:46:59.722390 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:46:59.715146    2577 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 02 02:46:59.722390 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.715157    2577 server.go:87] "Starting device plugin registration server"
Nov 02 02:46:59.722390 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.715418    2577 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 02 02:46:59.722390 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.715429    2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 02 02:46:59.722390 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.715548    2577 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 02 02:46:59.722390 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.715618    2577 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 02 02:46:59.722390 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.715626    2577 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 02 02:46:59.722390 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:46:59.716254    2577 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 02 02:46:59.722390 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:46:59.716303    2577 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-163.us-west-1.compute.internal\" not found"
Nov 02 02:46:59.775981 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.775945    2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-163.us-west-1.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Nov 02 02:46:59.776191 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:46:59.775949    2577 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-163.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Nov 02 02:46:59.776851 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:46:59.775942    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-163.us-west-1.compute.internal.18e41a6fedd83fcb  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-163.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-163.us-west-1.compute.internal,},FirstTimestamp:2026-11-02 02:46:59.634364363 +0000 UTC m=+0.349432490,LastTimestamp:2026-11-02 02:46:59.634364363 +0000 UTC m=+0.349432490,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-163.us-west-1.compute.internal,}"
Nov 02 02:46:59.816153 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.816086    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 02:46:59.816153 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.816114    2577 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-163.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-163.us-west-1.compute.internal"]
Nov 02 02:46:59.816287 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.816198    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 02:46:59.818051 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.818028    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-163.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 02 02:46:59.818159 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.818029    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-163.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 02 02:46:59.818159 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.818094    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-163.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 02:46:59.818159 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.818109    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-163.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 02 02:46:59.818159 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.818061    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-163.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 02:46:59.818159 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.818142    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-163.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 02 02:46:59.818307 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.818170    2577 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-133-163.us-west-1.compute.internal"
Nov 02 02:46:59.820622 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.820609    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 02:46:59.820761 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.820748    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-163.us-west-1.compute.internal"
Nov 02 02:46:59.820797 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.820777    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 02:46:59.821411 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.821398    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-163.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 02 02:46:59.821476 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.821419    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-163.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 02 02:46:59.821476 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.821440    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-163.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 02:46:59.821476 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.821451    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-163.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 02 02:46:59.821566 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.821423    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-163.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 02:46:59.821566 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.821504    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-163.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 02 02:46:59.823519 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.823505    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-163.us-west-1.compute.internal"
Nov 02 02:46:59.823570 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.823530    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 02:46:59.824222 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.824207    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-163.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 02 02:46:59.824295 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.824233    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-163.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 02:46:59.824295 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.824244    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-163.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 02 02:46:59.851616 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.851591    2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 02 02:46:59.851707 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.851654    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-hgmmz"
Nov 02 02:46:59.856595 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:46:59.856575    2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-163.us-west-1.compute.internal\" not found" node="ip-10-0-133-163.us-west-1.compute.internal"
Nov 02 02:46:59.859889 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:46:59.859873    2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-163.us-west-1.compute.internal\" not found" node="ip-10-0-133-163.us-west-1.compute.internal"
Nov 02 02:46:59.872230 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:46:59.872146    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-163.us-west-1.compute.internal.18e41a6fefbefc2f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-163.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-133-163.us-west-1.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-133-163.us-west-1.compute.internal,},FirstTimestamp:2026-11-02 02:46:59.666263087 +0000 UTC m=+0.381331214,LastTimestamp:2026-11-02 02:46:59.666263087 +0000 UTC m=+0.381331214,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-163.us-west-1.compute.internal,}"
Nov 02 02:46:59.872336 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:46:59.872260    2577 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Nov 02 02:46:59.872391 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:46:59.872347    2577 reflector.go:204] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass"
Nov 02 02:46:59.922220 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:46:59.922191    2577 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Nov 02 02:46:59.933401 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:46:59.933374    2577 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-133-163.us-west-1.compute.internal\" not found" node="ip-10-0-133-163.us-west-1.compute.internal"
Nov 02 02:46:59.933401 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.933388    2577 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 02:46:59.943678 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.943642    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/562e4cfd5885e6864e00b6a01f2722f3-config\") pod \"kube-apiserver-proxy-ip-10-0-133-163.us-west-1.compute.internal\" (UID: \"562e4cfd5885e6864e00b6a01f2722f3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-163.us-west-1.compute.internal"
Nov 02 02:46:59.943732 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.943685    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/950fb6772a40be3e22d676c446372284-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-163.us-west-1.compute.internal\" (UID: \"950fb6772a40be3e22d676c446372284\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-163.us-west-1.compute.internal"
Nov 02 02:46:59.943732 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:46:59.943706    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/950fb6772a40be3e22d676c446372284-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-163.us-west-1.compute.internal\" (UID: \"950fb6772a40be3e22d676c446372284\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-163.us-west-1.compute.internal"
Nov 02 02:47:00.043912 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.043885    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/950fb6772a40be3e22d676c446372284-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-163.us-west-1.compute.internal\" (UID: \"950fb6772a40be3e22d676c446372284\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-163.us-west-1.compute.internal"
Nov 02 02:47:00.043912 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.043911    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/950fb6772a40be3e22d676c446372284-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-163.us-west-1.compute.internal\" (UID: \"950fb6772a40be3e22d676c446372284\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-163.us-west-1.compute.internal"
Nov 02 02:47:00.044117 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.043930    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/562e4cfd5885e6864e00b6a01f2722f3-config\") pod \"kube-apiserver-proxy-ip-10-0-133-163.us-west-1.compute.internal\" (UID: \"562e4cfd5885e6864e00b6a01f2722f3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-163.us-west-1.compute.internal"
Nov 02 02:47:00.044117 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.043989    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/562e4cfd5885e6864e00b6a01f2722f3-config\") pod \"kube-apiserver-proxy-ip-10-0-133-163.us-west-1.compute.internal\" (UID: \"562e4cfd5885e6864e00b6a01f2722f3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-163.us-west-1.compute.internal"
Nov 02 02:47:00.044117 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.043996    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/950fb6772a40be3e22d676c446372284-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-163.us-west-1.compute.internal\" (UID: \"950fb6772a40be3e22d676c446372284\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-163.us-west-1.compute.internal"
Nov 02 02:47:00.044117 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.043998    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/950fb6772a40be3e22d676c446372284-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-163.us-west-1.compute.internal\" (UID: \"950fb6772a40be3e22d676c446372284\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-163.us-west-1.compute.internal"
Nov 02 02:47:00.078516 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:00.078473    2577 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-133-163.us-west-1.compute.internal\" not found" node="ip-10-0-133-163.us-west-1.compute.internal"
Nov 02 02:47:00.133693 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.133670    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 02:47:00.134717 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.134703    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-163.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 02 02:47:00.134768 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.134732    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-163.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 02:47:00.134768 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.134742    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-163.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 02 02:47:00.134768 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.134768    2577 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-133-163.us-west-1.compute.internal"
Nov 02 02:47:00.159518 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.159498    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-163.us-west-1.compute.internal"
Nov 02 02:47:00.162451 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.162436    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-163.us-west-1.compute.internal"
Nov 02 02:47:00.191114 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.191097    2577 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-133-163.us-west-1.compute.internal"
Nov 02 02:47:00.241935 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.241910    2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-163.us-west-1.compute.internal"
Nov 02 02:47:00.302940 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.302914    2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 02:47:00.303874 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.303859    2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-163.us-west-1.compute.internal"
Nov 02 02:47:00.360435 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.360380    2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 02:47:00.580806 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.580782    2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Nov 02 02:47:00.581337 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:00.580937    2577 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://af3a1c6fd9c4d43e7823e9d8be5f460a-a3480c90fef6e68b.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-133-163.us-west-1.compute.internal.18e41a6fefbf5359\": read tcp 10.0.133.163:39798->3.151.167.219:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-133-163.us-west-1.compute.internal.18e41a6fefbf5359  default   5111 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-163.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-133-163.us-west-1.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-133-163.us-west-1.compute.internal,},FirstTimestamp:2026-11-02 02:46:59 +0000 UTC,LastTimestamp:2026-11-02 02:46:59.818129706 +0000 UTC m=+0.533197842,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-163.us-west-1.compute.internal,}"
Nov 02 02:47:00.621683 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.621638    2577 apiserver.go:52] "Watching apiserver"
Nov 02 02:47:00.640983 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.640966    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Nov 02 02:47:00.856017 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.855977    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 02:42:00 +0000 UTC" deadline="2028-07-11 02:27:09.951492822 +0000 UTC"
Nov 02 02:47:00.856017 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.856012    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14807h40m9.095483365s"
Nov 02 02:47:00.900209 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.900159    2577 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 02 02:47:00.900688 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.900664    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zlplc","openshift-multus/network-metrics-daemon-zkjqm","openshift-network-operator/iptables-alerter-8sxd5","openshift-cluster-node-tuning-operator/tuned-7qpcf","openshift-dns/node-resolver-sztr6","openshift-multus/multus-additional-cni-plugins-69j4f","openshift-network-diagnostics/network-check-target-47f7x","kube-system/global-pull-secret-syncer-dxlld","openshift-multus/multus-tt68t","openshift-ovn-kubernetes/ovnkube-node-hww8q","kube-system/kube-apiserver-proxy-ip-10-0-133-163.us-west-1.compute.internal","openshift-image-registry/node-ca-kkqzw","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-163.us-west-1.compute.internal","kube-system/konnectivity-agent-z8sq6"]
Nov 02 02:47:00.901542 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.901526    2577 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 02:47:00.901818 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.901802    2577 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 02:47:00.903278 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.903259    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zlplc"
Nov 02 02:47:00.905509 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.905490    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zkjqm"
Nov 02 02:47:00.905606 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:00.905559    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zkjqm" podUID="086e3504-d2fe-43ad-bc5e-dca409a4e695"
Nov 02 02:47:00.907669 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.907658    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-8sxd5"
Nov 02 02:47:00.909930 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.909913    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7qpcf"
Nov 02 02:47:00.912130 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.912114    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-sztr6"
Nov 02 02:47:00.914571 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.914550    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-69j4f"
Nov 02 02:47:00.916864 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.916847    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-47f7x"
Nov 02 02:47:00.916959 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:00.916925    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-47f7x" podUID="44f31bc6-bd01-4f16-ad6e-0d1ec90bfb4f"
Nov 02 02:47:00.919168 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.919154    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dxlld"
Nov 02 02:47:00.919233 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:00.919215    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dxlld" podUID="e400bad6-59c3-4c43-aed0-40c731c967db"
Nov 02 02:47:00.921341 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.921325    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-tt68t"
Nov 02 02:47:00.923717 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.923700    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:00.926083 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.926064    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-kkqzw"
Nov 02 02:47:00.928577 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.928560    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-z8sq6"
Nov 02 02:47:00.943784 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.943765    2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 02 02:47:00.947719 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.947704    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/94fb7403-b6ef-4d42-be18-7c45b5c10692-lib-modules\") pod \"tuned-7qpcf\" (UID: \"94fb7403-b6ef-4d42-be18-7c45b5c10692\") " pod="openshift-cluster-node-tuning-operator/tuned-7qpcf"
Nov 02 02:47:00.947780 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.947738    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5c797b08-a314-44ed-8026-8a0ce1f6c36f-run-ovn\") pod \"ovnkube-node-hww8q\" (UID: \"5c797b08-a314-44ed-8026-8a0ce1f6c36f\") " pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:00.947780 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.947760    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68r2v\" (UniqueName: \"kubernetes.io/projected/a81017e4-d705-438e-833b-eedd6de0edde-kube-api-access-68r2v\") pod \"multus-additional-cni-plugins-69j4f\" (UID: \"a81017e4-d705-438e-833b-eedd6de0edde\") " pod="openshift-multus/multus-additional-cni-plugins-69j4f"
Nov 02 02:47:00.947852 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.947784    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c2f75412-4413-45d1-aa82-919a5206efae-cni-binary-copy\") pod \"multus-tt68t\" (UID: \"c2f75412-4413-45d1-aa82-919a5206efae\") " pod="openshift-multus/multus-tt68t"
Nov 02 02:47:00.947852 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.947799    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e701e783-5b43-45d7-b88b-83fd3c4af13b-serviceca\") pod \"node-ca-kkqzw\" (UID: \"e701e783-5b43-45d7-b88b-83fd3c4af13b\") " pod="openshift-image-registry/node-ca-kkqzw"
Nov 02 02:47:00.947852 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.947815    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5c797b08-a314-44ed-8026-8a0ce1f6c36f-ovnkube-script-lib\") pod \"ovnkube-node-hww8q\" (UID: \"5c797b08-a314-44ed-8026-8a0ce1f6c36f\") " pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:00.947852 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.947830    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/94fb7403-b6ef-4d42-be18-7c45b5c10692-etc-modprobe-d\") pod \"tuned-7qpcf\" (UID: \"94fb7403-b6ef-4d42-be18-7c45b5c10692\") " pod="openshift-cluster-node-tuning-operator/tuned-7qpcf"
Nov 02 02:47:00.948024 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.947971    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5dxg\" (UniqueName: \"kubernetes.io/projected/5c797b08-a314-44ed-8026-8a0ce1f6c36f-kube-api-access-b5dxg\") pod \"ovnkube-node-hww8q\" (UID: \"5c797b08-a314-44ed-8026-8a0ce1f6c36f\") " pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:00.948104 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.948011    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c2f75412-4413-45d1-aa82-919a5206efae-cnibin\") pod \"multus-tt68t\" (UID: \"c2f75412-4413-45d1-aa82-919a5206efae\") " pod="openshift-multus/multus-tt68t"
Nov 02 02:47:00.948104 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.948071    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c078a1ee-3179-45c9-9d3b-4d4a46f0c42f-agent-certs\") pod \"konnectivity-agent-z8sq6\" (UID: \"c078a1ee-3179-45c9-9d3b-4d4a46f0c42f\") " pod="kube-system/konnectivity-agent-z8sq6"
Nov 02 02:47:00.948215 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.948118    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/94fb7403-b6ef-4d42-be18-7c45b5c10692-tmp\") pod \"tuned-7qpcf\" (UID: \"94fb7403-b6ef-4d42-be18-7c45b5c10692\") " pod="openshift-cluster-node-tuning-operator/tuned-7qpcf"
Nov 02 02:47:00.948215 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.948173    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5c797b08-a314-44ed-8026-8a0ce1f6c36f-etc-openvswitch\") pod \"ovnkube-node-hww8q\" (UID: \"5c797b08-a314-44ed-8026-8a0ce1f6c36f\") " pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:00.948299 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.948233    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bb7ec0c6-6988-45e6-b578-1b68b3a51d9d-registration-dir\") pod \"aws-ebs-csi-driver-node-zlplc\" (UID: \"bb7ec0c6-6988-45e6-b578-1b68b3a51d9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zlplc"
Nov 02 02:47:00.948299 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.948255    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a81017e4-d705-438e-833b-eedd6de0edde-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-69j4f\" (UID: \"a81017e4-d705-438e-833b-eedd6de0edde\") " pod="openshift-multus/multus-additional-cni-plugins-69j4f"
Nov 02 02:47:00.948376 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.948316    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c2f75412-4413-45d1-aa82-919a5206efae-host-var-lib-cni-multus\") pod \"multus-tt68t\" (UID: \"c2f75412-4413-45d1-aa82-919a5206efae\") " pod="openshift-multus/multus-tt68t"
Nov 02 02:47:00.948376 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.948348    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5c797b08-a314-44ed-8026-8a0ce1f6c36f-host-run-netns\") pod \"ovnkube-node-hww8q\" (UID: \"5c797b08-a314-44ed-8026-8a0ce1f6c36f\") " pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:00.948453 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.948376    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5c797b08-a314-44ed-8026-8a0ce1f6c36f-host-cni-bin\") pod \"ovnkube-node-hww8q\" (UID: \"5c797b08-a314-44ed-8026-8a0ce1f6c36f\") " pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:00.948453 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.948426    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bb7ec0c6-6988-45e6-b578-1b68b3a51d9d-socket-dir\") pod \"aws-ebs-csi-driver-node-zlplc\" (UID: \"bb7ec0c6-6988-45e6-b578-1b68b3a51d9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zlplc"
Nov 02 02:47:00.948453 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.948446    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bb7ec0c6-6988-45e6-b578-1b68b3a51d9d-etc-selinux\") pod \"aws-ebs-csi-driver-node-zlplc\" (UID: \"bb7ec0c6-6988-45e6-b578-1b68b3a51d9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zlplc"
Nov 02 02:47:00.948577 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.948461    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e400bad6-59c3-4c43-aed0-40c731c967db-kubelet-config\") pod \"global-pull-secret-syncer-dxlld\" (UID: \"e400bad6-59c3-4c43-aed0-40c731c967db\") " pod="kube-system/global-pull-secret-syncer-dxlld"
Nov 02 02:47:00.948577 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.948476    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e400bad6-59c3-4c43-aed0-40c731c967db-dbus\") pod \"global-pull-secret-syncer-dxlld\" (UID: \"e400bad6-59c3-4c43-aed0-40c731c967db\") " pod="kube-system/global-pull-secret-syncer-dxlld"
Nov 02 02:47:00.948577 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.948505    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c2f75412-4413-45d1-aa82-919a5206efae-etc-kubernetes\") pod \"multus-tt68t\" (UID: \"c2f75412-4413-45d1-aa82-919a5206efae\") " pod="openshift-multus/multus-tt68t"
Nov 02 02:47:00.948577 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.948559    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a81017e4-d705-438e-833b-eedd6de0edde-system-cni-dir\") pod \"multus-additional-cni-plugins-69j4f\" (UID: \"a81017e4-d705-438e-833b-eedd6de0edde\") " pod="openshift-multus/multus-additional-cni-plugins-69j4f"
Nov 02 02:47:00.948722 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.948604    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/94fb7403-b6ef-4d42-be18-7c45b5c10692-etc-sysctl-conf\") pod \"tuned-7qpcf\" (UID: \"94fb7403-b6ef-4d42-be18-7c45b5c10692\") " pod="openshift-cluster-node-tuning-operator/tuned-7qpcf"
Nov 02 02:47:00.948722 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.948642    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/94fb7403-b6ef-4d42-be18-7c45b5c10692-var-lib-kubelet\") pod \"tuned-7qpcf\" (UID: \"94fb7403-b6ef-4d42-be18-7c45b5c10692\") " pod="openshift-cluster-node-tuning-operator/tuned-7qpcf"
Nov 02 02:47:00.948722 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.948705    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5c797b08-a314-44ed-8026-8a0ce1f6c36f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hww8q\" (UID: \"5c797b08-a314-44ed-8026-8a0ce1f6c36f\") " pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:00.948861 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.948736    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a81017e4-d705-438e-833b-eedd6de0edde-cnibin\") pod \"multus-additional-cni-plugins-69j4f\" (UID: \"a81017e4-d705-438e-833b-eedd6de0edde\") " pod="openshift-multus/multus-additional-cni-plugins-69j4f"
Nov 02 02:47:00.948861 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.948785    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a81017e4-d705-438e-833b-eedd6de0edde-cni-binary-copy\") pod \"multus-additional-cni-plugins-69j4f\" (UID: \"a81017e4-d705-438e-833b-eedd6de0edde\") " pod="openshift-multus/multus-additional-cni-plugins-69j4f"
Nov 02 02:47:00.948941 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.948870    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c2f75412-4413-45d1-aa82-919a5206efae-hostroot\") pod \"multus-tt68t\" (UID: \"c2f75412-4413-45d1-aa82-919a5206efae\") " pod="openshift-multus/multus-tt68t"
Nov 02 02:47:00.948941 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.948887    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/94fb7403-b6ef-4d42-be18-7c45b5c10692-etc-kubernetes\") pod \"tuned-7qpcf\" (UID: \"94fb7403-b6ef-4d42-be18-7c45b5c10692\") " pod="openshift-cluster-node-tuning-operator/tuned-7qpcf"
Nov 02 02:47:00.948941 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.948902    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5c797b08-a314-44ed-8026-8a0ce1f6c36f-run-systemd\") pod \"ovnkube-node-hww8q\" (UID: \"5c797b08-a314-44ed-8026-8a0ce1f6c36f\") " pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:00.948941 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.948916    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5c797b08-a314-44ed-8026-8a0ce1f6c36f-env-overrides\") pod \"ovnkube-node-hww8q\" (UID: \"5c797b08-a314-44ed-8026-8a0ce1f6c36f\") " pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:00.949130 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.948948    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-984cg\" (UniqueName: \"kubernetes.io/projected/086e3504-d2fe-43ad-bc5e-dca409a4e695-kube-api-access-984cg\") pod \"network-metrics-daemon-zkjqm\" (UID: \"086e3504-d2fe-43ad-bc5e-dca409a4e695\") " pod="openshift-multus/network-metrics-daemon-zkjqm"
Nov 02 02:47:00.949130 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.948989    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c2f75412-4413-45d1-aa82-919a5206efae-multus-socket-dir-parent\") pod \"multus-tt68t\" (UID: \"c2f75412-4413-45d1-aa82-919a5206efae\") " pod="openshift-multus/multus-tt68t"
Nov 02 02:47:00.949130 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.949036    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e701e783-5b43-45d7-b88b-83fd3c4af13b-host\") pod \"node-ca-kkqzw\" (UID: \"e701e783-5b43-45d7-b88b-83fd3c4af13b\") " pod="openshift-image-registry/node-ca-kkqzw"
Nov 02 02:47:00.949130 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.949058    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f2071fbb-7d2f-49d4-9366-34332e71c6bc-host-slash\") pod \"iptables-alerter-8sxd5\" (UID: \"f2071fbb-7d2f-49d4-9366-34332e71c6bc\") " pod="openshift-network-operator/iptables-alerter-8sxd5"
Nov 02 02:47:00.949130 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.949077    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4fec139d-da70-4cf6-8730-b578df3bebcc-hosts-file\") pod \"node-resolver-sztr6\" (UID: \"4fec139d-da70-4cf6-8730-b578df3bebcc\") " pod="openshift-dns/node-resolver-sztr6"
Nov 02 02:47:00.949130 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.949108    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bb7ec0c6-6988-45e6-b578-1b68b3a51d9d-device-dir\") pod \"aws-ebs-csi-driver-node-zlplc\" (UID: \"bb7ec0c6-6988-45e6-b578-1b68b3a51d9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zlplc"
Nov 02 02:47:00.949130 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.949132    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c2f75412-4413-45d1-aa82-919a5206efae-host-var-lib-kubelet\") pod \"multus-tt68t\" (UID: \"c2f75412-4413-45d1-aa82-919a5206efae\") " pod="openshift-multus/multus-tt68t"
Nov 02 02:47:00.949419 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.949149    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c2f75412-4413-45d1-aa82-919a5206efae-multus-conf-dir\") pod \"multus-tt68t\" (UID: \"c2f75412-4413-45d1-aa82-919a5206efae\") " pod="openshift-multus/multus-tt68t"
Nov 02 02:47:00.949419 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.949179    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c2f75412-4413-45d1-aa82-919a5206efae-host-run-multus-certs\") pod \"multus-tt68t\" (UID: \"c2f75412-4413-45d1-aa82-919a5206efae\") " pod="openshift-multus/multus-tt68t"
Nov 02 02:47:00.949419 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.949202    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5c797b08-a314-44ed-8026-8a0ce1f6c36f-systemd-units\") pod \"ovnkube-node-hww8q\" (UID: \"5c797b08-a314-44ed-8026-8a0ce1f6c36f\") " pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:00.949419 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.949219    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5c797b08-a314-44ed-8026-8a0ce1f6c36f-host-slash\") pod \"ovnkube-node-hww8q\" (UID: \"5c797b08-a314-44ed-8026-8a0ce1f6c36f\") " pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:00.949419 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.949236    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5c797b08-a314-44ed-8026-8a0ce1f6c36f-log-socket\") pod \"ovnkube-node-hww8q\" (UID: \"5c797b08-a314-44ed-8026-8a0ce1f6c36f\") " pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:00.949419 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.949251    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5c797b08-a314-44ed-8026-8a0ce1f6c36f-host-run-ovn-kubernetes\") pod \"ovnkube-node-hww8q\" (UID: \"5c797b08-a314-44ed-8026-8a0ce1f6c36f\") " pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:00.949419 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.949276    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bb7ec0c6-6988-45e6-b578-1b68b3a51d9d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zlplc\" (UID: \"bb7ec0c6-6988-45e6-b578-1b68b3a51d9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zlplc"
Nov 02 02:47:00.949419 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.949292    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5c797b08-a314-44ed-8026-8a0ce1f6c36f-run-openvswitch\") pod \"ovnkube-node-hww8q\" (UID: \"5c797b08-a314-44ed-8026-8a0ce1f6c36f\") " pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:00.949419 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.949308    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5c797b08-a314-44ed-8026-8a0ce1f6c36f-node-log\") pod \"ovnkube-node-hww8q\" (UID: \"5c797b08-a314-44ed-8026-8a0ce1f6c36f\") " pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:00.949419 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.949331    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bb7ec0c6-6988-45e6-b578-1b68b3a51d9d-sys-fs\") pod \"aws-ebs-csi-driver-node-zlplc\" (UID: \"bb7ec0c6-6988-45e6-b578-1b68b3a51d9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zlplc"
Nov 02 02:47:00.949419 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.949366    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ln99j\" (UniqueName: \"kubernetes.io/projected/bb7ec0c6-6988-45e6-b578-1b68b3a51d9d-kube-api-access-ln99j\") pod \"aws-ebs-csi-driver-node-zlplc\" (UID: \"bb7ec0c6-6988-45e6-b578-1b68b3a51d9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zlplc"
Nov 02 02:47:00.949419 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.949410    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rrzn\" (UniqueName: \"kubernetes.io/projected/44f31bc6-bd01-4f16-ad6e-0d1ec90bfb4f-kube-api-access-7rrzn\") pod \"network-check-target-47f7x\" (UID: \"44f31bc6-bd01-4f16-ad6e-0d1ec90bfb4f\") " pod="openshift-network-diagnostics/network-check-target-47f7x"
Nov 02 02:47:00.949918 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.949443    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hclh\" (UniqueName: \"kubernetes.io/projected/c2f75412-4413-45d1-aa82-919a5206efae-kube-api-access-5hclh\") pod \"multus-tt68t\" (UID: \"c2f75412-4413-45d1-aa82-919a5206efae\") " pod="openshift-multus/multus-tt68t"
Nov 02 02:47:00.949918 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.949492    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/94fb7403-b6ef-4d42-be18-7c45b5c10692-etc-sysconfig\") pod \"tuned-7qpcf\" (UID: \"94fb7403-b6ef-4d42-be18-7c45b5c10692\") " pod="openshift-cluster-node-tuning-operator/tuned-7qpcf"
Nov 02 02:47:00.949918 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.949515    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/94fb7403-b6ef-4d42-be18-7c45b5c10692-etc-sysctl-d\") pod \"tuned-7qpcf\" (UID: \"94fb7403-b6ef-4d42-be18-7c45b5c10692\") " pod="openshift-cluster-node-tuning-operator/tuned-7qpcf"
Nov 02 02:47:00.949918 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.949533    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqtjv\" (UniqueName: \"kubernetes.io/projected/94fb7403-b6ef-4d42-be18-7c45b5c10692-kube-api-access-cqtjv\") pod \"tuned-7qpcf\" (UID: \"94fb7403-b6ef-4d42-be18-7c45b5c10692\") " pod="openshift-cluster-node-tuning-operator/tuned-7qpcf"
Nov 02 02:47:00.949918 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.949561    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzwzx\" (UniqueName: \"kubernetes.io/projected/4fec139d-da70-4cf6-8730-b578df3bebcc-kube-api-access-kzwzx\") pod \"node-resolver-sztr6\" (UID: \"4fec139d-da70-4cf6-8730-b578df3bebcc\") " pod="openshift-dns/node-resolver-sztr6"
Nov 02 02:47:00.949918 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.949590    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5c797b08-a314-44ed-8026-8a0ce1f6c36f-host-kubelet\") pod \"ovnkube-node-hww8q\" (UID: \"5c797b08-a314-44ed-8026-8a0ce1f6c36f\") " pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:00.949918 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.949623    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c078a1ee-3179-45c9-9d3b-4d4a46f0c42f-konnectivity-ca\") pod \"konnectivity-agent-z8sq6\" (UID: \"c078a1ee-3179-45c9-9d3b-4d4a46f0c42f\") " pod="kube-system/konnectivity-agent-z8sq6"
Nov 02 02:47:00.949918 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.949676    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/94fb7403-b6ef-4d42-be18-7c45b5c10692-sys\") pod \"tuned-7qpcf\" (UID: \"94fb7403-b6ef-4d42-be18-7c45b5c10692\") " pod="openshift-cluster-node-tuning-operator/tuned-7qpcf"
Nov 02 02:47:00.949918 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.949698    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5c797b08-a314-44ed-8026-8a0ce1f6c36f-ovnkube-config\") pod \"ovnkube-node-hww8q\" (UID: \"5c797b08-a314-44ed-8026-8a0ce1f6c36f\") " pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:00.949918 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.949733    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5c797b08-a314-44ed-8026-8a0ce1f6c36f-ovn-node-metrics-cert\") pod \"ovnkube-node-hww8q\" (UID: \"5c797b08-a314-44ed-8026-8a0ce1f6c36f\") " pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:00.949918 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.949763    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c2f75412-4413-45d1-aa82-919a5206efae-host-run-k8s-cni-cncf-io\") pod \"multus-tt68t\" (UID: \"c2f75412-4413-45d1-aa82-919a5206efae\") " pod="openshift-multus/multus-tt68t"
Nov 02 02:47:00.949918 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.949809    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c2f75412-4413-45d1-aa82-919a5206efae-host-run-netns\") pod \"multus-tt68t\" (UID: \"c2f75412-4413-45d1-aa82-919a5206efae\") " pod="openshift-multus/multus-tt68t"
Nov 02 02:47:00.949918 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.949873    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c2f75412-4413-45d1-aa82-919a5206efae-host-var-lib-cni-bin\") pod \"multus-tt68t\" (UID: \"c2f75412-4413-45d1-aa82-919a5206efae\") " pod="openshift-multus/multus-tt68t"
Nov 02 02:47:00.949918 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.949910    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f2071fbb-7d2f-49d4-9366-34332e71c6bc-iptables-alerter-script\") pod \"iptables-alerter-8sxd5\" (UID: \"f2071fbb-7d2f-49d4-9366-34332e71c6bc\") " pod="openshift-network-operator/iptables-alerter-8sxd5"
Nov 02 02:47:00.950353 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.949956    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e400bad6-59c3-4c43-aed0-40c731c967db-original-pull-secret\") pod \"global-pull-secret-syncer-dxlld\" (UID: \"e400bad6-59c3-4c43-aed0-40c731c967db\") " pod="kube-system/global-pull-secret-syncer-dxlld"
Nov 02 02:47:00.950353 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.949987    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5c797b08-a314-44ed-8026-8a0ce1f6c36f-var-lib-openvswitch\") pod \"ovnkube-node-hww8q\" (UID: \"5c797b08-a314-44ed-8026-8a0ce1f6c36f\") " pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:00.950353 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.950035    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a81017e4-d705-438e-833b-eedd6de0edde-tuning-conf-dir\") pod \"multus-additional-cni-plugins-69j4f\" (UID: \"a81017e4-d705-438e-833b-eedd6de0edde\") " pod="openshift-multus/multus-additional-cni-plugins-69j4f"
Nov 02 02:47:00.950353 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.950070    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/94fb7403-b6ef-4d42-be18-7c45b5c10692-host\") pod \"tuned-7qpcf\" (UID: \"94fb7403-b6ef-4d42-be18-7c45b5c10692\") " pod="openshift-cluster-node-tuning-operator/tuned-7qpcf"
Nov 02 02:47:00.950353 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.950095    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/94fb7403-b6ef-4d42-be18-7c45b5c10692-etc-tuned\") pod \"tuned-7qpcf\" (UID: \"94fb7403-b6ef-4d42-be18-7c45b5c10692\") " pod="openshift-cluster-node-tuning-operator/tuned-7qpcf"
Nov 02 02:47:00.950353 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.950133    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fec139d-da70-4cf6-8730-b578df3bebcc-tmp-dir\") pod \"node-resolver-sztr6\" (UID: \"4fec139d-da70-4cf6-8730-b578df3bebcc\") " pod="openshift-dns/node-resolver-sztr6"
Nov 02 02:47:00.950353 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.950166    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c2f75412-4413-45d1-aa82-919a5206efae-system-cni-dir\") pod \"multus-tt68t\" (UID: \"c2f75412-4413-45d1-aa82-919a5206efae\") " pod="openshift-multus/multus-tt68t"
Nov 02 02:47:00.950353 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.950186    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/94fb7403-b6ef-4d42-be18-7c45b5c10692-etc-systemd\") pod \"tuned-7qpcf\" (UID: \"94fb7403-b6ef-4d42-be18-7c45b5c10692\") " pod="openshift-cluster-node-tuning-operator/tuned-7qpcf"
Nov 02 02:47:00.950353 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.950206    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/94fb7403-b6ef-4d42-be18-7c45b5c10692-run\") pod \"tuned-7qpcf\" (UID: \"94fb7403-b6ef-4d42-be18-7c45b5c10692\") " pod="openshift-cluster-node-tuning-operator/tuned-7qpcf"
Nov 02 02:47:00.950353 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.950224    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5c797b08-a314-44ed-8026-8a0ce1f6c36f-host-cni-netd\") pod \"ovnkube-node-hww8q\" (UID: \"5c797b08-a314-44ed-8026-8a0ce1f6c36f\") " pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:00.950353 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.950239    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/086e3504-d2fe-43ad-bc5e-dca409a4e695-metrics-certs\") pod \"network-metrics-daemon-zkjqm\" (UID: \"086e3504-d2fe-43ad-bc5e-dca409a4e695\") " pod="openshift-multus/network-metrics-daemon-zkjqm"
Nov 02 02:47:00.950353 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.950256    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a81017e4-d705-438e-833b-eedd6de0edde-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-69j4f\" (UID: \"a81017e4-d705-438e-833b-eedd6de0edde\") " pod="openshift-multus/multus-additional-cni-plugins-69j4f"
Nov 02 02:47:00.950353 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.950305    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c2f75412-4413-45d1-aa82-919a5206efae-multus-cni-dir\") pod \"multus-tt68t\" (UID: \"c2f75412-4413-45d1-aa82-919a5206efae\") " pod="openshift-multus/multus-tt68t"
Nov 02 02:47:00.950353 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.950334    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c2f75412-4413-45d1-aa82-919a5206efae-multus-daemon-config\") pod \"multus-tt68t\" (UID: \"c2f75412-4413-45d1-aa82-919a5206efae\") " pod="openshift-multus/multus-tt68t"
Nov 02 02:47:00.950353 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.950355    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqsvj\" (UniqueName: \"kubernetes.io/projected/e701e783-5b43-45d7-b88b-83fd3c4af13b-kube-api-access-kqsvj\") pod \"node-ca-kkqzw\" (UID: \"e701e783-5b43-45d7-b88b-83fd3c4af13b\") " pod="openshift-image-registry/node-ca-kkqzw"
Nov 02 02:47:00.950812 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.950393    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69zxc\" (UniqueName: \"kubernetes.io/projected/f2071fbb-7d2f-49d4-9366-34332e71c6bc-kube-api-access-69zxc\") pod \"iptables-alerter-8sxd5\" (UID: \"f2071fbb-7d2f-49d4-9366-34332e71c6bc\") " pod="openshift-network-operator/iptables-alerter-8sxd5"
Nov 02 02:47:00.960086 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.960059    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-cgnbb\""
Nov 02 02:47:00.960186 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.960127    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 02 02:47:00.960186 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.960160    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 02 02:47:00.962178 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.962157    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 02 02:47:00.962908 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.962890    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-ghkrt\""
Nov 02 02:47:00.963159 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.963142    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 02 02:47:00.963218 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.963143    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 02 02:47:00.963256 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.963229    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 02 02:47:00.965108 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.965077    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 02 02:47:00.965195 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.965180    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 02 02:47:00.965232 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.965187    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-b8q6l\""
Nov 02 02:47:00.967252 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.967234    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-5mjxc\""
Nov 02 02:47:00.967304 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.967235    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 02 02:47:00.967304 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:00.967290    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 02 02:47:01.010112 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.010090    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-cjmh9\""
Nov 02 02:47:01.010608 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.010582    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 02 02:47:01.010705 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.010618    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-7jkxs\""
Nov 02 02:47:01.010705 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.010660    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 02 02:47:01.010705 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.010666    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 02 02:47:01.010870 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.010730    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 02 02:47:01.010870 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.010738    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 02 02:47:01.010870 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.010746    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 02 02:47:01.010870 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.010847    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 02 02:47:01.011079 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.011071    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 02 02:47:01.011132 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.011118    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-5v9mr\""
Nov 02 02:47:01.013347 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.013190    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 02 02:47:01.013347 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.013214    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 02 02:47:01.013347 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.013253    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 02 02:47:01.013347 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.013262    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 02 02:47:01.013347 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.013280    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 02 02:47:01.013347 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.013292    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 02 02:47:01.013347 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.013301    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 02 02:47:01.013347 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.013300    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-nvmsr\""
Nov 02 02:47:01.013347 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.013336    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 02 02:47:01.013347 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.013350    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-pbbl9\""
Nov 02 02:47:01.013844 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.013357    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 02 02:47:01.013844 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.013607    2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 02 02:47:01.013844 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.013671    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-c2ptm"
Nov 02 02:47:01.050806 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.050779    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bb7ec0c6-6988-45e6-b578-1b68b3a51d9d-registration-dir\") pod \"aws-ebs-csi-driver-node-zlplc\" (UID: \"bb7ec0c6-6988-45e6-b578-1b68b3a51d9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zlplc"
Nov 02 02:47:01.050914 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.050814    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a81017e4-d705-438e-833b-eedd6de0edde-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-69j4f\" (UID: \"a81017e4-d705-438e-833b-eedd6de0edde\") " pod="openshift-multus/multus-additional-cni-plugins-69j4f"
Nov 02 02:47:01.050914 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.050843    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c2f75412-4413-45d1-aa82-919a5206efae-host-var-lib-cni-multus\") pod \"multus-tt68t\" (UID: \"c2f75412-4413-45d1-aa82-919a5206efae\") " pod="openshift-multus/multus-tt68t"
Nov 02 02:47:01.050914 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.050872    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5c797b08-a314-44ed-8026-8a0ce1f6c36f-host-run-netns\") pod \"ovnkube-node-hww8q\" (UID: \"5c797b08-a314-44ed-8026-8a0ce1f6c36f\") " pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:01.050914 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.050897    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bb7ec0c6-6988-45e6-b578-1b68b3a51d9d-registration-dir\") pod \"aws-ebs-csi-driver-node-zlplc\" (UID: \"bb7ec0c6-6988-45e6-b578-1b68b3a51d9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zlplc"
Nov 02 02:47:01.051148 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.050900    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5c797b08-a314-44ed-8026-8a0ce1f6c36f-host-cni-bin\") pod \"ovnkube-node-hww8q\" (UID: \"5c797b08-a314-44ed-8026-8a0ce1f6c36f\") " pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:01.051148 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.050947    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5c797b08-a314-44ed-8026-8a0ce1f6c36f-host-run-netns\") pod \"ovnkube-node-hww8q\" (UID: \"5c797b08-a314-44ed-8026-8a0ce1f6c36f\") " pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:01.051148 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.050958    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5c797b08-a314-44ed-8026-8a0ce1f6c36f-host-cni-bin\") pod \"ovnkube-node-hww8q\" (UID: \"5c797b08-a314-44ed-8026-8a0ce1f6c36f\") " pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:01.051148 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.050896    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c2f75412-4413-45d1-aa82-919a5206efae-host-var-lib-cni-multus\") pod \"multus-tt68t\" (UID: \"c2f75412-4413-45d1-aa82-919a5206efae\") " pod="openshift-multus/multus-tt68t"
Nov 02 02:47:01.051148 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.050999    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bb7ec0c6-6988-45e6-b578-1b68b3a51d9d-socket-dir\") pod \"aws-ebs-csi-driver-node-zlplc\" (UID: \"bb7ec0c6-6988-45e6-b578-1b68b3a51d9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zlplc"
Nov 02 02:47:01.051148 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.051049    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bb7ec0c6-6988-45e6-b578-1b68b3a51d9d-etc-selinux\") pod \"aws-ebs-csi-driver-node-zlplc\" (UID: \"bb7ec0c6-6988-45e6-b578-1b68b3a51d9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zlplc"
Nov 02 02:47:01.051148 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.051078    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e400bad6-59c3-4c43-aed0-40c731c967db-kubelet-config\") pod \"global-pull-secret-syncer-dxlld\" (UID: \"e400bad6-59c3-4c43-aed0-40c731c967db\") " pod="kube-system/global-pull-secret-syncer-dxlld"
Nov 02 02:47:01.051148 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.051121    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e400bad6-59c3-4c43-aed0-40c731c967db-dbus\") pod \"global-pull-secret-syncer-dxlld\" (UID: \"e400bad6-59c3-4c43-aed0-40c731c967db\") " pod="kube-system/global-pull-secret-syncer-dxlld"
Nov 02 02:47:01.051148 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.051146    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c2f75412-4413-45d1-aa82-919a5206efae-etc-kubernetes\") pod \"multus-tt68t\" (UID: \"c2f75412-4413-45d1-aa82-919a5206efae\") " pod="openshift-multus/multus-tt68t"
Nov 02 02:47:01.051598 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.051167    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bb7ec0c6-6988-45e6-b578-1b68b3a51d9d-etc-selinux\") pod \"aws-ebs-csi-driver-node-zlplc\" (UID: \"bb7ec0c6-6988-45e6-b578-1b68b3a51d9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zlplc"
Nov 02 02:47:01.051598 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.051176    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bb7ec0c6-6988-45e6-b578-1b68b3a51d9d-socket-dir\") pod \"aws-ebs-csi-driver-node-zlplc\" (UID: \"bb7ec0c6-6988-45e6-b578-1b68b3a51d9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zlplc"
Nov 02 02:47:01.051598 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.051171    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a81017e4-d705-438e-833b-eedd6de0edde-system-cni-dir\") pod \"multus-additional-cni-plugins-69j4f\" (UID: \"a81017e4-d705-438e-833b-eedd6de0edde\") " pod="openshift-multus/multus-additional-cni-plugins-69j4f"
Nov 02 02:47:01.051598 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.051210    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e400bad6-59c3-4c43-aed0-40c731c967db-kubelet-config\") pod \"global-pull-secret-syncer-dxlld\" (UID: \"e400bad6-59c3-4c43-aed0-40c731c967db\") " pod="kube-system/global-pull-secret-syncer-dxlld"
Nov 02 02:47:01.051598 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.051216    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/94fb7403-b6ef-4d42-be18-7c45b5c10692-etc-sysctl-conf\") pod \"tuned-7qpcf\" (UID: \"94fb7403-b6ef-4d42-be18-7c45b5c10692\") " pod="openshift-cluster-node-tuning-operator/tuned-7qpcf"
Nov 02 02:47:01.051598 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.051230    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a81017e4-d705-438e-833b-eedd6de0edde-system-cni-dir\") pod \"multus-additional-cni-plugins-69j4f\" (UID: \"a81017e4-d705-438e-833b-eedd6de0edde\") " pod="openshift-multus/multus-additional-cni-plugins-69j4f"
Nov 02 02:47:01.051598 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.051248    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/94fb7403-b6ef-4d42-be18-7c45b5c10692-var-lib-kubelet\") pod \"tuned-7qpcf\" (UID: \"94fb7403-b6ef-4d42-be18-7c45b5c10692\") " pod="openshift-cluster-node-tuning-operator/tuned-7qpcf"
Nov 02 02:47:01.051598 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.051284    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c2f75412-4413-45d1-aa82-919a5206efae-etc-kubernetes\") pod \"multus-tt68t\" (UID: \"c2f75412-4413-45d1-aa82-919a5206efae\") " pod="openshift-multus/multus-tt68t"
Nov 02 02:47:01.051598 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.051291    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5c797b08-a314-44ed-8026-8a0ce1f6c36f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hww8q\" (UID: \"5c797b08-a314-44ed-8026-8a0ce1f6c36f\") " pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:01.051598 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.051333    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5c797b08-a314-44ed-8026-8a0ce1f6c36f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hww8q\" (UID: \"5c797b08-a314-44ed-8026-8a0ce1f6c36f\") " pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:01.051598 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.051336    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/94fb7403-b6ef-4d42-be18-7c45b5c10692-etc-sysctl-conf\") pod \"tuned-7qpcf\" (UID: \"94fb7403-b6ef-4d42-be18-7c45b5c10692\") " pod="openshift-cluster-node-tuning-operator/tuned-7qpcf"
Nov 02 02:47:01.051598 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.051324    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a81017e4-d705-438e-833b-eedd6de0edde-cnibin\") pod \"multus-additional-cni-plugins-69j4f\" (UID: \"a81017e4-d705-438e-833b-eedd6de0edde\") " pod="openshift-multus/multus-additional-cni-plugins-69j4f"
Nov 02 02:47:01.051598 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.051376    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a81017e4-d705-438e-833b-eedd6de0edde-cni-binary-copy\") pod \"multus-additional-cni-plugins-69j4f\" (UID: \"a81017e4-d705-438e-833b-eedd6de0edde\") " pod="openshift-multus/multus-additional-cni-plugins-69j4f"
Nov 02 02:47:01.051598 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.051381    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/94fb7403-b6ef-4d42-be18-7c45b5c10692-var-lib-kubelet\") pod \"tuned-7qpcf\" (UID: \"94fb7403-b6ef-4d42-be18-7c45b5c10692\") " pod="openshift-cluster-node-tuning-operator/tuned-7qpcf"
Nov 02 02:47:01.051598 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.051388    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e400bad6-59c3-4c43-aed0-40c731c967db-dbus\") pod \"global-pull-secret-syncer-dxlld\" (UID: \"e400bad6-59c3-4c43-aed0-40c731c967db\") " pod="kube-system/global-pull-secret-syncer-dxlld"
Nov 02 02:47:01.051598 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.051401    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a81017e4-d705-438e-833b-eedd6de0edde-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-69j4f\" (UID: \"a81017e4-d705-438e-833b-eedd6de0edde\") " pod="openshift-multus/multus-additional-cni-plugins-69j4f"
Nov 02 02:47:01.051598 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.051378    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a81017e4-d705-438e-833b-eedd6de0edde-cnibin\") pod \"multus-additional-cni-plugins-69j4f\" (UID: \"a81017e4-d705-438e-833b-eedd6de0edde\") " pod="openshift-multus/multus-additional-cni-plugins-69j4f"
Nov 02 02:47:01.052498 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.051429    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c2f75412-4413-45d1-aa82-919a5206efae-hostroot\") pod \"multus-tt68t\" (UID: \"c2f75412-4413-45d1-aa82-919a5206efae\") " pod="openshift-multus/multus-tt68t"
Nov 02 02:47:01.052498 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.051404    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c2f75412-4413-45d1-aa82-919a5206efae-hostroot\") pod \"multus-tt68t\" (UID: \"c2f75412-4413-45d1-aa82-919a5206efae\") " pod="openshift-multus/multus-tt68t"
Nov 02 02:47:01.052498 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.051463    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/94fb7403-b6ef-4d42-be18-7c45b5c10692-etc-kubernetes\") pod \"tuned-7qpcf\" (UID: \"94fb7403-b6ef-4d42-be18-7c45b5c10692\") " pod="openshift-cluster-node-tuning-operator/tuned-7qpcf"
Nov 02 02:47:01.052498 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.051486    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5c797b08-a314-44ed-8026-8a0ce1f6c36f-run-systemd\") pod \"ovnkube-node-hww8q\" (UID: \"5c797b08-a314-44ed-8026-8a0ce1f6c36f\") " pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:01.052498 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.051502    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5c797b08-a314-44ed-8026-8a0ce1f6c36f-env-overrides\") pod \"ovnkube-node-hww8q\" (UID: \"5c797b08-a314-44ed-8026-8a0ce1f6c36f\") " pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:01.052498 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.051526    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-984cg\" (UniqueName: \"kubernetes.io/projected/086e3504-d2fe-43ad-bc5e-dca409a4e695-kube-api-access-984cg\") pod \"network-metrics-daemon-zkjqm\" (UID: \"086e3504-d2fe-43ad-bc5e-dca409a4e695\") " pod="openshift-multus/network-metrics-daemon-zkjqm"
Nov 02 02:47:01.052498 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.051546    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/94fb7403-b6ef-4d42-be18-7c45b5c10692-etc-kubernetes\") pod \"tuned-7qpcf\" (UID: \"94fb7403-b6ef-4d42-be18-7c45b5c10692\") " pod="openshift-cluster-node-tuning-operator/tuned-7qpcf"
Nov 02 02:47:01.052498 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.051552    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c2f75412-4413-45d1-aa82-919a5206efae-multus-socket-dir-parent\") pod \"multus-tt68t\" (UID: \"c2f75412-4413-45d1-aa82-919a5206efae\") " pod="openshift-multus/multus-tt68t"
Nov 02 02:47:01.052498 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.051552    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5c797b08-a314-44ed-8026-8a0ce1f6c36f-run-systemd\") pod \"ovnkube-node-hww8q\" (UID: \"5c797b08-a314-44ed-8026-8a0ce1f6c36f\") " pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:01.052498 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.051590    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e701e783-5b43-45d7-b88b-83fd3c4af13b-host\") pod \"node-ca-kkqzw\" (UID: \"e701e783-5b43-45d7-b88b-83fd3c4af13b\") " pod="openshift-image-registry/node-ca-kkqzw"
Nov 02 02:47:01.052498 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.051606    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c2f75412-4413-45d1-aa82-919a5206efae-multus-socket-dir-parent\") pod \"multus-tt68t\" (UID: \"c2f75412-4413-45d1-aa82-919a5206efae\") " pod="openshift-multus/multus-tt68t"
Nov 02 02:47:01.052498 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.051619    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f2071fbb-7d2f-49d4-9366-34332e71c6bc-host-slash\") pod \"iptables-alerter-8sxd5\" (UID: \"f2071fbb-7d2f-49d4-9366-34332e71c6bc\") " pod="openshift-network-operator/iptables-alerter-8sxd5"
Nov 02 02:47:01.052498 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.051645    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4fec139d-da70-4cf6-8730-b578df3bebcc-hosts-file\") pod \"node-resolver-sztr6\" (UID: \"4fec139d-da70-4cf6-8730-b578df3bebcc\") " pod="openshift-dns/node-resolver-sztr6"
Nov 02 02:47:01.052498 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.051661    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e701e783-5b43-45d7-b88b-83fd3c4af13b-host\") pod \"node-ca-kkqzw\" (UID: \"e701e783-5b43-45d7-b88b-83fd3c4af13b\") " pod="openshift-image-registry/node-ca-kkqzw"
Nov 02 02:47:01.052498 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.051669    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bb7ec0c6-6988-45e6-b578-1b68b3a51d9d-device-dir\") pod \"aws-ebs-csi-driver-node-zlplc\" (UID: \"bb7ec0c6-6988-45e6-b578-1b68b3a51d9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zlplc"
Nov 02 02:47:01.052498 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.051725    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c2f75412-4413-45d1-aa82-919a5206efae-host-var-lib-kubelet\") pod \"multus-tt68t\" (UID: \"c2f75412-4413-45d1-aa82-919a5206efae\") " pod="openshift-multus/multus-tt68t"
Nov 02 02:47:01.052498 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.051730    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f2071fbb-7d2f-49d4-9366-34332e71c6bc-host-slash\") pod \"iptables-alerter-8sxd5\" (UID: \"f2071fbb-7d2f-49d4-9366-34332e71c6bc\") " pod="openshift-network-operator/iptables-alerter-8sxd5"
Nov 02 02:47:01.052498 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.051748    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4fec139d-da70-4cf6-8730-b578df3bebcc-hosts-file\") pod \"node-resolver-sztr6\" (UID: \"4fec139d-da70-4cf6-8730-b578df3bebcc\") " pod="openshift-dns/node-resolver-sztr6"
Nov 02 02:47:01.053424 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.051762    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c2f75412-4413-45d1-aa82-919a5206efae-multus-conf-dir\") pod \"multus-tt68t\" (UID: \"c2f75412-4413-45d1-aa82-919a5206efae\") " pod="openshift-multus/multus-tt68t"
Nov 02 02:47:01.053424 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.051778    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c2f75412-4413-45d1-aa82-919a5206efae-host-var-lib-kubelet\") pod \"multus-tt68t\" (UID: \"c2f75412-4413-45d1-aa82-919a5206efae\") " pod="openshift-multus/multus-tt68t"
Nov 02 02:47:01.053424 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.051790    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c2f75412-4413-45d1-aa82-919a5206efae-host-run-multus-certs\") pod \"multus-tt68t\" (UID: \"c2f75412-4413-45d1-aa82-919a5206efae\") " pod="openshift-multus/multus-tt68t"
Nov 02 02:47:01.053424 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.051828    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c2f75412-4413-45d1-aa82-919a5206efae-multus-conf-dir\") pod \"multus-tt68t\" (UID: \"c2f75412-4413-45d1-aa82-919a5206efae\") " pod="openshift-multus/multus-tt68t"
Nov 02 02:47:01.053424 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.051830    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c2f75412-4413-45d1-aa82-919a5206efae-host-run-multus-certs\") pod \"multus-tt68t\" (UID: \"c2f75412-4413-45d1-aa82-919a5206efae\") " pod="openshift-multus/multus-tt68t"
Nov 02 02:47:01.053424 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.051840    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bb7ec0c6-6988-45e6-b578-1b68b3a51d9d-device-dir\") pod \"aws-ebs-csi-driver-node-zlplc\" (UID: \"bb7ec0c6-6988-45e6-b578-1b68b3a51d9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zlplc"
Nov 02 02:47:01.053424 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.051854    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5c797b08-a314-44ed-8026-8a0ce1f6c36f-systemd-units\") pod \"ovnkube-node-hww8q\" (UID: \"5c797b08-a314-44ed-8026-8a0ce1f6c36f\") " pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:01.053424 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.051881    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a81017e4-d705-438e-833b-eedd6de0edde-cni-binary-copy\") pod \"multus-additional-cni-plugins-69j4f\" (UID: \"a81017e4-d705-438e-833b-eedd6de0edde\") " pod="openshift-multus/multus-additional-cni-plugins-69j4f"
Nov 02 02:47:01.053424 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.051889    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5c797b08-a314-44ed-8026-8a0ce1f6c36f-host-slash\") pod \"ovnkube-node-hww8q\" (UID: \"5c797b08-a314-44ed-8026-8a0ce1f6c36f\") " pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:01.053424 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.051916    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5c797b08-a314-44ed-8026-8a0ce1f6c36f-log-socket\") pod \"ovnkube-node-hww8q\" (UID: \"5c797b08-a314-44ed-8026-8a0ce1f6c36f\") " pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:01.053424 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.051931    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5c797b08-a314-44ed-8026-8a0ce1f6c36f-systemd-units\") pod \"ovnkube-node-hww8q\" (UID: \"5c797b08-a314-44ed-8026-8a0ce1f6c36f\") " pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:01.053424 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.051953    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5c797b08-a314-44ed-8026-8a0ce1f6c36f-host-slash\") pod \"ovnkube-node-hww8q\" (UID: \"5c797b08-a314-44ed-8026-8a0ce1f6c36f\") " pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:01.053424 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.051972    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5c797b08-a314-44ed-8026-8a0ce1f6c36f-env-overrides\") pod \"ovnkube-node-hww8q\" (UID: \"5c797b08-a314-44ed-8026-8a0ce1f6c36f\") " pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:01.053424 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.051979    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5c797b08-a314-44ed-8026-8a0ce1f6c36f-host-run-ovn-kubernetes\") pod \"ovnkube-node-hww8q\" (UID: \"5c797b08-a314-44ed-8026-8a0ce1f6c36f\") " pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:01.053424 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.052008    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5c797b08-a314-44ed-8026-8a0ce1f6c36f-log-socket\") pod \"ovnkube-node-hww8q\" (UID: \"5c797b08-a314-44ed-8026-8a0ce1f6c36f\") " pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:01.053424 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.052012    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bb7ec0c6-6988-45e6-b578-1b68b3a51d9d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zlplc\" (UID: \"bb7ec0c6-6988-45e6-b578-1b68b3a51d9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zlplc"
Nov 02 02:47:01.053424 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.052025    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5c797b08-a314-44ed-8026-8a0ce1f6c36f-host-run-ovn-kubernetes\") pod \"ovnkube-node-hww8q\" (UID: \"5c797b08-a314-44ed-8026-8a0ce1f6c36f\") " pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:01.053424 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.052040    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5c797b08-a314-44ed-8026-8a0ce1f6c36f-run-openvswitch\") pod \"ovnkube-node-hww8q\" (UID: \"5c797b08-a314-44ed-8026-8a0ce1f6c36f\") " pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:01.054210 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.052058    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bb7ec0c6-6988-45e6-b578-1b68b3a51d9d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zlplc\" (UID: \"bb7ec0c6-6988-45e6-b578-1b68b3a51d9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zlplc"
Nov 02 02:47:01.054210 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.052084    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5c797b08-a314-44ed-8026-8a0ce1f6c36f-node-log\") pod \"ovnkube-node-hww8q\" (UID: \"5c797b08-a314-44ed-8026-8a0ce1f6c36f\") " pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:01.054210 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.052108    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bb7ec0c6-6988-45e6-b578-1b68b3a51d9d-sys-fs\") pod \"aws-ebs-csi-driver-node-zlplc\" (UID: \"bb7ec0c6-6988-45e6-b578-1b68b3a51d9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zlplc"
Nov 02 02:47:01.054210 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.052111    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5c797b08-a314-44ed-8026-8a0ce1f6c36f-run-openvswitch\") pod \"ovnkube-node-hww8q\" (UID: \"5c797b08-a314-44ed-8026-8a0ce1f6c36f\") " pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:01.054210 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.052134    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ln99j\" (UniqueName: \"kubernetes.io/projected/bb7ec0c6-6988-45e6-b578-1b68b3a51d9d-kube-api-access-ln99j\") pod \"aws-ebs-csi-driver-node-zlplc\" (UID: \"bb7ec0c6-6988-45e6-b578-1b68b3a51d9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zlplc"
Nov 02 02:47:01.054210 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.052138    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5c797b08-a314-44ed-8026-8a0ce1f6c36f-node-log\") pod \"ovnkube-node-hww8q\" (UID: \"5c797b08-a314-44ed-8026-8a0ce1f6c36f\") " pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:01.054210 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.052178    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rrzn\" (UniqueName: \"kubernetes.io/projected/44f31bc6-bd01-4f16-ad6e-0d1ec90bfb4f-kube-api-access-7rrzn\") pod \"network-check-target-47f7x\" (UID: \"44f31bc6-bd01-4f16-ad6e-0d1ec90bfb4f\") " pod="openshift-network-diagnostics/network-check-target-47f7x"
Nov 02 02:47:01.054210 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.052182    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bb7ec0c6-6988-45e6-b578-1b68b3a51d9d-sys-fs\") pod \"aws-ebs-csi-driver-node-zlplc\" (UID: \"bb7ec0c6-6988-45e6-b578-1b68b3a51d9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zlplc"
Nov 02 02:47:01.054210 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.052204    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hclh\" (UniqueName: \"kubernetes.io/projected/c2f75412-4413-45d1-aa82-919a5206efae-kube-api-access-5hclh\") pod \"multus-tt68t\" (UID: \"c2f75412-4413-45d1-aa82-919a5206efae\") " pod="openshift-multus/multus-tt68t"
Nov 02 02:47:01.054210 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.052231    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/94fb7403-b6ef-4d42-be18-7c45b5c10692-etc-sysconfig\") pod \"tuned-7qpcf\" (UID: \"94fb7403-b6ef-4d42-be18-7c45b5c10692\") " pod="openshift-cluster-node-tuning-operator/tuned-7qpcf"
Nov 02 02:47:01.054210 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.052253    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/94fb7403-b6ef-4d42-be18-7c45b5c10692-etc-sysctl-d\") pod \"tuned-7qpcf\" (UID: \"94fb7403-b6ef-4d42-be18-7c45b5c10692\") " pod="openshift-cluster-node-tuning-operator/tuned-7qpcf"
Nov 02 02:47:01.054210 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.052278    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqtjv\" (UniqueName: \"kubernetes.io/projected/94fb7403-b6ef-4d42-be18-7c45b5c10692-kube-api-access-cqtjv\") pod \"tuned-7qpcf\" (UID: \"94fb7403-b6ef-4d42-be18-7c45b5c10692\") " pod="openshift-cluster-node-tuning-operator/tuned-7qpcf"
Nov 02 02:47:01.054210 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.052304    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzwzx\" (UniqueName: \"kubernetes.io/projected/4fec139d-da70-4cf6-8730-b578df3bebcc-kube-api-access-kzwzx\") pod \"node-resolver-sztr6\" (UID: \"4fec139d-da70-4cf6-8730-b578df3bebcc\") " pod="openshift-dns/node-resolver-sztr6"
Nov 02 02:47:01.054210 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.052327    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5c797b08-a314-44ed-8026-8a0ce1f6c36f-host-kubelet\") pod \"ovnkube-node-hww8q\" (UID: \"5c797b08-a314-44ed-8026-8a0ce1f6c36f\") " pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:01.054210 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.052374    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c078a1ee-3179-45c9-9d3b-4d4a46f0c42f-konnectivity-ca\") pod \"konnectivity-agent-z8sq6\" (UID: \"c078a1ee-3179-45c9-9d3b-4d4a46f0c42f\") " pod="kube-system/konnectivity-agent-z8sq6"
Nov 02 02:47:01.054210 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.052400    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/94fb7403-b6ef-4d42-be18-7c45b5c10692-sys\") pod \"tuned-7qpcf\" (UID: \"94fb7403-b6ef-4d42-be18-7c45b5c10692\") " pod="openshift-cluster-node-tuning-operator/tuned-7qpcf"
Nov 02 02:47:01.054210 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.052409    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/94fb7403-b6ef-4d42-be18-7c45b5c10692-etc-sysconfig\") pod \"tuned-7qpcf\" (UID: \"94fb7403-b6ef-4d42-be18-7c45b5c10692\") " pod="openshift-cluster-node-tuning-operator/tuned-7qpcf"
Nov 02 02:47:01.055055 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.052425    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5c797b08-a314-44ed-8026-8a0ce1f6c36f-ovnkube-config\") pod \"ovnkube-node-hww8q\" (UID: \"5c797b08-a314-44ed-8026-8a0ce1f6c36f\") " pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:01.055055 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.052455    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5c797b08-a314-44ed-8026-8a0ce1f6c36f-ovn-node-metrics-cert\") pod \"ovnkube-node-hww8q\" (UID: \"5c797b08-a314-44ed-8026-8a0ce1f6c36f\") " pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:01.055055 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.052460    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5c797b08-a314-44ed-8026-8a0ce1f6c36f-host-kubelet\") pod \"ovnkube-node-hww8q\" (UID: \"5c797b08-a314-44ed-8026-8a0ce1f6c36f\") " pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:01.055055 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.052480    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c2f75412-4413-45d1-aa82-919a5206efae-host-run-k8s-cni-cncf-io\") pod \"multus-tt68t\" (UID: \"c2f75412-4413-45d1-aa82-919a5206efae\") " pod="openshift-multus/multus-tt68t"
Nov 02 02:47:01.055055 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.052505    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c2f75412-4413-45d1-aa82-919a5206efae-host-run-netns\") pod \"multus-tt68t\" (UID: \"c2f75412-4413-45d1-aa82-919a5206efae\") " pod="openshift-multus/multus-tt68t"
Nov 02 02:47:01.055055 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.052565    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c2f75412-4413-45d1-aa82-919a5206efae-host-run-netns\") pod \"multus-tt68t\" (UID: \"c2f75412-4413-45d1-aa82-919a5206efae\") " pod="openshift-multus/multus-tt68t"
Nov 02 02:47:01.055055 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.052570    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/94fb7403-b6ef-4d42-be18-7c45b5c10692-sys\") pod \"tuned-7qpcf\" (UID: \"94fb7403-b6ef-4d42-be18-7c45b5c10692\") " pod="openshift-cluster-node-tuning-operator/tuned-7qpcf"
Nov 02 02:47:01.055055 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.052608    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c2f75412-4413-45d1-aa82-919a5206efae-host-var-lib-cni-bin\") pod \"multus-tt68t\" (UID: \"c2f75412-4413-45d1-aa82-919a5206efae\") " pod="openshift-multus/multus-tt68t"
Nov 02 02:47:01.055055 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.052623    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c2f75412-4413-45d1-aa82-919a5206efae-host-run-k8s-cni-cncf-io\") pod \"multus-tt68t\" (UID: \"c2f75412-4413-45d1-aa82-919a5206efae\") " pod="openshift-multus/multus-tt68t"
Nov 02 02:47:01.055055 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.052640    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f2071fbb-7d2f-49d4-9366-34332e71c6bc-iptables-alerter-script\") pod \"iptables-alerter-8sxd5\" (UID: \"f2071fbb-7d2f-49d4-9366-34332e71c6bc\") " pod="openshift-network-operator/iptables-alerter-8sxd5"
Nov 02 02:47:01.055055 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.052667    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c2f75412-4413-45d1-aa82-919a5206efae-host-var-lib-cni-bin\") pod \"multus-tt68t\" (UID: \"c2f75412-4413-45d1-aa82-919a5206efae\") " pod="openshift-multus/multus-tt68t"
Nov 02 02:47:01.055055 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.052672    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e400bad6-59c3-4c43-aed0-40c731c967db-original-pull-secret\") pod \"global-pull-secret-syncer-dxlld\" (UID: \"e400bad6-59c3-4c43-aed0-40c731c967db\") " pod="kube-system/global-pull-secret-syncer-dxlld"
Nov 02 02:47:01.055055 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.052690    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/94fb7403-b6ef-4d42-be18-7c45b5c10692-etc-sysctl-d\") pod \"tuned-7qpcf\" (UID: \"94fb7403-b6ef-4d42-be18-7c45b5c10692\") " pod="openshift-cluster-node-tuning-operator/tuned-7qpcf"
Nov 02 02:47:01.055055 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.052702    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5c797b08-a314-44ed-8026-8a0ce1f6c36f-var-lib-openvswitch\") pod \"ovnkube-node-hww8q\" (UID: \"5c797b08-a314-44ed-8026-8a0ce1f6c36f\") " pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:01.055055 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.052738    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a81017e4-d705-438e-833b-eedd6de0edde-tuning-conf-dir\") pod \"multus-additional-cni-plugins-69j4f\" (UID: \"a81017e4-d705-438e-833b-eedd6de0edde\") " pod="openshift-multus/multus-additional-cni-plugins-69j4f"
Nov 02 02:47:01.055055 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:01.052766    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 02:47:01.055055 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.052770    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/94fb7403-b6ef-4d42-be18-7c45b5c10692-host\") pod \"tuned-7qpcf\" (UID: \"94fb7403-b6ef-4d42-be18-7c45b5c10692\") " pod="openshift-cluster-node-tuning-operator/tuned-7qpcf"
Nov 02 02:47:01.055055 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.052800    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/94fb7403-b6ef-4d42-be18-7c45b5c10692-etc-tuned\") pod \"tuned-7qpcf\" (UID: \"94fb7403-b6ef-4d42-be18-7c45b5c10692\") " pod="openshift-cluster-node-tuning-operator/tuned-7qpcf"
Nov 02 02:47:01.055937 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:01.052840    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e400bad6-59c3-4c43-aed0-40c731c967db-original-pull-secret podName:e400bad6-59c3-4c43-aed0-40c731c967db nodeName:}" failed. No retries permitted until 2026-11-02 02:47:01.552807259 +0000 UTC m=+2.267875411 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e400bad6-59c3-4c43-aed0-40c731c967db-original-pull-secret") pod "global-pull-secret-syncer-dxlld" (UID: "e400bad6-59c3-4c43-aed0-40c731c967db") : object "kube-system"/"original-pull-secret" not registered
Nov 02 02:47:01.055937 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.052866    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fec139d-da70-4cf6-8730-b578df3bebcc-tmp-dir\") pod \"node-resolver-sztr6\" (UID: \"4fec139d-da70-4cf6-8730-b578df3bebcc\") " pod="openshift-dns/node-resolver-sztr6"
Nov 02 02:47:01.055937 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.052885    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c2f75412-4413-45d1-aa82-919a5206efae-system-cni-dir\") pod \"multus-tt68t\" (UID: \"c2f75412-4413-45d1-aa82-919a5206efae\") " pod="openshift-multus/multus-tt68t"
Nov 02 02:47:01.055937 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.052903    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/94fb7403-b6ef-4d42-be18-7c45b5c10692-etc-systemd\") pod \"tuned-7qpcf\" (UID: \"94fb7403-b6ef-4d42-be18-7c45b5c10692\") " pod="openshift-cluster-node-tuning-operator/tuned-7qpcf"
Nov 02 02:47:01.055937 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.052919    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/94fb7403-b6ef-4d42-be18-7c45b5c10692-run\") pod \"tuned-7qpcf\" (UID: \"94fb7403-b6ef-4d42-be18-7c45b5c10692\") " pod="openshift-cluster-node-tuning-operator/tuned-7qpcf"
Nov 02 02:47:01.055937 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.052934    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5c797b08-a314-44ed-8026-8a0ce1f6c36f-host-cni-netd\") pod \"ovnkube-node-hww8q\" (UID: \"5c797b08-a314-44ed-8026-8a0ce1f6c36f\") " pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:01.055937 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.052954    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/086e3504-d2fe-43ad-bc5e-dca409a4e695-metrics-certs\") pod \"network-metrics-daemon-zkjqm\" (UID: \"086e3504-d2fe-43ad-bc5e-dca409a4e695\") " pod="openshift-multus/network-metrics-daemon-zkjqm"
Nov 02 02:47:01.055937 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.052972    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a81017e4-d705-438e-833b-eedd6de0edde-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-69j4f\" (UID: \"a81017e4-d705-438e-833b-eedd6de0edde\") " pod="openshift-multus/multus-additional-cni-plugins-69j4f"
Nov 02 02:47:01.055937 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.052988    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c2f75412-4413-45d1-aa82-919a5206efae-multus-cni-dir\") pod \"multus-tt68t\" (UID: \"c2f75412-4413-45d1-aa82-919a5206efae\") " pod="openshift-multus/multus-tt68t"
Nov 02 02:47:01.055937 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.053006    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c2f75412-4413-45d1-aa82-919a5206efae-multus-daemon-config\") pod \"multus-tt68t\" (UID: \"c2f75412-4413-45d1-aa82-919a5206efae\") " pod="openshift-multus/multus-tt68t"
Nov 02 02:47:01.055937 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.053040    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqsvj\" (UniqueName: \"kubernetes.io/projected/e701e783-5b43-45d7-b88b-83fd3c4af13b-kube-api-access-kqsvj\") pod \"node-ca-kkqzw\" (UID: \"e701e783-5b43-45d7-b88b-83fd3c4af13b\") " pod="openshift-image-registry/node-ca-kkqzw"
Nov 02 02:47:01.055937 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.053070    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69zxc\" (UniqueName: \"kubernetes.io/projected/f2071fbb-7d2f-49d4-9366-34332e71c6bc-kube-api-access-69zxc\") pod \"iptables-alerter-8sxd5\" (UID: \"f2071fbb-7d2f-49d4-9366-34332e71c6bc\") " pod="openshift-network-operator/iptables-alerter-8sxd5"
Nov 02 02:47:01.055937 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.053086    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/94fb7403-b6ef-4d42-be18-7c45b5c10692-lib-modules\") pod \"tuned-7qpcf\" (UID: \"94fb7403-b6ef-4d42-be18-7c45b5c10692\") " pod="openshift-cluster-node-tuning-operator/tuned-7qpcf"
Nov 02 02:47:01.055937 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.053101    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5c797b08-a314-44ed-8026-8a0ce1f6c36f-run-ovn\") pod \"ovnkube-node-hww8q\" (UID: \"5c797b08-a314-44ed-8026-8a0ce1f6c36f\") " pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:01.055937 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.053125    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68r2v\" (UniqueName: \"kubernetes.io/projected/a81017e4-d705-438e-833b-eedd6de0edde-kube-api-access-68r2v\") pod \"multus-additional-cni-plugins-69j4f\" (UID: \"a81017e4-d705-438e-833b-eedd6de0edde\") " pod="openshift-multus/multus-additional-cni-plugins-69j4f"
Nov 02 02:47:01.055937 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.053143    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c2f75412-4413-45d1-aa82-919a5206efae-cni-binary-copy\") pod \"multus-tt68t\" (UID: \"c2f75412-4413-45d1-aa82-919a5206efae\") " pod="openshift-multus/multus-tt68t"
Nov 02 02:47:01.055937 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.053131    2577 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 02 02:47:01.056815 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.053161    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e701e783-5b43-45d7-b88b-83fd3c4af13b-serviceca\") pod \"node-ca-kkqzw\" (UID: \"e701e783-5b43-45d7-b88b-83fd3c4af13b\") " pod="openshift-image-registry/node-ca-kkqzw"
Nov 02 02:47:01.056815 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.053178    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5c797b08-a314-44ed-8026-8a0ce1f6c36f-ovnkube-script-lib\") pod \"ovnkube-node-hww8q\" (UID: \"5c797b08-a314-44ed-8026-8a0ce1f6c36f\") " pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:01.056815 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.053228    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c2f75412-4413-45d1-aa82-919a5206efae-multus-cni-dir\") pod \"multus-tt68t\" (UID: \"c2f75412-4413-45d1-aa82-919a5206efae\") " pod="openshift-multus/multus-tt68t"
Nov 02 02:47:01.056815 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.053240    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c078a1ee-3179-45c9-9d3b-4d4a46f0c42f-konnectivity-ca\") pod \"konnectivity-agent-z8sq6\" (UID: \"c078a1ee-3179-45c9-9d3b-4d4a46f0c42f\") " pod="kube-system/konnectivity-agent-z8sq6"
Nov 02 02:47:01.056815 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.053261    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/94fb7403-b6ef-4d42-be18-7c45b5c10692-etc-modprobe-d\") pod \"tuned-7qpcf\" (UID: \"94fb7403-b6ef-4d42-be18-7c45b5c10692\") " pod="openshift-cluster-node-tuning-operator/tuned-7qpcf"
Nov 02 02:47:01.056815 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.053290    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5dxg\" (UniqueName: \"kubernetes.io/projected/5c797b08-a314-44ed-8026-8a0ce1f6c36f-kube-api-access-b5dxg\") pod \"ovnkube-node-hww8q\" (UID: \"5c797b08-a314-44ed-8026-8a0ce1f6c36f\") " pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:01.056815 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.053312    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c2f75412-4413-45d1-aa82-919a5206efae-cnibin\") pod \"multus-tt68t\" (UID: \"c2f75412-4413-45d1-aa82-919a5206efae\") " pod="openshift-multus/multus-tt68t"
Nov 02 02:47:01.056815 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.053333    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c078a1ee-3179-45c9-9d3b-4d4a46f0c42f-agent-certs\") pod \"konnectivity-agent-z8sq6\" (UID: \"c078a1ee-3179-45c9-9d3b-4d4a46f0c42f\") " pod="kube-system/konnectivity-agent-z8sq6"
Nov 02 02:47:01.056815 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.053345    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5c797b08-a314-44ed-8026-8a0ce1f6c36f-var-lib-openvswitch\") pod \"ovnkube-node-hww8q\" (UID: \"5c797b08-a314-44ed-8026-8a0ce1f6c36f\") " pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:01.056815 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.053352    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/94fb7403-b6ef-4d42-be18-7c45b5c10692-tmp\") pod \"tuned-7qpcf\" (UID: \"94fb7403-b6ef-4d42-be18-7c45b5c10692\") " pod="openshift-cluster-node-tuning-operator/tuned-7qpcf"
Nov 02 02:47:01.056815 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.053406    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5c797b08-a314-44ed-8026-8a0ce1f6c36f-etc-openvswitch\") pod \"ovnkube-node-hww8q\" (UID: \"5c797b08-a314-44ed-8026-8a0ce1f6c36f\") " pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:01.056815 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.053475    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5c797b08-a314-44ed-8026-8a0ce1f6c36f-etc-openvswitch\") pod \"ovnkube-node-hww8q\" (UID: \"5c797b08-a314-44ed-8026-8a0ce1f6c36f\") " pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:01.056815 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.053615    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a81017e4-d705-438e-833b-eedd6de0edde-tuning-conf-dir\") pod \"multus-additional-cni-plugins-69j4f\" (UID: \"a81017e4-d705-438e-833b-eedd6de0edde\") " pod="openshift-multus/multus-additional-cni-plugins-69j4f"
Nov 02 02:47:01.056815 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.053670    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/94fb7403-b6ef-4d42-be18-7c45b5c10692-host\") pod \"tuned-7qpcf\" (UID: \"94fb7403-b6ef-4d42-be18-7c45b5c10692\") " pod="openshift-cluster-node-tuning-operator/tuned-7qpcf"
Nov 02 02:47:01.056815 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.053704    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5c797b08-a314-44ed-8026-8a0ce1f6c36f-ovnkube-script-lib\") pod \"ovnkube-node-hww8q\" (UID: \"5c797b08-a314-44ed-8026-8a0ce1f6c36f\") " pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:01.056815 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.053750    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5c797b08-a314-44ed-8026-8a0ce1f6c36f-run-ovn\") pod \"ovnkube-node-hww8q\" (UID: \"5c797b08-a314-44ed-8026-8a0ce1f6c36f\") " pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:01.056815 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.053789    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/94fb7403-b6ef-4d42-be18-7c45b5c10692-lib-modules\") pod \"tuned-7qpcf\" (UID: \"94fb7403-b6ef-4d42-be18-7c45b5c10692\") " pod="openshift-cluster-node-tuning-operator/tuned-7qpcf"
Nov 02 02:47:01.056815 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.053824    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f2071fbb-7d2f-49d4-9366-34332e71c6bc-iptables-alerter-script\") pod \"iptables-alerter-8sxd5\" (UID: \"f2071fbb-7d2f-49d4-9366-34332e71c6bc\") " pod="openshift-network-operator/iptables-alerter-8sxd5"
Nov 02 02:47:01.057521 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.054162    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5c797b08-a314-44ed-8026-8a0ce1f6c36f-host-cni-netd\") pod \"ovnkube-node-hww8q\" (UID: \"5c797b08-a314-44ed-8026-8a0ce1f6c36f\") " pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:01.057521 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.054388    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c2f75412-4413-45d1-aa82-919a5206efae-multus-daemon-config\") pod \"multus-tt68t\" (UID: \"c2f75412-4413-45d1-aa82-919a5206efae\") " pod="openshift-multus/multus-tt68t"
Nov 02 02:47:01.057521 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.054402    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e701e783-5b43-45d7-b88b-83fd3c4af13b-serviceca\") pod \"node-ca-kkqzw\" (UID: \"e701e783-5b43-45d7-b88b-83fd3c4af13b\") " pod="openshift-image-registry/node-ca-kkqzw"
Nov 02 02:47:01.057521 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.054431    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c2f75412-4413-45d1-aa82-919a5206efae-cni-binary-copy\") pod \"multus-tt68t\" (UID: \"c2f75412-4413-45d1-aa82-919a5206efae\") " pod="openshift-multus/multus-tt68t"
Nov 02 02:47:01.057521 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.054482    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fec139d-da70-4cf6-8730-b578df3bebcc-tmp-dir\") pod \"node-resolver-sztr6\" (UID: \"4fec139d-da70-4cf6-8730-b578df3bebcc\") " pod="openshift-dns/node-resolver-sztr6"
Nov 02 02:47:01.057521 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.054551    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/94fb7403-b6ef-4d42-be18-7c45b5c10692-etc-modprobe-d\") pod \"tuned-7qpcf\" (UID: \"94fb7403-b6ef-4d42-be18-7c45b5c10692\") " pod="openshift-cluster-node-tuning-operator/tuned-7qpcf"
Nov 02 02:47:01.057521 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.054567    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/94fb7403-b6ef-4d42-be18-7c45b5c10692-etc-systemd\") pod \"tuned-7qpcf\" (UID: \"94fb7403-b6ef-4d42-be18-7c45b5c10692\") " pod="openshift-cluster-node-tuning-operator/tuned-7qpcf"
Nov 02 02:47:01.057521 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.054613    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/94fb7403-b6ef-4d42-be18-7c45b5c10692-run\") pod \"tuned-7qpcf\" (UID: \"94fb7403-b6ef-4d42-be18-7c45b5c10692\") " pod="openshift-cluster-node-tuning-operator/tuned-7qpcf"
Nov 02 02:47:01.057521 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.054567    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c2f75412-4413-45d1-aa82-919a5206efae-system-cni-dir\") pod \"multus-tt68t\" (UID: \"c2f75412-4413-45d1-aa82-919a5206efae\") " pod="openshift-multus/multus-tt68t"
Nov 02 02:47:01.057521 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:01.054643    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 02:47:01.057521 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.054670    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c2f75412-4413-45d1-aa82-919a5206efae-cnibin\") pod \"multus-tt68t\" (UID: \"c2f75412-4413-45d1-aa82-919a5206efae\") " pod="openshift-multus/multus-tt68t"
Nov 02 02:47:01.057521 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:01.054688    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/086e3504-d2fe-43ad-bc5e-dca409a4e695-metrics-certs podName:086e3504-d2fe-43ad-bc5e-dca409a4e695 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:01.55467353 +0000 UTC m=+2.269741651 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/086e3504-d2fe-43ad-bc5e-dca409a4e695-metrics-certs") pod "network-metrics-daemon-zkjqm" (UID: "086e3504-d2fe-43ad-bc5e-dca409a4e695") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 02:47:01.057521 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.055572    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a81017e4-d705-438e-833b-eedd6de0edde-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-69j4f\" (UID: \"a81017e4-d705-438e-833b-eedd6de0edde\") " pod="openshift-multus/multus-additional-cni-plugins-69j4f"
Nov 02 02:47:01.057521 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.056553    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5c797b08-a314-44ed-8026-8a0ce1f6c36f-ovnkube-config\") pod \"ovnkube-node-hww8q\" (UID: \"5c797b08-a314-44ed-8026-8a0ce1f6c36f\") " pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:01.057521 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.057421    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/94fb7403-b6ef-4d42-be18-7c45b5c10692-tmp\") pod \"tuned-7qpcf\" (UID: \"94fb7403-b6ef-4d42-be18-7c45b5c10692\") " pod="openshift-cluster-node-tuning-operator/tuned-7qpcf"
Nov 02 02:47:01.058019 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.057581    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/94fb7403-b6ef-4d42-be18-7c45b5c10692-etc-tuned\") pod \"tuned-7qpcf\" (UID: \"94fb7403-b6ef-4d42-be18-7c45b5c10692\") " pod="openshift-cluster-node-tuning-operator/tuned-7qpcf"
Nov 02 02:47:01.058358 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.058329    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c078a1ee-3179-45c9-9d3b-4d4a46f0c42f-agent-certs\") pod \"konnectivity-agent-z8sq6\" (UID: \"c078a1ee-3179-45c9-9d3b-4d4a46f0c42f\") " pod="kube-system/konnectivity-agent-z8sq6"
Nov 02 02:47:01.059501 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.059470    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5c797b08-a314-44ed-8026-8a0ce1f6c36f-ovn-node-metrics-cert\") pod \"ovnkube-node-hww8q\" (UID: \"5c797b08-a314-44ed-8026-8a0ce1f6c36f\") " pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:01.062216 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.062200    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-z8sq6"
Nov 02 02:47:01.110835 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:01.110812    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 02:47:01.110835 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:01.110834    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 02:47:01.110952 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:01.110847    2577 projected.go:196] Error preparing data for projected volume kube-api-access-7rrzn for pod openshift-network-diagnostics/network-check-target-47f7x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 02:47:01.110952 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:01.110892    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/44f31bc6-bd01-4f16-ad6e-0d1ec90bfb4f-kube-api-access-7rrzn podName:44f31bc6-bd01-4f16-ad6e-0d1ec90bfb4f nodeName:}" failed. No retries permitted until 2026-11-02 02:47:01.610880186 +0000 UTC m=+2.325948302 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-7rrzn" (UniqueName: "kubernetes.io/projected/44f31bc6-bd01-4f16-ad6e-0d1ec90bfb4f-kube-api-access-7rrzn") pod "network-check-target-47f7x" (UID: "44f31bc6-bd01-4f16-ad6e-0d1ec90bfb4f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 02:47:01.114699 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.113582    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-984cg\" (UniqueName: \"kubernetes.io/projected/086e3504-d2fe-43ad-bc5e-dca409a4e695-kube-api-access-984cg\") pod \"network-metrics-daemon-zkjqm\" (UID: \"086e3504-d2fe-43ad-bc5e-dca409a4e695\") " pod="openshift-multus/network-metrics-daemon-zkjqm"
Nov 02 02:47:01.114699 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.113940    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hclh\" (UniqueName: \"kubernetes.io/projected/c2f75412-4413-45d1-aa82-919a5206efae-kube-api-access-5hclh\") pod \"multus-tt68t\" (UID: \"c2f75412-4413-45d1-aa82-919a5206efae\") " pod="openshift-multus/multus-tt68t"
Nov 02 02:47:01.114699 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.114027    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ln99j\" (UniqueName: \"kubernetes.io/projected/bb7ec0c6-6988-45e6-b578-1b68b3a51d9d-kube-api-access-ln99j\") pod \"aws-ebs-csi-driver-node-zlplc\" (UID: \"bb7ec0c6-6988-45e6-b578-1b68b3a51d9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zlplc"
Nov 02 02:47:01.114699 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.114522    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68r2v\" (UniqueName: \"kubernetes.io/projected/a81017e4-d705-438e-833b-eedd6de0edde-kube-api-access-68r2v\") pod \"multus-additional-cni-plugins-69j4f\" (UID: \"a81017e4-d705-438e-833b-eedd6de0edde\") " pod="openshift-multus/multus-additional-cni-plugins-69j4f"
Nov 02 02:47:01.114928 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.114828    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqtjv\" (UniqueName: \"kubernetes.io/projected/94fb7403-b6ef-4d42-be18-7c45b5c10692-kube-api-access-cqtjv\") pod \"tuned-7qpcf\" (UID: \"94fb7403-b6ef-4d42-be18-7c45b5c10692\") " pod="openshift-cluster-node-tuning-operator/tuned-7qpcf"
Nov 02 02:47:01.115063 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.115028    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqsvj\" (UniqueName: \"kubernetes.io/projected/e701e783-5b43-45d7-b88b-83fd3c4af13b-kube-api-access-kqsvj\") pod \"node-ca-kkqzw\" (UID: \"e701e783-5b43-45d7-b88b-83fd3c4af13b\") " pod="openshift-image-registry/node-ca-kkqzw"
Nov 02 02:47:01.115127 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.115110    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69zxc\" (UniqueName: \"kubernetes.io/projected/f2071fbb-7d2f-49d4-9366-34332e71c6bc-kube-api-access-69zxc\") pod \"iptables-alerter-8sxd5\" (UID: \"f2071fbb-7d2f-49d4-9366-34332e71c6bc\") " pod="openshift-network-operator/iptables-alerter-8sxd5"
Nov 02 02:47:01.115406 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.115390    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzwzx\" (UniqueName: \"kubernetes.io/projected/4fec139d-da70-4cf6-8730-b578df3bebcc-kube-api-access-kzwzx\") pod \"node-resolver-sztr6\" (UID: \"4fec139d-da70-4cf6-8730-b578df3bebcc\") " pod="openshift-dns/node-resolver-sztr6"
Nov 02 02:47:01.115949 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.115928    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5dxg\" (UniqueName: \"kubernetes.io/projected/5c797b08-a314-44ed-8026-8a0ce1f6c36f-kube-api-access-b5dxg\") pod \"ovnkube-node-hww8q\" (UID: \"5c797b08-a314-44ed-8026-8a0ce1f6c36f\") " pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:01.231523 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.231499    2577 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 02:47:01.239119 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.239096    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zlplc"
Nov 02 02:47:01.254731 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.254705    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-8sxd5"
Nov 02 02:47:01.260886 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.260865    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7qpcf"
Nov 02 02:47:01.285337 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.285313    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-sztr6"
Nov 02 02:47:01.309924 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.309896    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-69j4f"
Nov 02 02:47:01.312201 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.312176    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-tt68t"
Nov 02 02:47:01.346487 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.346466    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:01.354319 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.354300    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-kkqzw"
Nov 02 02:47:01.383420 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:47:01.383394    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod562e4cfd5885e6864e00b6a01f2722f3.slice/crio-2ed71fa3aff1cd56e1226d56dbe15d965fc44bee13d69b0cb27b95c8d7e6e2e4 WatchSource:0}: Error finding container 2ed71fa3aff1cd56e1226d56dbe15d965fc44bee13d69b0cb27b95c8d7e6e2e4: Status 404 returned error can't find the container with id 2ed71fa3aff1cd56e1226d56dbe15d965fc44bee13d69b0cb27b95c8d7e6e2e4
Nov 02 02:47:01.383911 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:47:01.383894    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c797b08_a314_44ed_8026_8a0ce1f6c36f.slice/crio-6c4a6e1a0af8dafae71eada0be4fe75fc1bf6debb30f4ad76458efe124709def WatchSource:0}: Error finding container 6c4a6e1a0af8dafae71eada0be4fe75fc1bf6debb30f4ad76458efe124709def: Status 404 returned error can't find the container with id 6c4a6e1a0af8dafae71eada0be4fe75fc1bf6debb30f4ad76458efe124709def
Nov 02 02:47:01.384981 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:47:01.384954    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod950fb6772a40be3e22d676c446372284.slice/crio-85800559c3175d90ec68a3a769b750066c0d2a153ad689d1fca92627ac2c36da WatchSource:0}: Error finding container 85800559c3175d90ec68a3a769b750066c0d2a153ad689d1fca92627ac2c36da: Status 404 returned error can't find the container with id 85800559c3175d90ec68a3a769b750066c0d2a153ad689d1fca92627ac2c36da
Nov 02 02:47:01.385362 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:47:01.385339    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode701e783_5b43_45d7_b88b_83fd3c4af13b.slice/crio-8bba6a60c331d769827f2a8ce74bc143ee555be0b74c181fe482ca8b000ffce0 WatchSource:0}: Error finding container 8bba6a60c331d769827f2a8ce74bc143ee555be0b74c181fe482ca8b000ffce0: Status 404 returned error can't find the container with id 8bba6a60c331d769827f2a8ce74bc143ee555be0b74c181fe482ca8b000ffce0
Nov 02 02:47:01.388418 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.388401    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 02:47:01.557768 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.557677    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e400bad6-59c3-4c43-aed0-40c731c967db-original-pull-secret\") pod \"global-pull-secret-syncer-dxlld\" (UID: \"e400bad6-59c3-4c43-aed0-40c731c967db\") " pod="kube-system/global-pull-secret-syncer-dxlld"
Nov 02 02:47:01.557768 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.557722    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/086e3504-d2fe-43ad-bc5e-dca409a4e695-metrics-certs\") pod \"network-metrics-daemon-zkjqm\" (UID: \"086e3504-d2fe-43ad-bc5e-dca409a4e695\") " pod="openshift-multus/network-metrics-daemon-zkjqm"
Nov 02 02:47:01.557947 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:01.557846    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 02:47:01.557947 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:01.557852    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 02:47:01.557947 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:01.557915    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/086e3504-d2fe-43ad-bc5e-dca409a4e695-metrics-certs podName:086e3504-d2fe-43ad-bc5e-dca409a4e695 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:02.557898052 +0000 UTC m=+3.272966172 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/086e3504-d2fe-43ad-bc5e-dca409a4e695-metrics-certs") pod "network-metrics-daemon-zkjqm" (UID: "086e3504-d2fe-43ad-bc5e-dca409a4e695") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 02:47:01.557947 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:01.557929    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e400bad6-59c3-4c43-aed0-40c731c967db-original-pull-secret podName:e400bad6-59c3-4c43-aed0-40c731c967db nodeName:}" failed. No retries permitted until 2026-11-02 02:47:02.557922721 +0000 UTC m=+3.272990839 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e400bad6-59c3-4c43-aed0-40c731c967db-original-pull-secret") pod "global-pull-secret-syncer-dxlld" (UID: "e400bad6-59c3-4c43-aed0-40c731c967db") : object "kube-system"/"original-pull-secret" not registered
Nov 02 02:47:01.658223 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.658191    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rrzn\" (UniqueName: \"kubernetes.io/projected/44f31bc6-bd01-4f16-ad6e-0d1ec90bfb4f-kube-api-access-7rrzn\") pod \"network-check-target-47f7x\" (UID: \"44f31bc6-bd01-4f16-ad6e-0d1ec90bfb4f\") " pod="openshift-network-diagnostics/network-check-target-47f7x"
Nov 02 02:47:01.658849 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:01.658333    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 02:47:01.658849 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:01.658347    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 02:47:01.658849 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:01.658356    2577 projected.go:196] Error preparing data for projected volume kube-api-access-7rrzn for pod openshift-network-diagnostics/network-check-target-47f7x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 02:47:01.658849 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:01.658406    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/44f31bc6-bd01-4f16-ad6e-0d1ec90bfb4f-kube-api-access-7rrzn podName:44f31bc6-bd01-4f16-ad6e-0d1ec90bfb4f nodeName:}" failed. No retries permitted until 2026-11-02 02:47:02.658389922 +0000 UTC m=+3.373458054 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-7rrzn" (UniqueName: "kubernetes.io/projected/44f31bc6-bd01-4f16-ad6e-0d1ec90bfb4f-kube-api-access-7rrzn") pod "network-check-target-47f7x" (UID: "44f31bc6-bd01-4f16-ad6e-0d1ec90bfb4f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 02:47:01.716763 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.716714    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-163.us-west-1.compute.internal" event={"ID":"562e4cfd5885e6864e00b6a01f2722f3","Type":"ContainerStarted","Data":"2ed71fa3aff1cd56e1226d56dbe15d965fc44bee13d69b0cb27b95c8d7e6e2e4"}
Nov 02 02:47:01.718670 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.718642    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-163.us-west-1.compute.internal" event={"ID":"950fb6772a40be3e22d676c446372284","Type":"ContainerStarted","Data":"85800559c3175d90ec68a3a769b750066c0d2a153ad689d1fca92627ac2c36da"}
Nov 02 02:47:01.719677 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.719652    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-kkqzw" event={"ID":"e701e783-5b43-45d7-b88b-83fd3c4af13b","Type":"ContainerStarted","Data":"8bba6a60c331d769827f2a8ce74bc143ee555be0b74c181fe482ca8b000ffce0"}
Nov 02 02:47:01.720809 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:01.720785    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hww8q" event={"ID":"5c797b08-a314-44ed-8026-8a0ce1f6c36f","Type":"ContainerStarted","Data":"6c4a6e1a0af8dafae71eada0be4fe75fc1bf6debb30f4ad76458efe124709def"}
Nov 02 02:47:01.946677 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:47:01.946647    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc078a1ee_3179_45c9_9d3b_4d4a46f0c42f.slice/crio-1da77b7031557bd08d918b40a19724d095489096dfeca493347206c5d46660b2 WatchSource:0}: Error finding container 1da77b7031557bd08d918b40a19724d095489096dfeca493347206c5d46660b2: Status 404 returned error can't find the container with id 1da77b7031557bd08d918b40a19724d095489096dfeca493347206c5d46660b2
Nov 02 02:47:02.018304 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:02.018252    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 02:42:01 +0000 UTC" deadline="2028-07-03 02:43:45.457815521 +0000 UTC"
Nov 02 02:47:02.018304 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:02.018298    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14615h56m43.439519381s"
Nov 02 02:47:02.127127 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:47:02.127080    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94fb7403_b6ef_4d42_be18_7c45b5c10692.slice/crio-b0a80cef2bcd29515ea73e4558ba7076ea0209e1f42afa8f717f35b2249c6f27 WatchSource:0}: Error finding container b0a80cef2bcd29515ea73e4558ba7076ea0209e1f42afa8f717f35b2249c6f27: Status 404 returned error can't find the container with id b0a80cef2bcd29515ea73e4558ba7076ea0209e1f42afa8f717f35b2249c6f27
Nov 02 02:47:02.133633 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:47:02.133604    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb7ec0c6_6988_45e6_b578_1b68b3a51d9d.slice/crio-0b526f283f6b1ad6466bec01d5bd889c9cbdfdd4d0cc5c47caca5e20a11608de WatchSource:0}: Error finding container 0b526f283f6b1ad6466bec01d5bd889c9cbdfdd4d0cc5c47caca5e20a11608de: Status 404 returned error can't find the container with id 0b526f283f6b1ad6466bec01d5bd889c9cbdfdd4d0cc5c47caca5e20a11608de
Nov 02 02:47:02.193434 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:47:02.193391    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2071fbb_7d2f_49d4_9366_34332e71c6bc.slice/crio-7e010f75b5a70d368f92ff60896e9c094444b76136d38786a4f0f7024c99eb40 WatchSource:0}: Error finding container 7e010f75b5a70d368f92ff60896e9c094444b76136d38786a4f0f7024c99eb40: Status 404 returned error can't find the container with id 7e010f75b5a70d368f92ff60896e9c094444b76136d38786a4f0f7024c99eb40
Nov 02 02:47:02.237077 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:47:02.237028    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda81017e4_d705_438e_833b_eedd6de0edde.slice/crio-cb72bec312a628bd7dcc49627f8e12decbd44378a591f97d2c08eac503ed16c1 WatchSource:0}: Error finding container cb72bec312a628bd7dcc49627f8e12decbd44378a591f97d2c08eac503ed16c1: Status 404 returned error can't find the container with id cb72bec312a628bd7dcc49627f8e12decbd44378a591f97d2c08eac503ed16c1
Nov 02 02:47:02.247942 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:47:02.247913    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fec139d_da70_4cf6_8730_b578df3bebcc.slice/crio-1d66575f918ec32ab11a4b703f24211fac7d25eed28e5dbe273c52b07dfc60af WatchSource:0}: Error finding container 1d66575f918ec32ab11a4b703f24211fac7d25eed28e5dbe273c52b07dfc60af: Status 404 returned error can't find the container with id 1d66575f918ec32ab11a4b703f24211fac7d25eed28e5dbe273c52b07dfc60af
Nov 02 02:47:02.565619 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:02.565545    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/086e3504-d2fe-43ad-bc5e-dca409a4e695-metrics-certs\") pod \"network-metrics-daemon-zkjqm\" (UID: \"086e3504-d2fe-43ad-bc5e-dca409a4e695\") " pod="openshift-multus/network-metrics-daemon-zkjqm"
Nov 02 02:47:02.565819 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:02.565654    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e400bad6-59c3-4c43-aed0-40c731c967db-original-pull-secret\") pod \"global-pull-secret-syncer-dxlld\" (UID: \"e400bad6-59c3-4c43-aed0-40c731c967db\") " pod="kube-system/global-pull-secret-syncer-dxlld"
Nov 02 02:47:02.565819 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:02.565779    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 02:47:02.565947 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:02.565841    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e400bad6-59c3-4c43-aed0-40c731c967db-original-pull-secret podName:e400bad6-59c3-4c43-aed0-40c731c967db nodeName:}" failed. No retries permitted until 2026-11-02 02:47:04.565824596 +0000 UTC m=+5.280892717 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e400bad6-59c3-4c43-aed0-40c731c967db-original-pull-secret") pod "global-pull-secret-syncer-dxlld" (UID: "e400bad6-59c3-4c43-aed0-40c731c967db") : object "kube-system"/"original-pull-secret" not registered
Nov 02 02:47:02.565947 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:02.565906    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 02:47:02.565947 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:02.565938    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/086e3504-d2fe-43ad-bc5e-dca409a4e695-metrics-certs podName:086e3504-d2fe-43ad-bc5e-dca409a4e695 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:04.565926607 +0000 UTC m=+5.280994729 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/086e3504-d2fe-43ad-bc5e-dca409a4e695-metrics-certs") pod "network-metrics-daemon-zkjqm" (UID: "086e3504-d2fe-43ad-bc5e-dca409a4e695") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 02:47:02.667612 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:02.666951    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rrzn\" (UniqueName: \"kubernetes.io/projected/44f31bc6-bd01-4f16-ad6e-0d1ec90bfb4f-kube-api-access-7rrzn\") pod \"network-check-target-47f7x\" (UID: \"44f31bc6-bd01-4f16-ad6e-0d1ec90bfb4f\") " pod="openshift-network-diagnostics/network-check-target-47f7x"
Nov 02 02:47:02.667612 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:02.667395    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 02:47:02.667612 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:02.667417    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 02:47:02.667612 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:02.667428    2577 projected.go:196] Error preparing data for projected volume kube-api-access-7rrzn for pod openshift-network-diagnostics/network-check-target-47f7x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 02:47:02.667612 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:02.667476    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/44f31bc6-bd01-4f16-ad6e-0d1ec90bfb4f-kube-api-access-7rrzn podName:44f31bc6-bd01-4f16-ad6e-0d1ec90bfb4f nodeName:}" failed. No retries permitted until 2026-11-02 02:47:04.667462676 +0000 UTC m=+5.382530792 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-7rrzn" (UniqueName: "kubernetes.io/projected/44f31bc6-bd01-4f16-ad6e-0d1ec90bfb4f-kube-api-access-7rrzn") pod "network-check-target-47f7x" (UID: "44f31bc6-bd01-4f16-ad6e-0d1ec90bfb4f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 02:47:02.712570 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:02.712531    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-47f7x"
Nov 02 02:47:02.712757 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:02.712679    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-47f7x" podUID="44f31bc6-bd01-4f16-ad6e-0d1ec90bfb4f"
Nov 02 02:47:02.712808 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:02.712770    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dxlld"
Nov 02 02:47:02.712888 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:02.712848    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dxlld" podUID="e400bad6-59c3-4c43-aed0-40c731c967db"
Nov 02 02:47:02.712955 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:02.712921    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zkjqm"
Nov 02 02:47:02.713048 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:02.713012    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zkjqm" podUID="086e3504-d2fe-43ad-bc5e-dca409a4e695"
Nov 02 02:47:02.726789 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:02.726749    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7qpcf" event={"ID":"94fb7403-b6ef-4d42-be18-7c45b5c10692","Type":"ContainerStarted","Data":"b0a80cef2bcd29515ea73e4558ba7076ea0209e1f42afa8f717f35b2249c6f27"}
Nov 02 02:47:02.729829 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:02.729795    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-sztr6" event={"ID":"4fec139d-da70-4cf6-8730-b578df3bebcc","Type":"ContainerStarted","Data":"1d66575f918ec32ab11a4b703f24211fac7d25eed28e5dbe273c52b07dfc60af"}
Nov 02 02:47:02.731736 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:02.731708    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-tt68t" event={"ID":"c2f75412-4413-45d1-aa82-919a5206efae","Type":"ContainerStarted","Data":"dd9841fb555d9d098d25de67000c4d6821e0d1c1602302738c214238cada1c41"}
Nov 02 02:47:02.733819 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:02.733785    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zlplc" event={"ID":"bb7ec0c6-6988-45e6-b578-1b68b3a51d9d","Type":"ContainerStarted","Data":"0b526f283f6b1ad6466bec01d5bd889c9cbdfdd4d0cc5c47caca5e20a11608de"}
Nov 02 02:47:02.735505 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:02.735484    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-z8sq6" event={"ID":"c078a1ee-3179-45c9-9d3b-4d4a46f0c42f","Type":"ContainerStarted","Data":"1da77b7031557bd08d918b40a19724d095489096dfeca493347206c5d46660b2"}
Nov 02 02:47:02.739865 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:02.739837    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-69j4f" event={"ID":"a81017e4-d705-438e-833b-eedd6de0edde","Type":"ContainerStarted","Data":"cb72bec312a628bd7dcc49627f8e12decbd44378a591f97d2c08eac503ed16c1"}
Nov 02 02:47:02.741762 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:02.741735    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-8sxd5" event={"ID":"f2071fbb-7d2f-49d4-9366-34332e71c6bc","Type":"ContainerStarted","Data":"7e010f75b5a70d368f92ff60896e9c094444b76136d38786a4f0f7024c99eb40"}
Nov 02 02:47:03.020644 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:03.019448    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 02:42:01 +0000 UTC" deadline="2028-08-05 06:15:17.52963609 +0000 UTC"
Nov 02 02:47:03.020644 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:03.019511    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15411h28m14.510129855s"
Nov 02 02:47:04.584271 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:04.583808    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e400bad6-59c3-4c43-aed0-40c731c967db-original-pull-secret\") pod \"global-pull-secret-syncer-dxlld\" (UID: \"e400bad6-59c3-4c43-aed0-40c731c967db\") " pod="kube-system/global-pull-secret-syncer-dxlld"
Nov 02 02:47:04.584271 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:04.583861    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/086e3504-d2fe-43ad-bc5e-dca409a4e695-metrics-certs\") pod \"network-metrics-daemon-zkjqm\" (UID: \"086e3504-d2fe-43ad-bc5e-dca409a4e695\") " pod="openshift-multus/network-metrics-daemon-zkjqm"
Nov 02 02:47:04.584271 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:04.584007    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 02:47:04.584271 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:04.584090    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/086e3504-d2fe-43ad-bc5e-dca409a4e695-metrics-certs podName:086e3504-d2fe-43ad-bc5e-dca409a4e695 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:08.584070278 +0000 UTC m=+9.299138409 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/086e3504-d2fe-43ad-bc5e-dca409a4e695-metrics-certs") pod "network-metrics-daemon-zkjqm" (UID: "086e3504-d2fe-43ad-bc5e-dca409a4e695") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 02:47:04.584271 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:04.584187    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 02:47:04.584271 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:04.584229    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e400bad6-59c3-4c43-aed0-40c731c967db-original-pull-secret podName:e400bad6-59c3-4c43-aed0-40c731c967db nodeName:}" failed. No retries permitted until 2026-11-02 02:47:08.584215328 +0000 UTC m=+9.299283447 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e400bad6-59c3-4c43-aed0-40c731c967db-original-pull-secret") pod "global-pull-secret-syncer-dxlld" (UID: "e400bad6-59c3-4c43-aed0-40c731c967db") : object "kube-system"/"original-pull-secret" not registered
Nov 02 02:47:04.685234 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:04.685199    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rrzn\" (UniqueName: \"kubernetes.io/projected/44f31bc6-bd01-4f16-ad6e-0d1ec90bfb4f-kube-api-access-7rrzn\") pod \"network-check-target-47f7x\" (UID: \"44f31bc6-bd01-4f16-ad6e-0d1ec90bfb4f\") " pod="openshift-network-diagnostics/network-check-target-47f7x"
Nov 02 02:47:04.685410 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:04.685380    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 02:47:04.685410 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:04.685400    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 02:47:04.685531 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:04.685413    2577 projected.go:196] Error preparing data for projected volume kube-api-access-7rrzn for pod openshift-network-diagnostics/network-check-target-47f7x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 02:47:04.685531 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:04.685471    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/44f31bc6-bd01-4f16-ad6e-0d1ec90bfb4f-kube-api-access-7rrzn podName:44f31bc6-bd01-4f16-ad6e-0d1ec90bfb4f nodeName:}" failed. No retries permitted until 2026-11-02 02:47:08.685454225 +0000 UTC m=+9.400522360 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-7rrzn" (UniqueName: "kubernetes.io/projected/44f31bc6-bd01-4f16-ad6e-0d1ec90bfb4f-kube-api-access-7rrzn") pod "network-check-target-47f7x" (UID: "44f31bc6-bd01-4f16-ad6e-0d1ec90bfb4f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 02:47:04.712963 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:04.712883    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-47f7x"
Nov 02 02:47:04.713135 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:04.713030    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-47f7x" podUID="44f31bc6-bd01-4f16-ad6e-0d1ec90bfb4f"
Nov 02 02:47:04.713135 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:04.713120    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dxlld"
Nov 02 02:47:04.713254 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:04.713197    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dxlld" podUID="e400bad6-59c3-4c43-aed0-40c731c967db"
Nov 02 02:47:04.713254 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:04.713248    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zkjqm"
Nov 02 02:47:04.713370 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:04.713336    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zkjqm" podUID="086e3504-d2fe-43ad-bc5e-dca409a4e695"
Nov 02 02:47:06.712565 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:06.712468    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-47f7x"
Nov 02 02:47:06.713032 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:06.712621    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-47f7x" podUID="44f31bc6-bd01-4f16-ad6e-0d1ec90bfb4f"
Nov 02 02:47:06.713032 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:06.712696    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dxlld"
Nov 02 02:47:06.713032 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:06.712768    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dxlld" podUID="e400bad6-59c3-4c43-aed0-40c731c967db"
Nov 02 02:47:06.713032 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:06.712902    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zkjqm"
Nov 02 02:47:06.713032 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:06.712989    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zkjqm" podUID="086e3504-d2fe-43ad-bc5e-dca409a4e695"
Nov 02 02:47:06.761295 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:06.761260    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-163.us-west-1.compute.internal" event={"ID":"562e4cfd5885e6864e00b6a01f2722f3","Type":"ContainerStarted","Data":"737e95dd26182223150ee9677fbc7f60f4030198402cbc59d5b1288d14df6aa3"}
Nov 02 02:47:06.764058 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:06.764009    2577 generic.go:356] "Generic (PLEG): container finished" podID="950fb6772a40be3e22d676c446372284" containerID="c8ea7ef81711de6aa7fae28c1bcb705f30c984de61919cd1d0eec71fe57a04f7" exitCode=0
Nov 02 02:47:06.764192 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:06.764107    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-163.us-west-1.compute.internal" event={"ID":"950fb6772a40be3e22d676c446372284","Type":"ContainerDied","Data":"c8ea7ef81711de6aa7fae28c1bcb705f30c984de61919cd1d0eec71fe57a04f7"}
Nov 02 02:47:06.767787 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:06.767759    2577 generic.go:356] "Generic (PLEG): container finished" podID="a81017e4-d705-438e-833b-eedd6de0edde" containerID="b0dc03625aa435cf373a4f1913630511ce02ed069f9416d6f78287d1c4049270" exitCode=0
Nov 02 02:47:06.767903 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:06.767826    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-69j4f" event={"ID":"a81017e4-d705-438e-833b-eedd6de0edde","Type":"ContainerDied","Data":"b0dc03625aa435cf373a4f1913630511ce02ed069f9416d6f78287d1c4049270"}
Nov 02 02:47:06.990033 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:06.989967    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-163.us-west-1.compute.internal" podStartSLOduration=6.989947874 podStartE2EDuration="6.989947874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 02:47:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:47:06.872833247 +0000 UTC m=+7.587901384" watchObservedRunningTime="2026-11-02 02:47:06.989947874 +0000 UTC m=+7.705016012"
Nov 02 02:47:07.788732 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:07.786189    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-163.us-west-1.compute.internal" event={"ID":"950fb6772a40be3e22d676c446372284","Type":"ContainerStarted","Data":"7a7d5b406ff4da9eb330b2cef992961555bdcd50227e029f325e53c0decce674"}
Nov 02 02:47:07.797153 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:07.797122    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-kkqzw" event={"ID":"e701e783-5b43-45d7-b88b-83fd3c4af13b","Type":"ContainerStarted","Data":"c11dc64b1f35bb3eeea9122af29804b4325338d45322d7d053f29a7a1b12102e"}
Nov 02 02:47:07.803157 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:07.803131    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-z8sq6" event={"ID":"c078a1ee-3179-45c9-9d3b-4d4a46f0c42f","Type":"ContainerStarted","Data":"cdf523673289d09d0c0424d77446ca5b6bd3cfe6267963920d0fbadde041e965"}
Nov 02 02:47:08.010716 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:08.010594    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-kkqzw" podStartSLOduration=2.167800029 podStartE2EDuration="8.010580755s" totalImagesPullingTime="5.842780726s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 02:47:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:47:08.010373475 +0000 UTC m=+8.725441612" watchObservedRunningTime="2026-11-02 02:47:08.010580755 +0000 UTC m=+8.725648870"
Nov 02 02:47:08.010891 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:08.010733    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-163.us-west-1.compute.internal" podStartSLOduration=8.010725579 podStartE2EDuration="8.010725579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 02:47:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:47:07.898233343 +0000 UTC m=+8.613301484" watchObservedRunningTime="2026-11-02 02:47:08.010725579 +0000 UTC m=+8.725793717"
Nov 02 02:47:08.622861 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:08.622830    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e400bad6-59c3-4c43-aed0-40c731c967db-original-pull-secret\") pod \"global-pull-secret-syncer-dxlld\" (UID: \"e400bad6-59c3-4c43-aed0-40c731c967db\") " pod="kube-system/global-pull-secret-syncer-dxlld"
Nov 02 02:47:08.623035 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:08.622880    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/086e3504-d2fe-43ad-bc5e-dca409a4e695-metrics-certs\") pod \"network-metrics-daemon-zkjqm\" (UID: \"086e3504-d2fe-43ad-bc5e-dca409a4e695\") " pod="openshift-multus/network-metrics-daemon-zkjqm"
Nov 02 02:47:08.623035 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:08.622993    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 02:47:08.623146 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:08.623063    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/086e3504-d2fe-43ad-bc5e-dca409a4e695-metrics-certs podName:086e3504-d2fe-43ad-bc5e-dca409a4e695 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:16.623045151 +0000 UTC m=+17.338113288 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/086e3504-d2fe-43ad-bc5e-dca409a4e695-metrics-certs") pod "network-metrics-daemon-zkjqm" (UID: "086e3504-d2fe-43ad-bc5e-dca409a4e695") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 02:47:08.623451 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:08.623432    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 02:47:08.623529 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:08.623488    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e400bad6-59c3-4c43-aed0-40c731c967db-original-pull-secret podName:e400bad6-59c3-4c43-aed0-40c731c967db nodeName:}" failed. No retries permitted until 2026-11-02 02:47:16.623473643 +0000 UTC m=+17.338541772 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e400bad6-59c3-4c43-aed0-40c731c967db-original-pull-secret") pod "global-pull-secret-syncer-dxlld" (UID: "e400bad6-59c3-4c43-aed0-40c731c967db") : object "kube-system"/"original-pull-secret" not registered
Nov 02 02:47:08.712809 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:08.712779    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-47f7x"
Nov 02 02:47:08.712936 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:08.712905    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-47f7x" podUID="44f31bc6-bd01-4f16-ad6e-0d1ec90bfb4f"
Nov 02 02:47:08.713087 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:08.712974    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dxlld"
Nov 02 02:47:08.713364 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:08.713338    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dxlld" podUID="e400bad6-59c3-4c43-aed0-40c731c967db"
Nov 02 02:47:08.713439 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:08.713426    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zkjqm"
Nov 02 02:47:08.713547 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:08.713527    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zkjqm" podUID="086e3504-d2fe-43ad-bc5e-dca409a4e695"
Nov 02 02:47:08.723673 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:08.723644    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rrzn\" (UniqueName: \"kubernetes.io/projected/44f31bc6-bd01-4f16-ad6e-0d1ec90bfb4f-kube-api-access-7rrzn\") pod \"network-check-target-47f7x\" (UID: \"44f31bc6-bd01-4f16-ad6e-0d1ec90bfb4f\") " pod="openshift-network-diagnostics/network-check-target-47f7x"
Nov 02 02:47:08.723842 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:08.723823    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 02:47:08.723909 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:08.723856    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 02:47:08.723909 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:08.723869    2577 projected.go:196] Error preparing data for projected volume kube-api-access-7rrzn for pod openshift-network-diagnostics/network-check-target-47f7x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 02:47:08.724004 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:08.723920    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/44f31bc6-bd01-4f16-ad6e-0d1ec90bfb4f-kube-api-access-7rrzn podName:44f31bc6-bd01-4f16-ad6e-0d1ec90bfb4f nodeName:}" failed. No retries permitted until 2026-11-02 02:47:16.723903675 +0000 UTC m=+17.438971812 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-7rrzn" (UniqueName: "kubernetes.io/projected/44f31bc6-bd01-4f16-ad6e-0d1ec90bfb4f-kube-api-access-7rrzn") pod "network-check-target-47f7x" (UID: "44f31bc6-bd01-4f16-ad6e-0d1ec90bfb4f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 02:47:08.807484 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:08.807447    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zlplc" event={"ID":"bb7ec0c6-6988-45e6-b578-1b68b3a51d9d","Type":"ContainerStarted","Data":"4a4ac2ea2273602629a2e38ed9fc6402b396790d6f7c55d9ab0e0e96280c0ae5"}
Nov 02 02:47:10.017907 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:10.017045    2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-z8sq6"
Nov 02 02:47:10.018392 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:10.018054    2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-z8sq6"
Nov 02 02:47:10.130463 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:10.130406    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-z8sq6" podStartSLOduration=5.20541676 podStartE2EDuration="10.130386565s" totalImagesPullingTime="4.924969805s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:47:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:47:08.122133738 +0000 UTC m=+8.837201874" watchObservedRunningTime="2026-11-02 02:47:10.130386565 +0000 UTC m=+10.845454705"
Nov 02 02:47:10.717279 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:10.716898    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-47f7x"
Nov 02 02:47:10.717279 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:10.717028    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-47f7x" podUID="44f31bc6-bd01-4f16-ad6e-0d1ec90bfb4f"
Nov 02 02:47:10.717279 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:10.717074    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zkjqm"
Nov 02 02:47:10.717279 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:10.717189    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zkjqm" podUID="086e3504-d2fe-43ad-bc5e-dca409a4e695"
Nov 02 02:47:10.717597 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:10.717001    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dxlld"
Nov 02 02:47:10.717655 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:10.717625    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dxlld" podUID="e400bad6-59c3-4c43-aed0-40c731c967db"
Nov 02 02:47:10.753576 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:10.753522    2577 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 02 02:47:10.822058 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:10.821563    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-sztr6" event={"ID":"4fec139d-da70-4cf6-8730-b578df3bebcc","Type":"ContainerStarted","Data":"94e37c25d7b8fe2698aa1ad43b3edcd08809a9aaf9b5a34556578cef98e50334"}
Nov 02 02:47:10.825824 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:10.824651    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-tt68t" event={"ID":"c2f75412-4413-45d1-aa82-919a5206efae","Type":"ContainerStarted","Data":"ee50a19cf06dbbc8d9e850a4e2ef7e2cf58caac42a47968572829c0cb658d9cf"}
Nov 02 02:47:10.833200 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:10.833116    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zlplc" event={"ID":"bb7ec0c6-6988-45e6-b578-1b68b3a51d9d","Type":"ContainerStarted","Data":"11dd76bc3c765713a7a21979582501b3606c6db60f648d5d744d46bc04fa89d0"}
Nov 02 02:47:11.046896 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:11.046632    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-sztr6" podStartSLOduration=2.684695922 podStartE2EDuration="11.04661434s" totalImagesPullingTime="8.361918418s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 02:47:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:47:10.932947311 +0000 UTC m=+11.648015447" watchObservedRunningTime="2026-11-02 02:47:11.04661434 +0000 UTC m=+11.761682479"
Nov 02 02:47:11.723094 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:11.722893    2577 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T02:47:10.753547546Z","UUID":"f17ae340-a866-44f1-a715-edee3ea2d9cd","Handler":null,"Name":"","Endpoint":""}
Nov 02 02:47:11.725033 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:11.725001    2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 02 02:47:11.725033 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:11.725038    2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 02 02:47:11.838104 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:11.836959    2577 generic.go:356] "Generic (PLEG): container finished" podID="a81017e4-d705-438e-833b-eedd6de0edde" containerID="f24f80957ce420ef3480bfde153c6bd24b0e4b39ac0ba29a059d95b5ad30f38d" exitCode=0
Nov 02 02:47:11.838104 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:11.837742    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-69j4f" event={"ID":"a81017e4-d705-438e-833b-eedd6de0edde","Type":"ContainerDied","Data":"f24f80957ce420ef3480bfde153c6bd24b0e4b39ac0ba29a059d95b5ad30f38d"}
Nov 02 02:47:11.953893 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:11.953800    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-tt68t" podStartSLOduration=3.5955668100000002 podStartE2EDuration="11.953781953s" totalImagesPullingTime="8.358215143s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 02:47:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:47:11.047458266 +0000 UTC m=+11.762526402" watchObservedRunningTime="2026-11-02 02:47:11.953781953 +0000 UTC m=+12.668850091"
Nov 02 02:47:12.208192 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:12.207185    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-z8sq6"
Nov 02 02:47:12.208192 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:12.207622    2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 02:47:12.208192 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:12.207880    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-z8sq6"
Nov 02 02:47:12.713279 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:12.713235    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-47f7x"
Nov 02 02:47:12.713279 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:12.713256    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dxlld"
Nov 02 02:47:12.713486 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:12.713267    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zkjqm"
Nov 02 02:47:12.713486 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:12.713363    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-47f7x" podUID="44f31bc6-bd01-4f16-ad6e-0d1ec90bfb4f"
Nov 02 02:47:12.713486 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:12.713422    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dxlld" podUID="e400bad6-59c3-4c43-aed0-40c731c967db"
Nov 02 02:47:12.713651 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:12.713533    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zkjqm" podUID="086e3504-d2fe-43ad-bc5e-dca409a4e695"
Nov 02 02:47:12.843970 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:12.843939    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zlplc" event={"ID":"bb7ec0c6-6988-45e6-b578-1b68b3a51d9d","Type":"ContainerStarted","Data":"d24d4c336795b142d7338a997dc272276c7459d7814000ade45c1566428ceddc"}
Nov 02 02:47:12.846071 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:12.846045    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-8sxd5" event={"ID":"f2071fbb-7d2f-49d4-9366-34332e71c6bc","Type":"ContainerStarted","Data":"b40e3ba6a6b83c93cb486dba56ec562198e1bee804de6f5d70d9bed08dcdd86b"}
Nov 02 02:47:12.957591 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:12.957536    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zlplc" podStartSLOduration=3.262808872 podStartE2EDuration="12.95752078s" totalImagesPullingTime="9.694711908s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 02:47:00 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:47:12.957383184 +0000 UTC m=+13.672451323" watchObservedRunningTime="2026-11-02 02:47:12.95752078 +0000 UTC m=+13.672588907"
Nov 02 02:47:13.069206 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:13.069156    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-8sxd5" podStartSLOduration=4.343678614 podStartE2EDuration="13.06914276s" totalImagesPullingTime="8.725464146s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 02:47:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:47:13.068977912 +0000 UTC m=+13.784046048" watchObservedRunningTime="2026-11-02 02:47:13.06914276 +0000 UTC m=+13.784210897"
Nov 02 02:47:13.850707 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:13.850669    2577 generic.go:356] "Generic (PLEG): container finished" podID="a81017e4-d705-438e-833b-eedd6de0edde" containerID="3cd13392d3a79feeedc18ebc385fc826fc62e5b93ff55e365b2f075ee0322611" exitCode=0
Nov 02 02:47:13.851512 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:13.850751    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-69j4f" event={"ID":"a81017e4-d705-438e-833b-eedd6de0edde","Type":"ContainerDied","Data":"3cd13392d3a79feeedc18ebc385fc826fc62e5b93ff55e365b2f075ee0322611"}
Nov 02 02:47:13.852420 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:13.852394    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7qpcf" event={"ID":"94fb7403-b6ef-4d42-be18-7c45b5c10692","Type":"ContainerStarted","Data":"3c4009263230f15738cdfa8a5d4c16b541f1913143c4460a3bacbad7da40136f"}
Nov 02 02:47:14.079438 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:14.079383    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-7qpcf" podStartSLOduration=3.089345614 podStartE2EDuration="14.079368497s" totalImagesPullingTime="10.990022883s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 02:47:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:47:14.079213922 +0000 UTC m=+14.794282059" watchObservedRunningTime="2026-11-02 02:47:14.079368497 +0000 UTC m=+14.794436633"
Nov 02 02:47:14.713113 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:14.713080    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zkjqm"
Nov 02 02:47:14.713113 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:14.713101    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-47f7x"
Nov 02 02:47:14.713318 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:14.713109    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dxlld"
Nov 02 02:47:14.713318 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:14.713190    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zkjqm" podUID="086e3504-d2fe-43ad-bc5e-dca409a4e695"
Nov 02 02:47:14.713318 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:14.713298    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-47f7x" podUID="44f31bc6-bd01-4f16-ad6e-0d1ec90bfb4f"
Nov 02 02:47:14.713478 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:14.713390    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dxlld" podUID="e400bad6-59c3-4c43-aed0-40c731c967db"
Nov 02 02:47:15.858472 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:15.858289    2577 generic.go:356] "Generic (PLEG): container finished" podID="a81017e4-d705-438e-833b-eedd6de0edde" containerID="a987233257be4a22cb3b47f7ec931f535bdb695368b4402526d64bb2310a1dd5" exitCode=0
Nov 02 02:47:15.858898 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:15.858374    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-69j4f" event={"ID":"a81017e4-d705-438e-833b-eedd6de0edde","Type":"ContainerDied","Data":"a987233257be4a22cb3b47f7ec931f535bdb695368b4402526d64bb2310a1dd5"}
Nov 02 02:47:16.687997 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:16.687970    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e400bad6-59c3-4c43-aed0-40c731c967db-original-pull-secret\") pod \"global-pull-secret-syncer-dxlld\" (UID: \"e400bad6-59c3-4c43-aed0-40c731c967db\") " pod="kube-system/global-pull-secret-syncer-dxlld"
Nov 02 02:47:16.688106 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:16.688005    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/086e3504-d2fe-43ad-bc5e-dca409a4e695-metrics-certs\") pod \"network-metrics-daemon-zkjqm\" (UID: \"086e3504-d2fe-43ad-bc5e-dca409a4e695\") " pod="openshift-multus/network-metrics-daemon-zkjqm"
Nov 02 02:47:16.688166 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:16.688117    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 02:47:16.688166 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:16.688120    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 02:47:16.688166 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:16.688163    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/086e3504-d2fe-43ad-bc5e-dca409a4e695-metrics-certs podName:086e3504-d2fe-43ad-bc5e-dca409a4e695 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:32.688149835 +0000 UTC m=+33.403217951 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/086e3504-d2fe-43ad-bc5e-dca409a4e695-metrics-certs") pod "network-metrics-daemon-zkjqm" (UID: "086e3504-d2fe-43ad-bc5e-dca409a4e695") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 02:47:16.688314 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:16.688179    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e400bad6-59c3-4c43-aed0-40c731c967db-original-pull-secret podName:e400bad6-59c3-4c43-aed0-40c731c967db nodeName:}" failed. No retries permitted until 2026-11-02 02:47:32.688170082 +0000 UTC m=+33.403238198 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e400bad6-59c3-4c43-aed0-40c731c967db-original-pull-secret") pod "global-pull-secret-syncer-dxlld" (UID: "e400bad6-59c3-4c43-aed0-40c731c967db") : object "kube-system"/"original-pull-secret" not registered
Nov 02 02:47:16.712587 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:16.712565    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zkjqm"
Nov 02 02:47:16.712689 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:16.712591    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dxlld"
Nov 02 02:47:16.712743 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:16.712691    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zkjqm" podUID="086e3504-d2fe-43ad-bc5e-dca409a4e695"
Nov 02 02:47:16.712743 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:16.712715    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-47f7x"
Nov 02 02:47:16.712825 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:16.712780    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-47f7x" podUID="44f31bc6-bd01-4f16-ad6e-0d1ec90bfb4f"
Nov 02 02:47:16.712914 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:16.712861    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dxlld" podUID="e400bad6-59c3-4c43-aed0-40c731c967db"
Nov 02 02:47:16.788586 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:16.788556    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rrzn\" (UniqueName: \"kubernetes.io/projected/44f31bc6-bd01-4f16-ad6e-0d1ec90bfb4f-kube-api-access-7rrzn\") pod \"network-check-target-47f7x\" (UID: \"44f31bc6-bd01-4f16-ad6e-0d1ec90bfb4f\") " pod="openshift-network-diagnostics/network-check-target-47f7x"
Nov 02 02:47:16.788781 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:16.788738    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 02:47:16.788781 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:16.788756    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 02:47:16.788781 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:16.788765    2577 projected.go:196] Error preparing data for projected volume kube-api-access-7rrzn for pod openshift-network-diagnostics/network-check-target-47f7x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 02:47:16.788919 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:16.788807    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/44f31bc6-bd01-4f16-ad6e-0d1ec90bfb4f-kube-api-access-7rrzn podName:44f31bc6-bd01-4f16-ad6e-0d1ec90bfb4f nodeName:}" failed. No retries permitted until 2026-11-02 02:47:32.788795863 +0000 UTC m=+33.503863993 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-7rrzn" (UniqueName: "kubernetes.io/projected/44f31bc6-bd01-4f16-ad6e-0d1ec90bfb4f-kube-api-access-7rrzn") pod "network-check-target-47f7x" (UID: "44f31bc6-bd01-4f16-ad6e-0d1ec90bfb4f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 02:47:16.862971 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:16.862808    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hww8q_5c797b08-a314-44ed-8026-8a0ce1f6c36f/ovn-acl-logging/0.log"
Nov 02 02:47:16.863392 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:16.863365    2577 generic.go:356] "Generic (PLEG): container finished" podID="5c797b08-a314-44ed-8026-8a0ce1f6c36f" containerID="2b4fa01e32a51c3de62373eef442e0bcd848f8b0d41775d52ec2a9e7f2aeb60c" exitCode=1
Nov 02 02:47:16.863467 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:16.863408    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hww8q" event={"ID":"5c797b08-a314-44ed-8026-8a0ce1f6c36f","Type":"ContainerStarted","Data":"6490cdf74bfedb260e614f386d54e371d005987d5748f18a848a5310680411b5"}
Nov 02 02:47:16.863526 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:16.863479    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hww8q" event={"ID":"5c797b08-a314-44ed-8026-8a0ce1f6c36f","Type":"ContainerDied","Data":"2b4fa01e32a51c3de62373eef442e0bcd848f8b0d41775d52ec2a9e7f2aeb60c"}
Nov 02 02:47:16.863581 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:16.863527    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hww8q" event={"ID":"5c797b08-a314-44ed-8026-8a0ce1f6c36f","Type":"ContainerStarted","Data":"82b19e6e204fe001aac12e6b3393f15e4eca2acafe63675802b1db495e18cfbf"}
Nov 02 02:47:17.868398 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:17.868374    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hww8q_5c797b08-a314-44ed-8026-8a0ce1f6c36f/ovn-acl-logging/0.log"
Nov 02 02:47:17.868842 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:17.868756    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hww8q" event={"ID":"5c797b08-a314-44ed-8026-8a0ce1f6c36f","Type":"ContainerStarted","Data":"a837ed8a1de017235fd54944c1f7c38a434f31c55811ccc1988294780880315f"}
Nov 02 02:47:17.868904 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:17.868877    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hww8q" event={"ID":"5c797b08-a314-44ed-8026-8a0ce1f6c36f","Type":"ContainerStarted","Data":"3f03f092bea65241bfb69942437ba2644a7d96889f4bce94f0e781a174e1d583"}
Nov 02 02:47:17.868962 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:17.868929    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hww8q" event={"ID":"5c797b08-a314-44ed-8026-8a0ce1f6c36f","Type":"ContainerStarted","Data":"92c30ee3ade03d4661458da95dafa5f3a7c3072111fc9f876f9406aab1593108"}
Nov 02 02:47:18.713275 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:18.713249    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zkjqm"
Nov 02 02:47:18.713419 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:18.713258    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dxlld"
Nov 02 02:47:18.713419 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:18.713353    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zkjqm" podUID="086e3504-d2fe-43ad-bc5e-dca409a4e695"
Nov 02 02:47:18.713419 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:18.713368    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-47f7x"
Nov 02 02:47:18.713541 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:18.713446    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dxlld" podUID="e400bad6-59c3-4c43-aed0-40c731c967db"
Nov 02 02:47:18.713541 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:18.713515    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-47f7x" podUID="44f31bc6-bd01-4f16-ad6e-0d1ec90bfb4f"
Nov 02 02:47:18.873528 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:18.873491    2577 generic.go:356] "Generic (PLEG): container finished" podID="a81017e4-d705-438e-833b-eedd6de0edde" containerID="576ac7e2a1977b3114cdcdfec404ceae7ebd6de8fcd9a05cb878c7deae995df6" exitCode=0
Nov 02 02:47:18.873528 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:18.873532    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-69j4f" event={"ID":"a81017e4-d705-438e-833b-eedd6de0edde","Type":"ContainerDied","Data":"576ac7e2a1977b3114cdcdfec404ceae7ebd6de8fcd9a05cb878c7deae995df6"}
Nov 02 02:47:19.878383 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:19.878351    2577 generic.go:356] "Generic (PLEG): container finished" podID="a81017e4-d705-438e-833b-eedd6de0edde" containerID="b739c5193007811120325da8e065a1e2330cdf710b065b3e7a1d1ba472d33a9e" exitCode=0
Nov 02 02:47:19.878897 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:19.878435    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-69j4f" event={"ID":"a81017e4-d705-438e-833b-eedd6de0edde","Type":"ContainerDied","Data":"b739c5193007811120325da8e065a1e2330cdf710b065b3e7a1d1ba472d33a9e"}
Nov 02 02:47:19.881581 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:19.881566    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hww8q_5c797b08-a314-44ed-8026-8a0ce1f6c36f/ovn-acl-logging/0.log"
Nov 02 02:47:19.881897 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:19.881875    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hww8q" event={"ID":"5c797b08-a314-44ed-8026-8a0ce1f6c36f","Type":"ContainerStarted","Data":"60a836389f600cac7cc6535382c44cc47ec908bce56196a482249c53769240ef"}
Nov 02 02:47:20.713159 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:20.713125    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zkjqm"
Nov 02 02:47:20.713159 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:20.713161    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-47f7x"
Nov 02 02:47:20.713324 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:20.713127    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dxlld"
Nov 02 02:47:20.713324 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:20.713232    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zkjqm" podUID="086e3504-d2fe-43ad-bc5e-dca409a4e695"
Nov 02 02:47:20.713324 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:20.713282    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dxlld" podUID="e400bad6-59c3-4c43-aed0-40c731c967db"
Nov 02 02:47:20.713444 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:20.713327    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-47f7x" podUID="44f31bc6-bd01-4f16-ad6e-0d1ec90bfb4f"
Nov 02 02:47:20.887200 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:20.887173    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-69j4f" event={"ID":"a81017e4-d705-438e-833b-eedd6de0edde","Type":"ContainerStarted","Data":"f3ce011cc7ddfca15e481511d83f8c10e55f6791a20b60b09d894b14e3ae649e"}
Nov 02 02:47:21.002437 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:21.002402    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-69j4f" podStartSLOduration=7.942369108 podStartE2EDuration="21.002390415s" totalImagesPullingTime="13.060021307s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 02:47:00 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:47:21.002201295 +0000 UTC m=+21.717269432" watchObservedRunningTime="2026-11-02 02:47:21.002390415 +0000 UTC m=+21.717458551"
Nov 02 02:47:21.892851 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:21.892683    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hww8q_5c797b08-a314-44ed-8026-8a0ce1f6c36f/ovn-acl-logging/0.log"
Nov 02 02:47:21.893251 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:21.893174    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hww8q" event={"ID":"5c797b08-a314-44ed-8026-8a0ce1f6c36f","Type":"ContainerStarted","Data":"d32b6a17da6899f79df20eb5324cdbeb90bf7eca8364e207c6adde134924e5ed"}
Nov 02 02:47:21.893667 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:21.893652    2577 scope.go:122] "RemoveContainer" containerID="2b4fa01e32a51c3de62373eef442e0bcd848f8b0d41775d52ec2a9e7f2aeb60c"
Nov 02 02:47:22.712903 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:22.712875    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zkjqm"
Nov 02 02:47:22.713081 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:22.712951    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dxlld"
Nov 02 02:47:22.713081 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:22.713063    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-47f7x"
Nov 02 02:47:22.713081 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:22.713071    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zkjqm" podUID="086e3504-d2fe-43ad-bc5e-dca409a4e695"
Nov 02 02:47:22.713179 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:22.713150    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-47f7x" podUID="44f31bc6-bd01-4f16-ad6e-0d1ec90bfb4f"
Nov 02 02:47:22.713233 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:22.713217    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dxlld" podUID="e400bad6-59c3-4c43-aed0-40c731c967db"
Nov 02 02:47:22.899045 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:22.899002    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hww8q_5c797b08-a314-44ed-8026-8a0ce1f6c36f/ovn-acl-logging/0.log"
Nov 02 02:47:22.899429 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:22.899403    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hww8q" event={"ID":"5c797b08-a314-44ed-8026-8a0ce1f6c36f","Type":"ContainerStarted","Data":"53e87249bff46c922429e4f02e136c9edf1f4a4f40417ef0c86ecc8803f89b78"}
Nov 02 02:47:22.899507 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:22.899492    2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 02:47:22.899746 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:22.899731    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:22.899811 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:22.899800    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:22.913995 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:22.913975    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:22.914207 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:22.914194    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:23.166980 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:23.166928    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-hww8q" podStartSLOduration=7.984307476 podStartE2EDuration="23.166914378s" totalImagesPullingTime="15.182606902s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 02:47:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:47:23.166441653 +0000 UTC m=+23.881509789" watchObservedRunningTime="2026-11-02 02:47:23.166914378 +0000 UTC m=+23.881982516"
Nov 02 02:47:23.839874 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:23.839786    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-dxlld"]
Nov 02 02:47:23.840116 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:23.839875    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dxlld"
Nov 02 02:47:23.840116 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:23.839946    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dxlld" podUID="e400bad6-59c3-4c43-aed0-40c731c967db"
Nov 02 02:47:23.843112 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:23.843085    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-47f7x"]
Nov 02 02:47:23.843229 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:23.843211    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-47f7x"
Nov 02 02:47:23.843335 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:23.843314    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-47f7x" podUID="44f31bc6-bd01-4f16-ad6e-0d1ec90bfb4f"
Nov 02 02:47:23.843741 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:23.843715    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zkjqm"]
Nov 02 02:47:23.843843 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:23.843811    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zkjqm"
Nov 02 02:47:23.843941 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:23.843923    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zkjqm" podUID="086e3504-d2fe-43ad-bc5e-dca409a4e695"
Nov 02 02:47:23.902088 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:23.902063    2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 02:47:24.903914 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:24.903883    2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 02:47:25.712815 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:25.712747    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dxlld"
Nov 02 02:47:25.712815 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:25.712786    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-47f7x"
Nov 02 02:47:25.712973 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:25.712754    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zkjqm"
Nov 02 02:47:25.712973 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:25.712854    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dxlld" podUID="e400bad6-59c3-4c43-aed0-40c731c967db"
Nov 02 02:47:25.712973 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:25.712918    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zkjqm" podUID="086e3504-d2fe-43ad-bc5e-dca409a4e695"
Nov 02 02:47:25.713093 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:25.712974    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-47f7x" podUID="44f31bc6-bd01-4f16-ad6e-0d1ec90bfb4f"
Nov 02 02:47:27.713280 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:27.713216    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dxlld"
Nov 02 02:47:27.713734 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:27.713226    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zkjqm"
Nov 02 02:47:27.713734 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:27.713319    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dxlld" podUID="e400bad6-59c3-4c43-aed0-40c731c967db"
Nov 02 02:47:27.713734 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:27.713423    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zkjqm" podUID="086e3504-d2fe-43ad-bc5e-dca409a4e695"
Nov 02 02:47:27.713734 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:27.713226    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-47f7x"
Nov 02 02:47:27.713734 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:27.713512    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-47f7x" podUID="44f31bc6-bd01-4f16-ad6e-0d1ec90bfb4f"
Nov 02 02:47:29.614981 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:29.614769    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-163.us-west-1.compute.internal" event="NodeReady"
Nov 02 02:47:29.615336 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:29.615038    2577 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Nov 02 02:47:29.713728 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:29.713698    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dxlld"
Nov 02 02:47:29.713882 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:29.713852    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zkjqm"
Nov 02 02:47:29.713946 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:29.713938    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-47f7x"
Nov 02 02:47:29.725890 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:29.725868    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-fbvxx"]
Nov 02 02:47:29.729970 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:29.729940    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-zt7hz"]
Nov 02 02:47:29.730197 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:29.730179    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fbvxx"
Nov 02 02:47:29.733517 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:29.733493    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-zt7hz"
Nov 02 02:47:29.764641 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:29.764616    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-fbvxx"]
Nov 02 02:47:29.774997 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:29.774973    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 02 02:47:29.775119 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:29.775032    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 02 02:47:29.775186 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:29.775121    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 02 02:47:29.775186 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:29.775138    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 02 02:47:29.775186 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:29.775161    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-6md5s\""
Nov 02 02:47:29.775186 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:29.775177    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-x92pp\""
Nov 02 02:47:29.775387 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:29.775211    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-9gkth\""
Nov 02 02:47:29.775611 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:29.775591    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-zt7hz"]
Nov 02 02:47:29.780819 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:29.780801    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2d4b0e00-8bb2-4b4c-9805-af5c51c65405-config-volume\") pod \"dns-default-fbvxx\" (UID: \"2d4b0e00-8bb2-4b4c-9805-af5c51c65405\") " pod="openshift-dns/dns-default-fbvxx"
Nov 02 02:47:29.780905 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:29.780824    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d4b0e00-8bb2-4b4c-9805-af5c51c65405-tmp-dir\") pod \"dns-default-fbvxx\" (UID: \"2d4b0e00-8bb2-4b4c-9805-af5c51c65405\") " pod="openshift-dns/dns-default-fbvxx"
Nov 02 02:47:29.780905 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:29.780859    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xrdb\" (UniqueName: \"kubernetes.io/projected/2d4b0e00-8bb2-4b4c-9805-af5c51c65405-kube-api-access-4xrdb\") pod \"dns-default-fbvxx\" (UID: \"2d4b0e00-8bb2-4b4c-9805-af5c51c65405\") " pod="openshift-dns/dns-default-fbvxx"
Nov 02 02:47:29.780985 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:29.780974    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2d4b0e00-8bb2-4b4c-9805-af5c51c65405-metrics-tls\") pod \"dns-default-fbvxx\" (UID: \"2d4b0e00-8bb2-4b4c-9805-af5c51c65405\") " pod="openshift-dns/dns-default-fbvxx"
Nov 02 02:47:29.781491 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:29.781476    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 02 02:47:29.824361 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:29.824341    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 02 02:47:29.824444 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:29.824429    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 02 02:47:29.824500 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:29.824450    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-wsjnf\""
Nov 02 02:47:29.824552 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:29.824495    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 02 02:47:29.824552 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:29.824525    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-gdqg4\""
Nov 02 02:47:29.824651 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:29.824583    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 02 02:47:29.881768 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:29.881708    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmz88\" (UniqueName: \"kubernetes.io/projected/797676e6-5545-41ad-a4d1-c15f26db122a-kube-api-access-zmz88\") pod \"ingress-canary-zt7hz\" (UID: \"797676e6-5545-41ad-a4d1-c15f26db122a\") " pod="openshift-ingress-canary/ingress-canary-zt7hz"
Nov 02 02:47:29.881768 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:29.881745    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2d4b0e00-8bb2-4b4c-9805-af5c51c65405-metrics-tls\") pod \"dns-default-fbvxx\" (UID: \"2d4b0e00-8bb2-4b4c-9805-af5c51c65405\") " pod="openshift-dns/dns-default-fbvxx"
Nov 02 02:47:29.881887 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:29.881774    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/797676e6-5545-41ad-a4d1-c15f26db122a-cert\") pod \"ingress-canary-zt7hz\" (UID: \"797676e6-5545-41ad-a4d1-c15f26db122a\") " pod="openshift-ingress-canary/ingress-canary-zt7hz"
Nov 02 02:47:29.881887 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:29.881875    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 02:47:29.881974 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:29.881945    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d4b0e00-8bb2-4b4c-9805-af5c51c65405-metrics-tls podName:2d4b0e00-8bb2-4b4c-9805-af5c51c65405 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:30.38192867 +0000 UTC m=+31.096996798 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2d4b0e00-8bb2-4b4c-9805-af5c51c65405-metrics-tls") pod "dns-default-fbvxx" (UID: "2d4b0e00-8bb2-4b4c-9805-af5c51c65405") : secret "dns-default-metrics-tls" not found
Nov 02 02:47:29.882050 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:29.881984    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2d4b0e00-8bb2-4b4c-9805-af5c51c65405-config-volume\") pod \"dns-default-fbvxx\" (UID: \"2d4b0e00-8bb2-4b4c-9805-af5c51c65405\") " pod="openshift-dns/dns-default-fbvxx"
Nov 02 02:47:29.882050 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:29.882002    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d4b0e00-8bb2-4b4c-9805-af5c51c65405-tmp-dir\") pod \"dns-default-fbvxx\" (UID: \"2d4b0e00-8bb2-4b4c-9805-af5c51c65405\") " pod="openshift-dns/dns-default-fbvxx"
Nov 02 02:47:29.882150 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:29.882061    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xrdb\" (UniqueName: \"kubernetes.io/projected/2d4b0e00-8bb2-4b4c-9805-af5c51c65405-kube-api-access-4xrdb\") pod \"dns-default-fbvxx\" (UID: \"2d4b0e00-8bb2-4b4c-9805-af5c51c65405\") " pod="openshift-dns/dns-default-fbvxx"
Nov 02 02:47:29.882357 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:29.882338    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d4b0e00-8bb2-4b4c-9805-af5c51c65405-tmp-dir\") pod \"dns-default-fbvxx\" (UID: \"2d4b0e00-8bb2-4b4c-9805-af5c51c65405\") " pod="openshift-dns/dns-default-fbvxx"
Nov 02 02:47:29.882498 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:29.882482    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2d4b0e00-8bb2-4b4c-9805-af5c51c65405-config-volume\") pod \"dns-default-fbvxx\" (UID: \"2d4b0e00-8bb2-4b4c-9805-af5c51c65405\") " pod="openshift-dns/dns-default-fbvxx"
Nov 02 02:47:29.940287 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:29.940263    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xrdb\" (UniqueName: \"kubernetes.io/projected/2d4b0e00-8bb2-4b4c-9805-af5c51c65405-kube-api-access-4xrdb\") pod \"dns-default-fbvxx\" (UID: \"2d4b0e00-8bb2-4b4c-9805-af5c51c65405\") " pod="openshift-dns/dns-default-fbvxx"
Nov 02 02:47:29.982464 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:29.982408    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmz88\" (UniqueName: \"kubernetes.io/projected/797676e6-5545-41ad-a4d1-c15f26db122a-kube-api-access-zmz88\") pod \"ingress-canary-zt7hz\" (UID: \"797676e6-5545-41ad-a4d1-c15f26db122a\") " pod="openshift-ingress-canary/ingress-canary-zt7hz"
Nov 02 02:47:29.982583 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:29.982507    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/797676e6-5545-41ad-a4d1-c15f26db122a-cert\") pod \"ingress-canary-zt7hz\" (UID: \"797676e6-5545-41ad-a4d1-c15f26db122a\") " pod="openshift-ingress-canary/ingress-canary-zt7hz"
Nov 02 02:47:29.984673 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:29.984656    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/797676e6-5545-41ad-a4d1-c15f26db122a-cert\") pod \"ingress-canary-zt7hz\" (UID: \"797676e6-5545-41ad-a4d1-c15f26db122a\") " pod="openshift-ingress-canary/ingress-canary-zt7hz"
Nov 02 02:47:30.038759 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:30.038729    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmz88\" (UniqueName: \"kubernetes.io/projected/797676e6-5545-41ad-a4d1-c15f26db122a-kube-api-access-zmz88\") pod \"ingress-canary-zt7hz\" (UID: \"797676e6-5545-41ad-a4d1-c15f26db122a\") " pod="openshift-ingress-canary/ingress-canary-zt7hz"
Nov 02 02:47:30.052640 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:30.052609    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-zt7hz"
Nov 02 02:47:30.384965 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:30.384933    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2d4b0e00-8bb2-4b4c-9805-af5c51c65405-metrics-tls\") pod \"dns-default-fbvxx\" (UID: \"2d4b0e00-8bb2-4b4c-9805-af5c51c65405\") " pod="openshift-dns/dns-default-fbvxx"
Nov 02 02:47:30.385141 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:30.385085    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 02:47:30.385141 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:30.385136    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d4b0e00-8bb2-4b4c-9805-af5c51c65405-metrics-tls podName:2d4b0e00-8bb2-4b4c-9805-af5c51c65405 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:31.385125066 +0000 UTC m=+32.100193195 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2d4b0e00-8bb2-4b4c-9805-af5c51c65405-metrics-tls") pod "dns-default-fbvxx" (UID: "2d4b0e00-8bb2-4b4c-9805-af5c51c65405") : secret "dns-default-metrics-tls" not found
Nov 02 02:47:30.428988 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:30.428960    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-zt7hz"]
Nov 02 02:47:30.432314 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:47:30.432293    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod797676e6_5545_41ad_a4d1_c15f26db122a.slice/crio-f227768a19caf7a40a90610cc586c26dfd19d06ff91053ac825736132ff9df09 WatchSource:0}: Error finding container f227768a19caf7a40a90610cc586c26dfd19d06ff91053ac825736132ff9df09: Status 404 returned error can't find the container with id f227768a19caf7a40a90610cc586c26dfd19d06ff91053ac825736132ff9df09
Nov 02 02:47:30.618947 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:30.618914    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:30.619575 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:30.619189    2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 02:47:30.629299 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:30.629278    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:30.917071 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:30.917037    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-zt7hz" event={"ID":"797676e6-5545-41ad-a4d1-c15f26db122a","Type":"ContainerStarted","Data":"f227768a19caf7a40a90610cc586c26dfd19d06ff91053ac825736132ff9df09"}
Nov 02 02:47:31.392669 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:31.392636    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2d4b0e00-8bb2-4b4c-9805-af5c51c65405-metrics-tls\") pod \"dns-default-fbvxx\" (UID: \"2d4b0e00-8bb2-4b4c-9805-af5c51c65405\") " pod="openshift-dns/dns-default-fbvxx"
Nov 02 02:47:31.392887 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:31.392763    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 02:47:31.392887 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:31.392829    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d4b0e00-8bb2-4b4c-9805-af5c51c65405-metrics-tls podName:2d4b0e00-8bb2-4b4c-9805-af5c51c65405 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:33.392810936 +0000 UTC m=+34.107879065 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2d4b0e00-8bb2-4b4c-9805-af5c51c65405-metrics-tls") pod "dns-default-fbvxx" (UID: "2d4b0e00-8bb2-4b4c-9805-af5c51c65405") : secret "dns-default-metrics-tls" not found
Nov 02 02:47:32.702054 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:32.702004    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e400bad6-59c3-4c43-aed0-40c731c967db-original-pull-secret\") pod \"global-pull-secret-syncer-dxlld\" (UID: \"e400bad6-59c3-4c43-aed0-40c731c967db\") " pod="kube-system/global-pull-secret-syncer-dxlld"
Nov 02 02:47:32.702488 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:32.702070    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/086e3504-d2fe-43ad-bc5e-dca409a4e695-metrics-certs\") pod \"network-metrics-daemon-zkjqm\" (UID: \"086e3504-d2fe-43ad-bc5e-dca409a4e695\") " pod="openshift-multus/network-metrics-daemon-zkjqm"
Nov 02 02:47:32.702488 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:32.702186    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 02 02:47:32.702488 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:32.702268    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/086e3504-d2fe-43ad-bc5e-dca409a4e695-metrics-certs podName:086e3504-d2fe-43ad-bc5e-dca409a4e695 nodeName:}" failed. No retries permitted until 2026-11-02 02:48:04.702248516 +0000 UTC m=+65.417316651 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/086e3504-d2fe-43ad-bc5e-dca409a4e695-metrics-certs") pod "network-metrics-daemon-zkjqm" (UID: "086e3504-d2fe-43ad-bc5e-dca409a4e695") : secret "metrics-daemon-secret" not found
Nov 02 02:47:32.704852 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:32.704826    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e400bad6-59c3-4c43-aed0-40c731c967db-original-pull-secret\") pod \"global-pull-secret-syncer-dxlld\" (UID: \"e400bad6-59c3-4c43-aed0-40c731c967db\") " pod="kube-system/global-pull-secret-syncer-dxlld"
Nov 02 02:47:32.728908 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:32.728884    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dxlld"
Nov 02 02:47:32.802991 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:32.802963    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rrzn\" (UniqueName: \"kubernetes.io/projected/44f31bc6-bd01-4f16-ad6e-0d1ec90bfb4f-kube-api-access-7rrzn\") pod \"network-check-target-47f7x\" (UID: \"44f31bc6-bd01-4f16-ad6e-0d1ec90bfb4f\") " pod="openshift-network-diagnostics/network-check-target-47f7x"
Nov 02 02:47:32.805886 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:32.805857    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rrzn\" (UniqueName: \"kubernetes.io/projected/44f31bc6-bd01-4f16-ad6e-0d1ec90bfb4f-kube-api-access-7rrzn\") pod \"network-check-target-47f7x\" (UID: \"44f31bc6-bd01-4f16-ad6e-0d1ec90bfb4f\") " pod="openshift-network-diagnostics/network-check-target-47f7x"
Nov 02 02:47:33.035696 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:33.035650    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-47f7x"
Nov 02 02:47:33.098610 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:33.098581    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-dxlld"]
Nov 02 02:47:33.105089 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:47:33.105061    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode400bad6_59c3_4c43_aed0_40c731c967db.slice/crio-199cbd2b34394d4646dabbcc83851c90c1b35b513f8789a43d2ab8a72d0d41dd WatchSource:0}: Error finding container 199cbd2b34394d4646dabbcc83851c90c1b35b513f8789a43d2ab8a72d0d41dd: Status 404 returned error can't find the container with id 199cbd2b34394d4646dabbcc83851c90c1b35b513f8789a43d2ab8a72d0d41dd
Nov 02 02:47:33.405761 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:33.405687    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-47f7x"]
Nov 02 02:47:33.408243 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:33.408216    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2d4b0e00-8bb2-4b4c-9805-af5c51c65405-metrics-tls\") pod \"dns-default-fbvxx\" (UID: \"2d4b0e00-8bb2-4b4c-9805-af5c51c65405\") " pod="openshift-dns/dns-default-fbvxx"
Nov 02 02:47:33.408344 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:33.408315    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 02:47:33.408386 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:33.408376    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d4b0e00-8bb2-4b4c-9805-af5c51c65405-metrics-tls podName:2d4b0e00-8bb2-4b4c-9805-af5c51c65405 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:37.408360105 +0000 UTC m=+38.123428235 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2d4b0e00-8bb2-4b4c-9805-af5c51c65405-metrics-tls") pod "dns-default-fbvxx" (UID: "2d4b0e00-8bb2-4b4c-9805-af5c51c65405") : secret "dns-default-metrics-tls" not found
Nov 02 02:47:33.409265 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:47:33.409241    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44f31bc6_bd01_4f16_ad6e_0d1ec90bfb4f.slice/crio-9f13665adfc009e86a8e18b45db3c81b5fa09a6748c1eee67f7fb555e86bfa8b WatchSource:0}: Error finding container 9f13665adfc009e86a8e18b45db3c81b5fa09a6748c1eee67f7fb555e86bfa8b: Status 404 returned error can't find the container with id 9f13665adfc009e86a8e18b45db3c81b5fa09a6748c1eee67f7fb555e86bfa8b
Nov 02 02:47:33.926745 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:33.926706    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-zt7hz" event={"ID":"797676e6-5545-41ad-a4d1-c15f26db122a","Type":"ContainerStarted","Data":"73a3dd32de431abad896cd4ea921fad96b1f77fa6f4f79f9112a250431ed6a0e"}
Nov 02 02:47:33.927890 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:33.927864    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-47f7x" event={"ID":"44f31bc6-bd01-4f16-ad6e-0d1ec90bfb4f","Type":"ContainerStarted","Data":"9f13665adfc009e86a8e18b45db3c81b5fa09a6748c1eee67f7fb555e86bfa8b"}
Nov 02 02:47:33.928961 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:33.928942    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-dxlld" event={"ID":"e400bad6-59c3-4c43-aed0-40c731c967db","Type":"ContainerStarted","Data":"199cbd2b34394d4646dabbcc83851c90c1b35b513f8789a43d2ab8a72d0d41dd"}
Nov 02 02:47:34.039328 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:34.039279    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-zt7hz" podStartSLOduration=2.197224372 podStartE2EDuration="5.039262307s" totalImagesPullingTime="2.842037935s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:47:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:47:34.038345279 +0000 UTC m=+34.753413416" watchObservedRunningTime="2026-11-02 02:47:34.039262307 +0000 UTC m=+34.754330443"
Nov 02 02:47:35.267453 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:35.267415    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-r4zs9"]
Nov 02 02:47:35.304096 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:35.304073    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-r4zs9"
Nov 02 02:47:35.331734 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:35.331706    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-r4zs9"]
Nov 02 02:47:35.380191 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:35.380159    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 02 02:47:35.380349 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:35.380281    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-vvbct\""
Nov 02 02:47:35.380921 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:35.380902    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 02 02:47:35.381030 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:35.381000    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 02 02:47:35.412470 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:35.412435    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 02 02:47:35.423865 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:35.423833    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/93e3306c-8c14-4bd4-ae04-08bc4b01a5fd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r4zs9\" (UID: \"93e3306c-8c14-4bd4-ae04-08bc4b01a5fd\") " pod="openshift-insights/insights-runtime-extractor-r4zs9"
Nov 02 02:47:35.423997 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:35.423890    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/93e3306c-8c14-4bd4-ae04-08bc4b01a5fd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r4zs9\" (UID: \"93e3306c-8c14-4bd4-ae04-08bc4b01a5fd\") " pod="openshift-insights/insights-runtime-extractor-r4zs9"
Nov 02 02:47:35.423997 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:35.423929    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/93e3306c-8c14-4bd4-ae04-08bc4b01a5fd-crio-socket\") pod \"insights-runtime-extractor-r4zs9\" (UID: \"93e3306c-8c14-4bd4-ae04-08bc4b01a5fd\") " pod="openshift-insights/insights-runtime-extractor-r4zs9"
Nov 02 02:47:35.423997 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:35.423966    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/93e3306c-8c14-4bd4-ae04-08bc4b01a5fd-data-volume\") pod \"insights-runtime-extractor-r4zs9\" (UID: \"93e3306c-8c14-4bd4-ae04-08bc4b01a5fd\") " pod="openshift-insights/insights-runtime-extractor-r4zs9"
Nov 02 02:47:35.424193 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:35.424094    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b47mc\" (UniqueName: \"kubernetes.io/projected/93e3306c-8c14-4bd4-ae04-08bc4b01a5fd-kube-api-access-b47mc\") pod \"insights-runtime-extractor-r4zs9\" (UID: \"93e3306c-8c14-4bd4-ae04-08bc4b01a5fd\") " pod="openshift-insights/insights-runtime-extractor-r4zs9"
Nov 02 02:47:35.494522 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:35.494488    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-sp26d"]
Nov 02 02:47:35.508751 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:35.508680    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-t9g7w-hqs6p"]
Nov 02 02:47:35.508899 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:35.508849    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-sp26d"
Nov 02 02:47:35.523188 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:35.523133    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-t9g7w-hqs6p"
Nov 02 02:47:35.525402 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:35.525378    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b47mc\" (UniqueName: \"kubernetes.io/projected/93e3306c-8c14-4bd4-ae04-08bc4b01a5fd-kube-api-access-b47mc\") pod \"insights-runtime-extractor-r4zs9\" (UID: \"93e3306c-8c14-4bd4-ae04-08bc4b01a5fd\") " pod="openshift-insights/insights-runtime-extractor-r4zs9"
Nov 02 02:47:35.525525 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:35.525450    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/93e3306c-8c14-4bd4-ae04-08bc4b01a5fd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r4zs9\" (UID: \"93e3306c-8c14-4bd4-ae04-08bc4b01a5fd\") " pod="openshift-insights/insights-runtime-extractor-r4zs9"
Nov 02 02:47:35.525525 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:35.525491    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/93e3306c-8c14-4bd4-ae04-08bc4b01a5fd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r4zs9\" (UID: \"93e3306c-8c14-4bd4-ae04-08bc4b01a5fd\") " pod="openshift-insights/insights-runtime-extractor-r4zs9"
Nov 02 02:47:35.525642 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:35.525535    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/93e3306c-8c14-4bd4-ae04-08bc4b01a5fd-crio-socket\") pod \"insights-runtime-extractor-r4zs9\" (UID: \"93e3306c-8c14-4bd4-ae04-08bc4b01a5fd\") " pod="openshift-insights/insights-runtime-extractor-r4zs9"
Nov 02 02:47:35.525642 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:35.525569    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/93e3306c-8c14-4bd4-ae04-08bc4b01a5fd-data-volume\") pod \"insights-runtime-extractor-r4zs9\" (UID: \"93e3306c-8c14-4bd4-ae04-08bc4b01a5fd\") " pod="openshift-insights/insights-runtime-extractor-r4zs9"
Nov 02 02:47:35.525642 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:35.525593    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 02 02:47:35.525781 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:35.525674    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/93e3306c-8c14-4bd4-ae04-08bc4b01a5fd-kube-rbac-proxy-cm podName:93e3306c-8c14-4bd4-ae04-08bc4b01a5fd nodeName:}" failed. No retries permitted until 2026-11-02 02:47:36.025655818 +0000 UTC m=+36.740723951 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/93e3306c-8c14-4bd4-ae04-08bc4b01a5fd-kube-rbac-proxy-cm") pod "insights-runtime-extractor-r4zs9" (UID: "93e3306c-8c14-4bd4-ae04-08bc4b01a5fd") : configmap "kube-rbac-proxy" not found
Nov 02 02:47:35.525781 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:35.525733    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 02:47:35.525885 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:35.525788    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93e3306c-8c14-4bd4-ae04-08bc4b01a5fd-insights-runtime-extractor-tls podName:93e3306c-8c14-4bd4-ae04-08bc4b01a5fd nodeName:}" failed. No retries permitted until 2026-11-02 02:47:36.025772439 +0000 UTC m=+36.740840580 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/93e3306c-8c14-4bd4-ae04-08bc4b01a5fd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-r4zs9" (UID: "93e3306c-8c14-4bd4-ae04-08bc4b01a5fd") : secret "insights-runtime-extractor-tls" not found
Nov 02 02:47:35.525946 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:35.525914    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/93e3306c-8c14-4bd4-ae04-08bc4b01a5fd-crio-socket\") pod \"insights-runtime-extractor-r4zs9\" (UID: \"93e3306c-8c14-4bd4-ae04-08bc4b01a5fd\") " pod="openshift-insights/insights-runtime-extractor-r4zs9"
Nov 02 02:47:35.526125 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:35.526100    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/93e3306c-8c14-4bd4-ae04-08bc4b01a5fd-data-volume\") pod \"insights-runtime-extractor-r4zs9\" (UID: \"93e3306c-8c14-4bd4-ae04-08bc4b01a5fd\") " pod="openshift-insights/insights-runtime-extractor-r4zs9"
Nov 02 02:47:35.553159 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:35.553132    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-sp26d"]
Nov 02 02:47:35.560457 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:35.560425    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Nov 02 02:47:35.560571 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:35.560528    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-85n6h\""
Nov 02 02:47:35.560631 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:35.560571    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Nov 02 02:47:35.566130 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:35.566109    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-t9g7w-hqs6p"]
Nov 02 02:47:35.582067 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:35.582042    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b47mc\" (UniqueName: \"kubernetes.io/projected/93e3306c-8c14-4bd4-ae04-08bc4b01a5fd-kube-api-access-b47mc\") pod \"insights-runtime-extractor-r4zs9\" (UID: \"93e3306c-8c14-4bd4-ae04-08bc4b01a5fd\") " pod="openshift-insights/insights-runtime-extractor-r4zs9"
Nov 02 02:47:35.626594 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:35.626510    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpg9j\" (UniqueName: \"kubernetes.io/projected/d66bfee1-59f7-4ed9-90e1-31582a4c4784-kube-api-access-vpg9j\") pod \"migrator-66df5d496b-sp26d\" (UID: \"d66bfee1-59f7-4ed9-90e1-31582a4c4784\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-sp26d"
Nov 02 02:47:35.626720 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:35.626649    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95effc61-7dc3-42c1-9609-be607c3bc65f-service-ca-bundle\") pod \"periodic-gathering-t9g7w-hqs6p\" (UID: \"95effc61-7dc3-42c1-9609-be607c3bc65f\") " pod="openshift-insights/periodic-gathering-t9g7w-hqs6p"
Nov 02 02:47:35.626720 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:35.626694    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/95effc61-7dc3-42c1-9609-be607c3bc65f-archives-path\") pod \"periodic-gathering-t9g7w-hqs6p\" (UID: \"95effc61-7dc3-42c1-9609-be607c3bc65f\") " pod="openshift-insights/periodic-gathering-t9g7w-hqs6p"
Nov 02 02:47:35.626825 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:35.626721    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95effc61-7dc3-42c1-9609-be607c3bc65f-trusted-ca-bundle\") pod \"periodic-gathering-t9g7w-hqs6p\" (UID: \"95effc61-7dc3-42c1-9609-be607c3bc65f\") " pod="openshift-insights/periodic-gathering-t9g7w-hqs6p"
Nov 02 02:47:35.626873 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:35.626826    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dl7x\" (UniqueName: \"kubernetes.io/projected/95effc61-7dc3-42c1-9609-be607c3bc65f-kube-api-access-2dl7x\") pod \"periodic-gathering-t9g7w-hqs6p\" (UID: \"95effc61-7dc3-42c1-9609-be607c3bc65f\") " pod="openshift-insights/periodic-gathering-t9g7w-hqs6p"
Nov 02 02:47:35.656264 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:35.656239    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-dgnd5\""
Nov 02 02:47:35.656421 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:35.656389    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 02 02:47:35.719596 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:35.719566    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 02 02:47:35.727723 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:35.727693    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/95effc61-7dc3-42c1-9609-be607c3bc65f-archives-path\") pod \"periodic-gathering-t9g7w-hqs6p\" (UID: \"95effc61-7dc3-42c1-9609-be607c3bc65f\") " pod="openshift-insights/periodic-gathering-t9g7w-hqs6p"
Nov 02 02:47:35.727844 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:35.727743    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95effc61-7dc3-42c1-9609-be607c3bc65f-trusted-ca-bundle\") pod \"periodic-gathering-t9g7w-hqs6p\" (UID: \"95effc61-7dc3-42c1-9609-be607c3bc65f\") " pod="openshift-insights/periodic-gathering-t9g7w-hqs6p"
Nov 02 02:47:35.727844 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:35.727792    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dl7x\" (UniqueName: \"kubernetes.io/projected/95effc61-7dc3-42c1-9609-be607c3bc65f-kube-api-access-2dl7x\") pod \"periodic-gathering-t9g7w-hqs6p\" (UID: \"95effc61-7dc3-42c1-9609-be607c3bc65f\") " pod="openshift-insights/periodic-gathering-t9g7w-hqs6p"
Nov 02 02:47:35.727844 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:35.727819    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpg9j\" (UniqueName: \"kubernetes.io/projected/d66bfee1-59f7-4ed9-90e1-31582a4c4784-kube-api-access-vpg9j\") pod \"migrator-66df5d496b-sp26d\" (UID: \"d66bfee1-59f7-4ed9-90e1-31582a4c4784\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-sp26d"
Nov 02 02:47:35.727972 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:35.727896    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95effc61-7dc3-42c1-9609-be607c3bc65f-service-ca-bundle\") pod \"periodic-gathering-t9g7w-hqs6p\" (UID: \"95effc61-7dc3-42c1-9609-be607c3bc65f\") " pod="openshift-insights/periodic-gathering-t9g7w-hqs6p"
Nov 02 02:47:35.728129 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:35.728108    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/95effc61-7dc3-42c1-9609-be607c3bc65f-archives-path\") pod \"periodic-gathering-t9g7w-hqs6p\" (UID: \"95effc61-7dc3-42c1-9609-be607c3bc65f\") " pod="openshift-insights/periodic-gathering-t9g7w-hqs6p"
Nov 02 02:47:35.728501 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:35.728480    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95effc61-7dc3-42c1-9609-be607c3bc65f-service-ca-bundle\") pod \"periodic-gathering-t9g7w-hqs6p\" (UID: \"95effc61-7dc3-42c1-9609-be607c3bc65f\") " pod="openshift-insights/periodic-gathering-t9g7w-hqs6p"
Nov 02 02:47:35.729065 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:35.729042    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95effc61-7dc3-42c1-9609-be607c3bc65f-trusted-ca-bundle\") pod \"periodic-gathering-t9g7w-hqs6p\" (UID: \"95effc61-7dc3-42c1-9609-be607c3bc65f\") " pod="openshift-insights/periodic-gathering-t9g7w-hqs6p"
Nov 02 02:47:35.784915 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:35.784852    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpg9j\" (UniqueName: \"kubernetes.io/projected/d66bfee1-59f7-4ed9-90e1-31582a4c4784-kube-api-access-vpg9j\") pod \"migrator-66df5d496b-sp26d\" (UID: \"d66bfee1-59f7-4ed9-90e1-31582a4c4784\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-sp26d"
Nov 02 02:47:35.785319 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:35.785297    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dl7x\" (UniqueName: \"kubernetes.io/projected/95effc61-7dc3-42c1-9609-be607c3bc65f-kube-api-access-2dl7x\") pod \"periodic-gathering-t9g7w-hqs6p\" (UID: \"95effc61-7dc3-42c1-9609-be607c3bc65f\") " pod="openshift-insights/periodic-gathering-t9g7w-hqs6p"
Nov 02 02:47:35.841898 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:35.841869    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-sp26d"
Nov 02 02:47:35.865053 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:35.865008    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-t9g7w-hqs6p"
Nov 02 02:47:36.031876 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:36.031644    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/93e3306c-8c14-4bd4-ae04-08bc4b01a5fd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r4zs9\" (UID: \"93e3306c-8c14-4bd4-ae04-08bc4b01a5fd\") " pod="openshift-insights/insights-runtime-extractor-r4zs9"
Nov 02 02:47:36.031876 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:36.031682    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/93e3306c-8c14-4bd4-ae04-08bc4b01a5fd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r4zs9\" (UID: \"93e3306c-8c14-4bd4-ae04-08bc4b01a5fd\") " pod="openshift-insights/insights-runtime-extractor-r4zs9"
Nov 02 02:47:36.031876 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:36.031766    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 02 02:47:36.031876 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:36.031792    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 02:47:36.031876 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:36.031835    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/93e3306c-8c14-4bd4-ae04-08bc4b01a5fd-kube-rbac-proxy-cm podName:93e3306c-8c14-4bd4-ae04-08bc4b01a5fd nodeName:}" failed. No retries permitted until 2026-11-02 02:47:37.031819445 +0000 UTC m=+37.746887559 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/93e3306c-8c14-4bd4-ae04-08bc4b01a5fd-kube-rbac-proxy-cm") pod "insights-runtime-extractor-r4zs9" (UID: "93e3306c-8c14-4bd4-ae04-08bc4b01a5fd") : configmap "kube-rbac-proxy" not found
Nov 02 02:47:36.031876 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:36.031853    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93e3306c-8c14-4bd4-ae04-08bc4b01a5fd-insights-runtime-extractor-tls podName:93e3306c-8c14-4bd4-ae04-08bc4b01a5fd nodeName:}" failed. No retries permitted until 2026-11-02 02:47:37.031841818 +0000 UTC m=+37.746909934 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/93e3306c-8c14-4bd4-ae04-08bc4b01a5fd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-r4zs9" (UID: "93e3306c-8c14-4bd4-ae04-08bc4b01a5fd") : secret "insights-runtime-extractor-tls" not found
Nov 02 02:47:36.240146 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:36.240108    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-sp26d"]
Nov 02 02:47:36.284883 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:47:36.284851    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd66bfee1_59f7_4ed9_90e1_31582a4c4784.slice/crio-73ab21a3c4797d944b2b19c2abe98ef17171bfe56972b32ba8c542cb38144243 WatchSource:0}: Error finding container 73ab21a3c4797d944b2b19c2abe98ef17171bfe56972b32ba8c542cb38144243: Status 404 returned error can't find the container with id 73ab21a3c4797d944b2b19c2abe98ef17171bfe56972b32ba8c542cb38144243
Nov 02 02:47:36.290198 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:36.290171    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-t9g7w-hqs6p"]
Nov 02 02:47:36.294739 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:47:36.294712    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95effc61_7dc3_42c1_9609_be607c3bc65f.slice/crio-d960a2b9fc534555ae1a4989812606630fdb546f04750fe5c7a6129dd61abe4d WatchSource:0}: Error finding container d960a2b9fc534555ae1a4989812606630fdb546f04750fe5c7a6129dd61abe4d: Status 404 returned error can't find the container with id d960a2b9fc534555ae1a4989812606630fdb546f04750fe5c7a6129dd61abe4d
Nov 02 02:47:36.493827 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:36.493754    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-rspt2"]
Nov 02 02:47:36.516838 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:36.516803    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-rspt2"
Nov 02 02:47:36.552244 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:36.552207    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-rspt2"]
Nov 02 02:47:36.569448 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:36.569411    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Nov 02 02:47:36.569448 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:36.569415    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Nov 02 02:47:36.569448 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:36.569415    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Nov 02 02:47:36.569685 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:36.569449    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-xmln4\""
Nov 02 02:47:36.569685 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:36.569482    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Nov 02 02:47:36.636997 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:36.636965    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thncs\" (UniqueName: \"kubernetes.io/projected/39d09ab7-5724-44d4-a731-92eabb9280a6-kube-api-access-thncs\") pod \"service-ca-5fffc97bdf-rspt2\" (UID: \"39d09ab7-5724-44d4-a731-92eabb9280a6\") " pod="openshift-service-ca/service-ca-5fffc97bdf-rspt2"
Nov 02 02:47:36.637132 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:36.637049    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/39d09ab7-5724-44d4-a731-92eabb9280a6-signing-key\") pod \"service-ca-5fffc97bdf-rspt2\" (UID: \"39d09ab7-5724-44d4-a731-92eabb9280a6\") " pod="openshift-service-ca/service-ca-5fffc97bdf-rspt2"
Nov 02 02:47:36.637132 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:36.637105    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/39d09ab7-5724-44d4-a731-92eabb9280a6-signing-cabundle\") pod \"service-ca-5fffc97bdf-rspt2\" (UID: \"39d09ab7-5724-44d4-a731-92eabb9280a6\") " pod="openshift-service-ca/service-ca-5fffc97bdf-rspt2"
Nov 02 02:47:36.738332 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:36.738288    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/39d09ab7-5724-44d4-a731-92eabb9280a6-signing-cabundle\") pod \"service-ca-5fffc97bdf-rspt2\" (UID: \"39d09ab7-5724-44d4-a731-92eabb9280a6\") " pod="openshift-service-ca/service-ca-5fffc97bdf-rspt2"
Nov 02 02:47:36.738507 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:36.738404    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thncs\" (UniqueName: \"kubernetes.io/projected/39d09ab7-5724-44d4-a731-92eabb9280a6-kube-api-access-thncs\") pod \"service-ca-5fffc97bdf-rspt2\" (UID: \"39d09ab7-5724-44d4-a731-92eabb9280a6\") " pod="openshift-service-ca/service-ca-5fffc97bdf-rspt2"
Nov 02 02:47:36.738507 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:36.738436    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/39d09ab7-5724-44d4-a731-92eabb9280a6-signing-key\") pod \"service-ca-5fffc97bdf-rspt2\" (UID: \"39d09ab7-5724-44d4-a731-92eabb9280a6\") " pod="openshift-service-ca/service-ca-5fffc97bdf-rspt2"
Nov 02 02:47:36.739041 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:36.738982    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/39d09ab7-5724-44d4-a731-92eabb9280a6-signing-cabundle\") pod \"service-ca-5fffc97bdf-rspt2\" (UID: \"39d09ab7-5724-44d4-a731-92eabb9280a6\") " pod="openshift-service-ca/service-ca-5fffc97bdf-rspt2"
Nov 02 02:47:36.741867 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:36.741841    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/39d09ab7-5724-44d4-a731-92eabb9280a6-signing-key\") pod \"service-ca-5fffc97bdf-rspt2\" (UID: \"39d09ab7-5724-44d4-a731-92eabb9280a6\") " pod="openshift-service-ca/service-ca-5fffc97bdf-rspt2"
Nov 02 02:47:36.795493 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:36.795426    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thncs\" (UniqueName: \"kubernetes.io/projected/39d09ab7-5724-44d4-a731-92eabb9280a6-kube-api-access-thncs\") pod \"service-ca-5fffc97bdf-rspt2\" (UID: \"39d09ab7-5724-44d4-a731-92eabb9280a6\") " pod="openshift-service-ca/service-ca-5fffc97bdf-rspt2"
Nov 02 02:47:36.843527 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:36.843342    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-rspt2"
Nov 02 02:47:36.941066 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:36.941035    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-sp26d" event={"ID":"d66bfee1-59f7-4ed9-90e1-31582a4c4784","Type":"ContainerStarted","Data":"73ab21a3c4797d944b2b19c2abe98ef17171bfe56972b32ba8c542cb38144243"}
Nov 02 02:47:36.942554 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:36.942521    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-t9g7w-hqs6p" event={"ID":"95effc61-7dc3-42c1-9609-be607c3bc65f","Type":"ContainerStarted","Data":"d960a2b9fc534555ae1a4989812606630fdb546f04750fe5c7a6129dd61abe4d"}
Nov 02 02:47:37.041257 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:37.041215    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/93e3306c-8c14-4bd4-ae04-08bc4b01a5fd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r4zs9\" (UID: \"93e3306c-8c14-4bd4-ae04-08bc4b01a5fd\") " pod="openshift-insights/insights-runtime-extractor-r4zs9"
Nov 02 02:47:37.041437 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:37.041282    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/93e3306c-8c14-4bd4-ae04-08bc4b01a5fd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r4zs9\" (UID: \"93e3306c-8c14-4bd4-ae04-08bc4b01a5fd\") " pod="openshift-insights/insights-runtime-extractor-r4zs9"
Nov 02 02:47:37.041437 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:37.041348    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 02 02:47:37.041437 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:37.041416    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 02:47:37.041606 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:37.041421    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/93e3306c-8c14-4bd4-ae04-08bc4b01a5fd-kube-rbac-proxy-cm podName:93e3306c-8c14-4bd4-ae04-08bc4b01a5fd nodeName:}" failed. No retries permitted until 2026-11-02 02:47:39.041407526 +0000 UTC m=+39.756475641 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/93e3306c-8c14-4bd4-ae04-08bc4b01a5fd-kube-rbac-proxy-cm") pod "insights-runtime-extractor-r4zs9" (UID: "93e3306c-8c14-4bd4-ae04-08bc4b01a5fd") : configmap "kube-rbac-proxy" not found
Nov 02 02:47:37.041606 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:37.041489    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93e3306c-8c14-4bd4-ae04-08bc4b01a5fd-insights-runtime-extractor-tls podName:93e3306c-8c14-4bd4-ae04-08bc4b01a5fd nodeName:}" failed. No retries permitted until 2026-11-02 02:47:39.041470421 +0000 UTC m=+39.756538542 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/93e3306c-8c14-4bd4-ae04-08bc4b01a5fd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-r4zs9" (UID: "93e3306c-8c14-4bd4-ae04-08bc4b01a5fd") : secret "insights-runtime-extractor-tls" not found
Nov 02 02:47:37.227034 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:37.226794    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-rspt2"]
Nov 02 02:47:37.229680 ip-10-0-133-163 kubenswrapper[2577]: W1102 02:47:37.229651    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39d09ab7_5724_44d4_a731_92eabb9280a6.slice/crio-5203b74b3926e29a5b3848e9d8aab0c1c47ed977b976ed8bd7e613c79b15d91e WatchSource:0}: Error finding container 5203b74b3926e29a5b3848e9d8aab0c1c47ed977b976ed8bd7e613c79b15d91e: Status 404 returned error can't find the container with id 5203b74b3926e29a5b3848e9d8aab0c1c47ed977b976ed8bd7e613c79b15d91e
Nov 02 02:47:37.445216 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:37.445178    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2d4b0e00-8bb2-4b4c-9805-af5c51c65405-metrics-tls\") pod \"dns-default-fbvxx\" (UID: \"2d4b0e00-8bb2-4b4c-9805-af5c51c65405\") " pod="openshift-dns/dns-default-fbvxx"
Nov 02 02:47:37.445674 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:37.445354    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 02:47:37.445674 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:37.445416    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d4b0e00-8bb2-4b4c-9805-af5c51c65405-metrics-tls podName:2d4b0e00-8bb2-4b4c-9805-af5c51c65405 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:45.445397675 +0000 UTC m=+46.160465804 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2d4b0e00-8bb2-4b4c-9805-af5c51c65405-metrics-tls") pod "dns-default-fbvxx" (UID: "2d4b0e00-8bb2-4b4c-9805-af5c51c65405") : secret "dns-default-metrics-tls" not found
Nov 02 02:47:37.947635 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:37.947522    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-47f7x" event={"ID":"44f31bc6-bd01-4f16-ad6e-0d1ec90bfb4f","Type":"ContainerStarted","Data":"4861bdc260fdb328e74437fd44c6f33f23ea9b6bdb0dc6a2186dbbe400bc798b"}
Nov 02 02:47:37.947635 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:37.947618    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-47f7x"
Nov 02 02:47:37.949087 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:37.949057    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-rspt2" event={"ID":"39d09ab7-5724-44d4-a731-92eabb9280a6","Type":"ContainerStarted","Data":"5203b74b3926e29a5b3848e9d8aab0c1c47ed977b976ed8bd7e613c79b15d91e"}
Nov 02 02:47:38.062592 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:38.062413    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-47f7x" podStartSLOduration=33.926595017 podStartE2EDuration="38.062394968s" totalImagesPullingTime="4.135799951s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:47:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:47:38.061282115 +0000 UTC m=+38.776350254" watchObservedRunningTime="2026-11-02 02:47:38.062394968 +0000 UTC m=+38.777463105"
Nov 02 02:47:38.953795 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:38.953757    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-sp26d" event={"ID":"d66bfee1-59f7-4ed9-90e1-31582a4c4784","Type":"ContainerStarted","Data":"7c8a2be528c14bf72f45bda3a3f9be80d799ec53180845c71c895969ccc21488"}
Nov 02 02:47:39.056633 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:39.056545    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/93e3306c-8c14-4bd4-ae04-08bc4b01a5fd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r4zs9\" (UID: \"93e3306c-8c14-4bd4-ae04-08bc4b01a5fd\") " pod="openshift-insights/insights-runtime-extractor-r4zs9"
Nov 02 02:47:39.056755 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:39.056658    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/93e3306c-8c14-4bd4-ae04-08bc4b01a5fd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r4zs9\" (UID: \"93e3306c-8c14-4bd4-ae04-08bc4b01a5fd\") " pod="openshift-insights/insights-runtime-extractor-r4zs9"
Nov 02 02:47:39.056755 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:39.056710    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 02 02:47:39.056859 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:39.056784    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/93e3306c-8c14-4bd4-ae04-08bc4b01a5fd-kube-rbac-proxy-cm podName:93e3306c-8c14-4bd4-ae04-08bc4b01a5fd nodeName:}" failed. No retries permitted until 2026-11-02 02:47:43.05676256 +0000 UTC m=+43.771830675 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/93e3306c-8c14-4bd4-ae04-08bc4b01a5fd-kube-rbac-proxy-cm") pod "insights-runtime-extractor-r4zs9" (UID: "93e3306c-8c14-4bd4-ae04-08bc4b01a5fd") : configmap "kube-rbac-proxy" not found
Nov 02 02:47:39.056859 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:39.056823    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 02:47:39.056957 ip-10-0-133-163 kubenswrapper[2577]: E1102 02:47:39.056881    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93e3306c-8c14-4bd4-ae04-08bc4b01a5fd-insights-runtime-extractor-tls podName:93e3306c-8c14-4bd4-ae04-08bc4b01a5fd nodeName:}" failed. No retries permitted until 2026-11-02 02:47:43.056863891 +0000 UTC m=+43.771932023 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/93e3306c-8c14-4bd4-ae04-08bc4b01a5fd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-r4zs9" (UID: "93e3306c-8c14-4bd4-ae04-08bc4b01a5fd") : secret "insights-runtime-extractor-tls" not found
Nov 02 02:47:39.109961 ip-10-0-133-163 kubenswrapper[2577]: I1102 02:47:39.109918    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-sp26d" podStartSLOduration=1.680184516 podStartE2EDuration="4.109902109s" totalImagesPullingTime="2.429717593s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:47:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:47:39.06768673 +0000 UTC m=+39.782754861" watchObservedRunningTime="2026-11-02 02:47:39.109902109 +0000 UTC m=+39.824970247"
Nov 02 02:47:39.364003 ip-10-0-133-163 systemd[1]: Stopping Kubernetes Kubelet...
Nov 02 02:47:39.379555 ip-10-0-133-163 systemd[1]: kubelet.service: Deactivated successfully.
Nov 02 02:47:39.379731 ip-10-0-133-163 systemd[1]: Stopped Kubernetes Kubelet.
Nov 02 02:47:39.379807 ip-10-0-133-163 systemd[1]: kubelet.service: Consumed 2.323s CPU time.
Nov 02 02:47:39.392565 ip-10-0-133-163 systemd[1]: Starting Kubernetes Kubelet...
Nov 02 02:47:39.490937 ip-10-0-133-163 kubenswrapper[4496]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 02:47:39.490937 ip-10-0-133-163 kubenswrapper[4496]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 02 02:47:39.490937 ip-10-0-133-163 kubenswrapper[4496]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 02:47:39.490937 ip-10-0-133-163 kubenswrapper[4496]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 02:47:39.493040 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.492944    4496 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 02:47:39.493040 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.493039    4496 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 02:47:39.493040 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.493043    4496 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 02:47:39.493131 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.493046    4496 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 02:47:39.493131 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.493049    4496 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 02:47:39.493131 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.493052    4496 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 02:47:39.493131 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.493055    4496 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 02:47:39.493131 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.493057    4496 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 02:47:39.493131 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.493060    4496 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 02:47:39.493131 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.493063    4496 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 02:47:39.493131 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.493065    4496 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 02:47:39.493131 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.493068    4496 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 02:47:39.493131 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.493070    4496 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 02:47:39.493131 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.493073    4496 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 02:47:39.493131 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.493076    4496 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 02:47:39.493131 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.493078    4496 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 02:47:39.493131 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.493081    4496 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 02:47:39.493131 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.493083    4496 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 02:47:39.493131 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.493085    4496 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 02:47:39.493131 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.493088    4496 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 02:47:39.493131 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.493091    4496 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 02:47:39.493131 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.493096    4496 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 02:47:39.493612 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.493100    4496 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 02:47:39.493612 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.493110    4496 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 02:47:39.493612 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.493114    4496 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 02:47:39.493612 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.493117    4496 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 02:47:39.493612 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.493120    4496 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 02:47:39.493612 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.493122    4496 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 02:47:39.493612 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.493125    4496 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 02:47:39.493612 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.493128    4496 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 02:47:39.493612 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.493131    4496 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 02:47:39.493612 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.493134    4496 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 02:47:39.493612 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.493137    4496 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 02:47:39.493612 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.493140    4496 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 02:47:39.493612 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.493142    4496 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 02:47:39.493612 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.493145    4496 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 02:47:39.493612 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.493148    4496 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 02:47:39.493612 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.493151    4496 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 02:47:39.493612 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.493153    4496 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 02:47:39.493612 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.493156    4496 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 02:47:39.493612 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.493159    4496 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 02:47:39.494127 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.493161    4496 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 02:47:39.494127 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.493164    4496 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 02:47:39.494127 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.493166    4496 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 02:47:39.494127 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.493169    4496 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 02:47:39.494127 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.493172    4496 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 02:47:39.494127 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.493174    4496 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 02:47:39.494127 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.493177    4496 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 02:47:39.494127 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.493179    4496 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 02:47:39.494127 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.493182    4496 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 02:47:39.494127 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.493185    4496 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 02:47:39.494127 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.493187    4496 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 02:47:39.494127 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.493190    4496 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 02:47:39.494127 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.493193    4496 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 02:47:39.494127 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.493195    4496 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 02:47:39.494127 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.493198    4496 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 02:47:39.494127 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.493202    4496 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 02:47:39.494127 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.493205    4496 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 02:47:39.494127 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.493207    4496 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 02:47:39.494127 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.493210    4496 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 02:47:39.494127 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.493214    4496 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 02:47:39.494615 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.493216    4496 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 02:47:39.494615 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.493222    4496 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 02:47:39.494615 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.493226    4496 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 02:47:39.494615 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.493229    4496 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 02:47:39.494615 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.493232    4496 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 02:47:39.494615 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.493235    4496 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 02:47:39.494615 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.493238    4496 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 02:47:39.494615 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.493241    4496 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 02:47:39.494615 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.493243    4496 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 02:47:39.494615 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.493246    4496 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 02:47:39.494615 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.493249    4496 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 02:47:39.494615 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.493251    4496 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 02:47:39.494615 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.493254    4496 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 02:47:39.494615 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.493256    4496 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 02:47:39.494615 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.493259    4496 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 02:47:39.494615 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.493262    4496 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 02:47:39.494615 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.493264    4496 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 02:47:39.494615 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.493267    4496 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 02:47:39.494615 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.493269    4496 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 02:47:39.495103 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.493272    4496 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 02:47:39.495103 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.493275    4496 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 02:47:39.495103 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.493277    4496 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 02:47:39.495103 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.493280    4496 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 02:47:39.495103 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.493283    4496 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 02:47:39.495103 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.493285    4496 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 02:47:39.495103 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.493288    4496 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 02:47:39.495103 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.493291    4496 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 02:47:39.495103 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.493293    4496 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 02:47:39.495103 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.493297    4496 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 02:47:39.495103 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.493300    4496 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 02:47:39.495103 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.493303    4496 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 02:47:39.495103 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.493306    4496 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 02:47:39.495103 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.493309    4496 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 02:47:39.495103 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.493312    4496 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 02:47:39.495103 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.493315    4496 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 02:47:39.495103 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.493317    4496 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 02:47:39.495103 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.493320    4496 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 02:47:39.495103 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.493322    4496 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 02:47:39.495560 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.493325    4496 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 02:47:39.495560 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.493327    4496 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 02:47:39.495560 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.493330    4496 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 02:47:39.495560 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.493332    4496 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 02:47:39.495560 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.493335    4496 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 02:47:39.495560 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.493339    4496 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 02:47:39.495560 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.493341    4496 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 02:47:39.495560 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494123    4496 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 02:47:39.495560 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494133    4496 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 02:47:39.495560 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494137    4496 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 02:47:39.495560 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494140    4496 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 02:47:39.495560 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494142    4496 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 02:47:39.495560 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494145    4496 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 02:47:39.495560 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494148    4496 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 02:47:39.495560 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494150    4496 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 02:47:39.495560 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494153    4496 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 02:47:39.495560 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494156    4496 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 02:47:39.495560 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494159    4496 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 02:47:39.495560 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494161    4496 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 02:47:39.495560 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494164    4496 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 02:47:39.496058 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494167    4496 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 02:47:39.496058 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494169    4496 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 02:47:39.496058 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494172    4496 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 02:47:39.496058 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494175    4496 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 02:47:39.496058 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494178    4496 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 02:47:39.496058 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494182    4496 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 02:47:39.496058 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494184    4496 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 02:47:39.496058 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494187    4496 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 02:47:39.496058 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494190    4496 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 02:47:39.496058 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494193    4496 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 02:47:39.496058 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494195    4496 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 02:47:39.496058 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494198    4496 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 02:47:39.496058 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494201    4496 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 02:47:39.496058 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494203    4496 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 02:47:39.496058 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494206    4496 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 02:47:39.496058 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494209    4496 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 02:47:39.496058 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494212    4496 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 02:47:39.496058 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494214    4496 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 02:47:39.496058 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494217    4496 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 02:47:39.496526 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494219    4496 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 02:47:39.496526 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494222    4496 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 02:47:39.496526 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494224    4496 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 02:47:39.496526 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494227    4496 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 02:47:39.496526 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494230    4496 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 02:47:39.496526 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494233    4496 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 02:47:39.496526 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494236    4496 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 02:47:39.496526 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494238    4496 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 02:47:39.496526 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494241    4496 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 02:47:39.496526 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494244    4496 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 02:47:39.496526 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494246    4496 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 02:47:39.496526 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494249    4496 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 02:47:39.496526 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494251    4496 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 02:47:39.496526 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494254    4496 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 02:47:39.496526 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494257    4496 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 02:47:39.496526 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494259    4496 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 02:47:39.496526 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494263    4496 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 02:47:39.496526 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494269    4496 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 02:47:39.496526 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494272    4496 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 02:47:39.496994 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494275    4496 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 02:47:39.496994 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494278    4496 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 02:47:39.496994 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494280    4496 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 02:47:39.496994 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494283    4496 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 02:47:39.496994 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494286    4496 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 02:47:39.496994 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494288    4496 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 02:47:39.496994 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494291    4496 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 02:47:39.496994 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494293    4496 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 02:47:39.496994 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494296    4496 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 02:47:39.496994 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494298    4496 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 02:47:39.496994 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494301    4496 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 02:47:39.496994 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494303    4496 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 02:47:39.496994 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494306    4496 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 02:47:39.496994 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494308    4496 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 02:47:39.496994 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494311    4496 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 02:47:39.496994 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494314    4496 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 02:47:39.496994 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494317    4496 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 02:47:39.496994 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494320    4496 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 02:47:39.496994 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494323    4496 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 02:47:39.496994 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494325    4496 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 02:47:39.497490 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494329    4496 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 02:47:39.497490 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494331    4496 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 02:47:39.497490 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494334    4496 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 02:47:39.497490 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494337    4496 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 02:47:39.497490 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494340    4496 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 02:47:39.497490 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494343    4496 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 02:47:39.497490 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494345    4496 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 02:47:39.497490 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494348    4496 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 02:47:39.497490 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494350    4496 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 02:47:39.497490 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494353    4496 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 02:47:39.497490 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494355    4496 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 02:47:39.497490 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494358    4496 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 02:47:39.497490 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494362    4496 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 02:47:39.497490 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494366    4496 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 02:47:39.497490 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494368    4496 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 02:47:39.497490 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494371    4496 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 02:47:39.497490 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494374    4496 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 02:47:39.497490 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494376    4496 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 02:47:39.497490 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494379    4496 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 02:47:39.497957 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494382    4496 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 02:47:39.497957 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494385    4496 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 02:47:39.497957 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494388    4496 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 02:47:39.497957 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494391    4496 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 02:47:39.497957 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494393    4496 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 02:47:39.497957 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494396    4496 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 02:47:39.497957 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494399    4496 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 02:47:39.497957 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494401    4496 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 02:47:39.497957 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494404    4496 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 02:47:39.497957 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494407    4496 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 02:47:39.497957 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494409    4496 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 02:47:39.497957 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494412    4496 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 02:47:39.497957 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494415    4496 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 02:47:39.497957 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494417    4496 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 02:47:39.497957 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494420    4496 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 02:47:39.497957 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494423    4496 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 02:47:39.497957 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494882    4496 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 02:47:39.497957 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494887    4496 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 02:47:39.497957 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494890    4496 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 02:47:39.497957 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494892    4496 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 02:47:39.498464 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494895    4496 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 02:47:39.498464 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494898    4496 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 02:47:39.498464 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494901    4496 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 02:47:39.498464 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494903    4496 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 02:47:39.498464 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494912    4496 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 02:47:39.498464 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494915    4496 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 02:47:39.498464 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494918    4496 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 02:47:39.498464 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494921    4496 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 02:47:39.498464 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494924    4496 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 02:47:39.498464 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494926    4496 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 02:47:39.498464 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494929    4496 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 02:47:39.498464 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494932    4496 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 02:47:39.498464 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494935    4496 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 02:47:39.498464 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494938    4496 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 02:47:39.498464 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494940    4496 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 02:47:39.498464 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494943    4496 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 02:47:39.498464 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494946    4496 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 02:47:39.498464 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494948    4496 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 02:47:39.498464 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494951    4496 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 02:47:39.498464 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494954    4496 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 02:47:39.498980 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494956    4496 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 02:47:39.498980 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494958    4496 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 02:47:39.498980 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494961    4496 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 02:47:39.498980 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494964    4496 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 02:47:39.498980 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494966    4496 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 02:47:39.498980 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494969    4496 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 02:47:39.498980 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494972    4496 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 02:47:39.498980 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494976    4496 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 02:47:39.498980 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494979    4496 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 02:47:39.498980 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494981    4496 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 02:47:39.498980 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494984    4496 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 02:47:39.498980 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494987    4496 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 02:47:39.498980 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494990    4496 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 02:47:39.498980 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494992    4496 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 02:47:39.498980 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494995    4496 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 02:47:39.498980 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.494998    4496 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 02:47:39.498980 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.495000    4496 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 02:47:39.498980 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.495027    4496 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 02:47:39.498980 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.495032    4496 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 02:47:39.499451 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.495037    4496 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 02:47:39.499451 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.495041    4496 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 02:47:39.499451 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.495049    4496 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 02:47:39.499451 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.495052    4496 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 02:47:39.499451 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.495057    4496 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 02:47:39.499451 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.495060    4496 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 02:47:39.499451 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.495063    4496 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 02:47:39.499451 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.495065    4496 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 02:47:39.499451 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.495068    4496 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 02:47:39.499451 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.495071    4496 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 02:47:39.499451 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.495074    4496 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 02:47:39.499451 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.495076    4496 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 02:47:39.499451 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.495079    4496 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 02:47:39.499451 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.495082    4496 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 02:47:39.499451 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.495085    4496 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 02:47:39.499451 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.495088    4496 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 02:47:39.499451 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.495090    4496 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 02:47:39.499451 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.495093    4496 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 02:47:39.499451 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.495096    4496 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 02:47:39.499902 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.495098    4496 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 02:47:39.499902 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.495103    4496 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 02:47:39.499902 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.495106    4496 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 02:47:39.499902 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.495109    4496 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 02:47:39.499902 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.495112    4496 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 02:47:39.499902 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.495115    4496 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 02:47:39.499902 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.495118    4496 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 02:47:39.499902 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.495120    4496 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 02:47:39.499902 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.495123    4496 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 02:47:39.499902 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.495126    4496 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 02:47:39.499902 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.495128    4496 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 02:47:39.499902 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.495131    4496 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 02:47:39.499902 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.495140    4496 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 02:47:39.499902 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.495143    4496 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 02:47:39.499902 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.495146    4496 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 02:47:39.499902 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.495152    4496 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 02:47:39.499902 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.495154    4496 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 02:47:39.499902 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.495158    4496 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 02:47:39.499902 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.495161    4496 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 02:47:39.500387 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.495163    4496 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 02:47:39.500387 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.495166    4496 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 02:47:39.500387 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.495169    4496 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 02:47:39.500387 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.495171    4496 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 02:47:39.500387 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.495174    4496 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 02:47:39.500387 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.495176    4496 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 02:47:39.500387 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.495179    4496 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 02:47:39.500387 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.495182    4496 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 02:47:39.500387 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.495184    4496 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 02:47:39.500387 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.495187    4496 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 02:47:39.500387 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.495189    4496 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 02:47:39.500387 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.495192    4496 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 02:47:39.500387 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.495194    4496 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 02:47:39.500387 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.495197    4496 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 02:47:39.500387 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.495200    4496 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 02:47:39.500387 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.495202    4496 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 02:47:39.500387 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.495205    4496 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 02:47:39.500387 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.495207    4496 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 02:47:39.500387 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.495210    4496 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 02:47:39.500387 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.495213    4496 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 02:47:39.500867 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.495216    4496 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 02:47:39.500867 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.495219    4496 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 02:47:39.500867 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.495221    4496 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 02:47:39.500867 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.495224    4496 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 02:47:39.500867 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.495226    4496 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 02:47:39.500867 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495330    4496 flags.go:64] FLAG: --address="0.0.0.0"
Nov 02 02:47:39.500867 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495341    4496 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 02 02:47:39.500867 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495351    4496 flags.go:64] FLAG: --anonymous-auth="true"
Nov 02 02:47:39.500867 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495358    4496 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 02 02:47:39.500867 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495362    4496 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 02 02:47:39.500867 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495367    4496 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 02 02:47:39.500867 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495371    4496 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 02 02:47:39.500867 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495375    4496 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 02 02:47:39.500867 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495379    4496 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 02 02:47:39.500867 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495382    4496 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 02 02:47:39.500867 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495385    4496 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 02 02:47:39.500867 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495389    4496 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 02 02:47:39.500867 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495392    4496 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 02 02:47:39.500867 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495395    4496 flags.go:64] FLAG: --cgroup-root=""
Nov 02 02:47:39.500867 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495398    4496 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 02 02:47:39.500867 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495401    4496 flags.go:64] FLAG: --client-ca-file=""
Nov 02 02:47:39.500867 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495404    4496 flags.go:64] FLAG: --cloud-provider="external"
Nov 02 02:47:39.500867 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495416    4496 flags.go:64] FLAG: --cluster-dns="[]"
Nov 02 02:47:39.501541 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495424    4496 flags.go:64] FLAG: --cluster-domain=""
Nov 02 02:47:39.501541 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495427    4496 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 02 02:47:39.501541 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495431    4496 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 02 02:47:39.501541 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495434    4496 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 02 02:47:39.501541 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495437    4496 flags.go:64] FLAG: --container-log-max-files="5"
Nov 02 02:47:39.501541 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495441    4496 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 02 02:47:39.501541 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495444    4496 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 02 02:47:39.501541 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495447    4496 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 02 02:47:39.501541 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495451    4496 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 02 02:47:39.501541 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495454    4496 flags.go:64] FLAG: --contention-profiling="false"
Nov 02 02:47:39.501541 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495457    4496 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 02 02:47:39.501541 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495460    4496 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 02 02:47:39.501541 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495463    4496 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 02 02:47:39.501541 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495466    4496 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 02 02:47:39.501541 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495470    4496 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 02 02:47:39.501541 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495474    4496 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 02 02:47:39.501541 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495483    4496 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 02 02:47:39.501541 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495488    4496 flags.go:64] FLAG: --enable-load-reader="false"
Nov 02 02:47:39.501541 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495491    4496 flags.go:64] FLAG: --enable-server="true"
Nov 02 02:47:39.501541 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495495    4496 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 02 02:47:39.501541 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495502    4496 flags.go:64] FLAG: --event-burst="100"
Nov 02 02:47:39.501541 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495506    4496 flags.go:64] FLAG: --event-qps="50"
Nov 02 02:47:39.501541 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495509    4496 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 02 02:47:39.501541 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495512    4496 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 02 02:47:39.502470 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495515    4496 flags.go:64] FLAG: --eviction-hard=""
Nov 02 02:47:39.502470 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495519    4496 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 02 02:47:39.502470 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495522    4496 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 02 02:47:39.502470 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495525    4496 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 02 02:47:39.502470 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495528    4496 flags.go:64] FLAG: --eviction-soft=""
Nov 02 02:47:39.502470 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495531    4496 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 02 02:47:39.502470 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495534    4496 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 02 02:47:39.502470 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495537    4496 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 02 02:47:39.502470 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495540    4496 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 02 02:47:39.502470 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495543    4496 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 02 02:47:39.502470 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495546    4496 flags.go:64] FLAG: --fail-swap-on="true"
Nov 02 02:47:39.502470 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495550    4496 flags.go:64] FLAG: --feature-gates=""
Nov 02 02:47:39.502470 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495554    4496 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 02 02:47:39.502470 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495557    4496 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 02 02:47:39.502470 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495560    4496 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 02 02:47:39.502470 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495564    4496 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 02 02:47:39.502470 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495567    4496 flags.go:64] FLAG: --healthz-port="10248"
Nov 02 02:47:39.502470 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495570    4496 flags.go:64] FLAG: --help="false"
Nov 02 02:47:39.502470 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495573    4496 flags.go:64] FLAG: --hostname-override="ip-10-0-133-163.us-west-1.compute.internal"
Nov 02 02:47:39.502470 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495577    4496 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 02 02:47:39.502470 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495580    4496 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 02 02:47:39.502470 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495583    4496 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 02 02:47:39.502470 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495586    4496 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 02 02:47:39.503392 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495590    4496 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 02 02:47:39.503392 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495593    4496 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 02 02:47:39.503392 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495603    4496 flags.go:64] FLAG: --image-service-endpoint=""
Nov 02 02:47:39.503392 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495607    4496 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 02 02:47:39.503392 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495611    4496 flags.go:64] FLAG: --kube-api-burst="100"
Nov 02 02:47:39.503392 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495614    4496 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 02 02:47:39.503392 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495618    4496 flags.go:64] FLAG: --kube-api-qps="50"
Nov 02 02:47:39.503392 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495621    4496 flags.go:64] FLAG: --kube-reserved=""
Nov 02 02:47:39.503392 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495624    4496 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 02 02:47:39.503392 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495627    4496 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 02 02:47:39.503392 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495630    4496 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 02 02:47:39.503392 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495633    4496 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 02 02:47:39.503392 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495636    4496 flags.go:64] FLAG: --lock-file=""
Nov 02 02:47:39.503392 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495638    4496 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 02 02:47:39.503392 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495642    4496 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 02 02:47:39.503392 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495645    4496 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 02 02:47:39.503392 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495650    4496 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 02 02:47:39.503392 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495654    4496 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 02 02:47:39.503392 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495657    4496 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 02 02:47:39.503392 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495660    4496 flags.go:64] FLAG: --logging-format="text"
Nov 02 02:47:39.503392 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495663    4496 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 02 02:47:39.503392 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495666    4496 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 02 02:47:39.503392 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495669    4496 flags.go:64] FLAG: --manifest-url=""
Nov 02 02:47:39.503392 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495672    4496 flags.go:64] FLAG: --manifest-url-header=""
Nov 02 02:47:39.503392 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495676    4496 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 02 02:47:39.504128 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495679    4496 flags.go:64] FLAG: --max-open-files="1000000"
Nov 02 02:47:39.504128 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495683    4496 flags.go:64] FLAG: --max-pods="110"
Nov 02 02:47:39.504128 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495686    4496 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 02 02:47:39.504128 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495689    4496 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 02 02:47:39.504128 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495693    4496 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 02 02:47:39.504128 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495696    4496 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 02 02:47:39.504128 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495699    4496 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 02 02:47:39.504128 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495702    4496 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 02 02:47:39.504128 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495705    4496 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 02 02:47:39.504128 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495713    4496 flags.go:64] FLAG: --node-status-max-images="50"
Nov 02 02:47:39.504128 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495722    4496 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 02 02:47:39.504128 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495727    4496 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 02 02:47:39.504128 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495730    4496 flags.go:64] FLAG: --pod-cidr=""
Nov 02 02:47:39.504128 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495733    4496 flags.go:64] FLAG: --pod-manifest-path=""
Nov 02 02:47:39.504128 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495736    4496 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 02 02:47:39.504128 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495739    4496 flags.go:64] FLAG: --pods-per-core="0"
Nov 02 02:47:39.504128 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495742    4496 flags.go:64] FLAG: --port="10250"
Nov 02 02:47:39.504128 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495745    4496 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 02 02:47:39.504128 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495748    4496 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0b4f9600c3b715dbd"
Nov 02 02:47:39.504128 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495751    4496 flags.go:64] FLAG: --qos-reserved=""
Nov 02 02:47:39.504128 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495754    4496 flags.go:64] FLAG: --read-only-port="10255"
Nov 02 02:47:39.504128 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495757    4496 flags.go:64] FLAG: --register-node="true"
Nov 02 02:47:39.504128 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495760    4496 flags.go:64] FLAG: --register-with-taints=""
Nov 02 02:47:39.504128 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495767    4496 flags.go:64] FLAG: --registry-burst="10"
Nov 02 02:47:39.504128 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495770    4496 flags.go:64] FLAG: --registry-qps="5"
Nov 02 02:47:39.504128 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495773    4496 flags.go:64] FLAG: --reserved-cpus=""
Nov 02 02:47:39.504811 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495775    4496 flags.go:64] FLAG: --reserved-memory=""
Nov 02 02:47:39.504811 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495779    4496 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 02 02:47:39.504811 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495782    4496 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 02 02:47:39.504811 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495785    4496 flags.go:64] FLAG: --rotate-certificates="false"
Nov 02 02:47:39.504811 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495788    4496 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 02 02:47:39.504811 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495791    4496 flags.go:64] FLAG: --runonce="false"
Nov 02 02:47:39.504811 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495794    4496 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 02 02:47:39.504811 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495797    4496 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 02 02:47:39.504811 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495800    4496 flags.go:64] FLAG: --seccomp-default="false"
Nov 02 02:47:39.504811 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495803    4496 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 02 02:47:39.504811 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495806    4496 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 02 02:47:39.504811 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495810    4496 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 02 02:47:39.504811 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495813    4496 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 02 02:47:39.504811 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495816    4496 flags.go:64] FLAG: --storage-driver-password="root"
Nov 02 02:47:39.504811 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495819    4496 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 02 02:47:39.504811 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495823    4496 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 02 02:47:39.504811 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495828    4496 flags.go:64] FLAG: --storage-driver-user="root"
Nov 02 02:47:39.504811 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495842    4496 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 02 02:47:39.504811 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495848    4496 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 02 02:47:39.504811 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495853    4496 flags.go:64] FLAG: --system-cgroups=""
Nov 02 02:47:39.504811 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495859    4496 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 02 02:47:39.504811 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495865    4496 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 02 02:47:39.504811 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495868    4496 flags.go:64] FLAG: --tls-cert-file=""
Nov 02 02:47:39.504811 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495871    4496 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 02 02:47:39.504811 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495877    4496 flags.go:64] FLAG: --tls-min-version=""
Nov 02 02:47:39.505440 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495879    4496 flags.go:64] FLAG: --tls-private-key-file=""
Nov 02 02:47:39.505440 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495882    4496 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 02 02:47:39.505440 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495885    4496 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 02 02:47:39.505440 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495888    4496 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 02 02:47:39.505440 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495891    4496 flags.go:64] FLAG: --v="2"
Nov 02 02:47:39.505440 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495896    4496 flags.go:64] FLAG: --version="false"
Nov 02 02:47:39.505440 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495900    4496 flags.go:64] FLAG: --vmodule=""
Nov 02 02:47:39.505440 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495904    4496 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 02 02:47:39.505440 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.495908    4496 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 02 02:47:39.505440 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496075    4496 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 02:47:39.505440 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496081    4496 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 02:47:39.505440 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496084    4496 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 02:47:39.505440 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496087    4496 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 02:47:39.505440 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496105    4496 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 02:47:39.505440 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496110    4496 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 02:47:39.505440 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496113    4496 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 02:47:39.505440 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496117    4496 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 02:47:39.505440 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496119    4496 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 02:47:39.505440 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496122    4496 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 02:47:39.505440 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496125    4496 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 02:47:39.505440 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496129    4496 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 02:47:39.505440 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496132    4496 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 02:47:39.506002 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496135    4496 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 02:47:39.506002 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496138    4496 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 02:47:39.506002 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496143    4496 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 02:47:39.506002 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496146    4496 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 02:47:39.506002 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496149    4496 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 02:47:39.506002 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496151    4496 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 02:47:39.506002 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496154    4496 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 02:47:39.506002 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496157    4496 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 02:47:39.506002 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496159    4496 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 02:47:39.506002 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496162    4496 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 02:47:39.506002 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496164    4496 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 02:47:39.506002 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496167    4496 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 02:47:39.506002 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496170    4496 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 02:47:39.506002 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496172    4496 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 02:47:39.506002 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496175    4496 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 02:47:39.506002 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496177    4496 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 02:47:39.506002 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496180    4496 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 02:47:39.506002 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496182    4496 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 02:47:39.506002 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496185    4496 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 02:47:39.506002 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496187    4496 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 02:47:39.506615 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496190    4496 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 02:47:39.506615 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496193    4496 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 02:47:39.506615 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496195    4496 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 02:47:39.506615 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496198    4496 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 02:47:39.506615 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496200    4496 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 02:47:39.506615 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496203    4496 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 02:47:39.506615 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496205    4496 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 02:47:39.506615 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496208    4496 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 02:47:39.506615 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496210    4496 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 02:47:39.506615 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496213    4496 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 02:47:39.506615 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496215    4496 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 02:47:39.506615 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496218    4496 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 02:47:39.506615 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496221    4496 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 02:47:39.506615 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496225    4496 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 02:47:39.506615 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496231    4496 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 02:47:39.506615 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496234    4496 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 02:47:39.506615 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496238    4496 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 02:47:39.506615 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496240    4496 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 02:47:39.506615 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496243    4496 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 02:47:39.507188 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496246    4496 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 02:47:39.507188 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496248    4496 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 02:47:39.507188 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496251    4496 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 02:47:39.507188 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496253    4496 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 02:47:39.507188 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496256    4496 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 02:47:39.507188 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496259    4496 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 02:47:39.507188 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496261    4496 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 02:47:39.507188 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496264    4496 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 02:47:39.507188 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496266    4496 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 02:47:39.507188 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496269    4496 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 02:47:39.507188 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496272    4496 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 02:47:39.507188 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496274    4496 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 02:47:39.507188 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496277    4496 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 02:47:39.507188 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496280    4496 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 02:47:39.507188 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496284    4496 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 02:47:39.507188 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496288    4496 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 02:47:39.507188 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496291    4496 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 02:47:39.507188 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496294    4496 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 02:47:39.507188 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496296    4496 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 02:47:39.507731 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496299    4496 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 02:47:39.507731 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496302    4496 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 02:47:39.507731 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496304    4496 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 02:47:39.507731 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496307    4496 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 02:47:39.507731 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496310    4496 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 02:47:39.507731 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496312    4496 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 02:47:39.507731 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496315    4496 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 02:47:39.507731 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496317    4496 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 02:47:39.507731 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496321    4496 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 02:47:39.507731 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496324    4496 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 02:47:39.507731 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496326    4496 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 02:47:39.507731 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496329    4496 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 02:47:39.507731 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496332    4496 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 02:47:39.507731 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496335    4496 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 02:47:39.507731 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496337    4496 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 02:47:39.507731 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496340    4496 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 02:47:39.507731 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496342    4496 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 02:47:39.507731 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496345    4496 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 02:47:39.507731 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496347    4496 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 02:47:39.508299 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496350    4496 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 02:47:39.508299 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496352    4496 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 02:47:39.508299 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496355    4496 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 02:47:39.508299 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496357    4496 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 02:47:39.508299 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496360    4496 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 02:47:39.508299 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496362    4496 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 02:47:39.508299 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496365    4496 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 02:47:39.508299 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496367    4496 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 02:47:39.508299 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496370    4496 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 02:47:39.508299 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496372    4496 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 02:47:39.508299 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496375    4496 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 02:47:39.508299 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496378    4496 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 02:47:39.508299 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496380    4496 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 02:47:39.508299 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496383    4496 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 02:47:39.508299 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496385    4496 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 02:47:39.508299 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496388    4496 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 02:47:39.508843 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.496415    4496 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 02:47:39.508843 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496450    4496 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 02:47:39.508843 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496454    4496 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 02:47:39.508843 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496457    4496 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 02:47:39.508843 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496459    4496 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 02:47:39.508843 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496464    4496 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 02:47:39.508843 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496466    4496 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 02:47:39.508843 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496469    4496 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 02:47:39.508843 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496472    4496 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 02:47:39.508843 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496475    4496 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 02:47:39.508843 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496478    4496 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 02:47:39.508843 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496482    4496 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 02:47:39.508843 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496486    4496 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 02:47:39.508843 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496489    4496 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 02:47:39.508843 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496492    4496 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 02:47:39.508843 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496495    4496 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 02:47:39.509313 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496497    4496 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 02:47:39.509313 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496500    4496 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 02:47:39.509313 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496502    4496 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 02:47:39.509313 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496505    4496 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 02:47:39.509313 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496508    4496 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 02:47:39.509313 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496510    4496 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 02:47:39.509313 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496513    4496 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 02:47:39.509313 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496515    4496 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 02:47:39.509313 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496518    4496 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 02:47:39.509313 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496522    4496 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 02:47:39.509313 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496525    4496 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 02:47:39.509313 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496527    4496 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 02:47:39.509313 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496530    4496 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 02:47:39.509313 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496533    4496 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 02:47:39.509313 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496537    4496 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 02:47:39.509313 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496540    4496 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 02:47:39.509313 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496543    4496 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 02:47:39.509313 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496546    4496 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 02:47:39.509313 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496549    4496 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 02:47:39.509802 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496551    4496 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 02:47:39.509802 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496554    4496 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 02:47:39.509802 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496558    4496 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 02:47:39.509802 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496560    4496 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 02:47:39.509802 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496564    4496 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 02:47:39.509802 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496566    4496 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 02:47:39.509802 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496569    4496 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 02:47:39.509802 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496572    4496 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 02:47:39.509802 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496575    4496 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 02:47:39.509802 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496577    4496 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 02:47:39.509802 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496580    4496 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 02:47:39.509802 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496582    4496 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 02:47:39.509802 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496585    4496 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 02:47:39.509802 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496588    4496 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 02:47:39.509802 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496590    4496 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 02:47:39.509802 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496593    4496 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 02:47:39.509802 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496595    4496 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 02:47:39.509802 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496598    4496 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 02:47:39.509802 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496600    4496 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 02:47:39.510275 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496603    4496 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 02:47:39.510275 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496606    4496 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 02:47:39.510275 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496609    4496 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 02:47:39.510275 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496613    4496 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 02:47:39.510275 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496616    4496 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 02:47:39.510275 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496619    4496 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 02:47:39.510275 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496621    4496 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 02:47:39.510275 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496624    4496 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 02:47:39.510275 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496627    4496 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 02:47:39.510275 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496630    4496 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 02:47:39.510275 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496632    4496 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 02:47:39.510275 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496635    4496 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 02:47:39.510275 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496637    4496 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 02:47:39.510275 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496640    4496 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 02:47:39.510275 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496643    4496 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 02:47:39.510275 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496645    4496 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 02:47:39.510275 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496648    4496 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 02:47:39.510275 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496650    4496 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 02:47:39.510275 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496653    4496 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 02:47:39.510763 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496656    4496 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 02:47:39.510763 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496658    4496 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 02:47:39.510763 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496665    4496 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 02:47:39.510763 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496668    4496 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 02:47:39.510763 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496670    4496 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 02:47:39.510763 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496673    4496 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 02:47:39.510763 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496675    4496 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 02:47:39.510763 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496678    4496 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 02:47:39.510763 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496681    4496 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 02:47:39.510763 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496683    4496 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 02:47:39.510763 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496686    4496 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 02:47:39.510763 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496689    4496 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 02:47:39.510763 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496693    4496 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 02:47:39.510763 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496695    4496 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 02:47:39.510763 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496698    4496 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 02:47:39.510763 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496700    4496 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 02:47:39.510763 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496704    4496 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 02:47:39.510763 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496707    4496 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 02:47:39.510763 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496709    4496 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 02:47:39.510763 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496712    4496 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 02:47:39.511271 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496715    4496 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 02:47:39.511271 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496717    4496 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 02:47:39.511271 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496720    4496 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 02:47:39.511271 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496723    4496 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 02:47:39.511271 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496725    4496 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 02:47:39.511271 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496728    4496 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 02:47:39.511271 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496730    4496 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 02:47:39.511271 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496733    4496 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 02:47:39.511271 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496735    4496 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 02:47:39.511271 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496738    4496 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 02:47:39.511271 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496741    4496 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 02:47:39.511271 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496743    4496 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 02:47:39.511271 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496746    4496 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 02:47:39.511271 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.496748    4496 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 02:47:39.511271 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.500295    4496 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 02 02:47:39.511271 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.500309    4496 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 02 02:47:39.511271 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.500328    4496 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 02 02:47:39.511271 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.500332    4496 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 02 02:47:39.511271 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500400    4496 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 02:47:39.511735 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500407    4496 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 02:47:39.511735 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500410    4496 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 02:47:39.511735 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500413    4496 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 02:47:39.511735 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500416    4496 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 02:47:39.511735 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500420    4496 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 02:47:39.511735 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500422    4496 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 02:47:39.511735 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500425    4496 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 02:47:39.511735 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500428    4496 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 02:47:39.511735 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500431    4496 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 02:47:39.511735 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500434    4496 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 02:47:39.511735 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500436    4496 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 02:47:39.511735 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500439    4496 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 02:47:39.511735 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500443    4496 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 02:47:39.511735 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500447    4496 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 02:47:39.511735 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500450    4496 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 02:47:39.511735 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500452    4496 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 02:47:39.511735 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500455    4496 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 02:47:39.511735 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500458    4496 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 02:47:39.511735 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500460    4496 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 02:47:39.511735 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500463    4496 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 02:47:39.512338 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500466    4496 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 02:47:39.512338 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500468    4496 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 02:47:39.512338 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500471    4496 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 02:47:39.512338 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500474    4496 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 02:47:39.512338 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500477    4496 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 02:47:39.512338 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500480    4496 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 02:47:39.512338 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500482    4496 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 02:47:39.512338 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500485    4496 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 02:47:39.512338 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500489    4496 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 02:47:39.512338 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500492    4496 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 02:47:39.512338 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500494    4496 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 02:47:39.512338 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500497    4496 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 02:47:39.512338 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500500    4496 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 02:47:39.512338 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500502    4496 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 02:47:39.512338 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500505    4496 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 02:47:39.512338 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500507    4496 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 02:47:39.512338 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500510    4496 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 02:47:39.512338 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500513    4496 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 02:47:39.512338 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500515    4496 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 02:47:39.512338 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500518    4496 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 02:47:39.512814 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500520    4496 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 02:47:39.512814 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500523    4496 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 02:47:39.512814 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500526    4496 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 02:47:39.512814 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500528    4496 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 02:47:39.512814 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500531    4496 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 02:47:39.512814 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500534    4496 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 02:47:39.512814 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500536    4496 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 02:47:39.512814 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500539    4496 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 02:47:39.512814 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500541    4496 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 02:47:39.512814 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500544    4496 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 02:47:39.512814 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500546    4496 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 02:47:39.512814 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500549    4496 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 02:47:39.512814 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500552    4496 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 02:47:39.512814 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500554    4496 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 02:47:39.512814 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500557    4496 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 02:47:39.512814 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500559    4496 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 02:47:39.512814 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500562    4496 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 02:47:39.512814 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500565    4496 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 02:47:39.512814 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500567    4496 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 02:47:39.512814 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500570    4496 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 02:47:39.513298 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500574    4496 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 02:47:39.513298 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500577    4496 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 02:47:39.513298 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500579    4496 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 02:47:39.513298 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500582    4496 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 02:47:39.513298 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500584    4496 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 02:47:39.513298 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500587    4496 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 02:47:39.513298 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500590    4496 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 02:47:39.513298 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500592    4496 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 02:47:39.513298 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500595    4496 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 02:47:39.513298 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500597    4496 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 02:47:39.513298 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500600    4496 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 02:47:39.513298 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500602    4496 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 02:47:39.513298 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500605    4496 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 02:47:39.513298 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500607    4496 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 02:47:39.513298 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500610    4496 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 02:47:39.513298 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500612    4496 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 02:47:39.513298 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500615    4496 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 02:47:39.513298 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500617    4496 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 02:47:39.513298 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500620    4496 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 02:47:39.513298 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500623    4496 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 02:47:39.513790 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500625    4496 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 02:47:39.513790 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500628    4496 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 02:47:39.513790 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500631    4496 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 02:47:39.513790 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500634    4496 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 02:47:39.513790 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500636    4496 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 02:47:39.513790 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500639    4496 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 02:47:39.513790 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500642    4496 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 02:47:39.513790 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500645    4496 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 02:47:39.513790 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500647    4496 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 02:47:39.513790 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500650    4496 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 02:47:39.513790 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500652    4496 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 02:47:39.513790 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500655    4496 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 02:47:39.513790 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500658    4496 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 02:47:39.513790 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500660    4496 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 02:47:39.513790 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500663    4496 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 02:47:39.513790 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500666    4496 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 02:47:39.513790 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500668    4496 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 02:47:39.513790 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500671    4496 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 02:47:39.513790 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500674    4496 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 02:47:39.514257 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500676    4496 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 02:47:39.514257 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500678    4496 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 02:47:39.514257 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500681    4496 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 02:47:39.514257 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500684    4496 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 02:47:39.514257 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500686    4496 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 02:47:39.514257 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500688    4496 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 02:47:39.514257 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.500717    4496 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 02:47:39.514257 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500855    4496 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 02:47:39.514257 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500859    4496 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 02:47:39.514257 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500862    4496 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 02:47:39.514257 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500865    4496 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 02:47:39.514257 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500868    4496 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 02:47:39.514257 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500872    4496 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 02:47:39.514257 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500875    4496 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 02:47:39.514257 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500877    4496 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 02:47:39.514611 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500880    4496 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 02:47:39.514611 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500883    4496 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 02:47:39.514611 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500885    4496 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 02:47:39.514611 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500888    4496 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 02:47:39.514611 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500891    4496 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 02:47:39.514611 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500893    4496 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 02:47:39.514611 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500896    4496 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 02:47:39.514611 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500899    4496 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 02:47:39.514611 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500901    4496 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 02:47:39.514611 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500904    4496 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 02:47:39.514611 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500908    4496 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 02:47:39.514611 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500912    4496 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 02:47:39.514611 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500915    4496 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 02:47:39.514611 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500919    4496 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 02:47:39.514611 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500922    4496 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 02:47:39.514611 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500924    4496 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 02:47:39.514611 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500927    4496 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 02:47:39.514611 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500929    4496 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 02:47:39.514611 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500932    4496 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 02:47:39.515081 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500935    4496 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 02:47:39.515081 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500937    4496 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 02:47:39.515081 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500940    4496 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 02:47:39.515081 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500942    4496 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 02:47:39.515081 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500945    4496 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 02:47:39.515081 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500947    4496 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 02:47:39.515081 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500950    4496 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 02:47:39.515081 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500952    4496 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 02:47:39.515081 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500955    4496 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 02:47:39.515081 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500958    4496 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 02:47:39.515081 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500961    4496 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 02:47:39.515081 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500963    4496 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 02:47:39.515081 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500966    4496 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 02:47:39.515081 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500968    4496 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 02:47:39.515081 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500971    4496 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 02:47:39.515081 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500973    4496 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 02:47:39.515081 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500976    4496 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 02:47:39.515081 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500978    4496 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 02:47:39.515081 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500981    4496 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 02:47:39.515081 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500983    4496 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 02:47:39.515575 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500986    4496 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 02:47:39.515575 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500988    4496 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 02:47:39.515575 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500991    4496 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 02:47:39.515575 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500995    4496 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 02:47:39.515575 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.500998    4496 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 02:47:39.515575 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501001    4496 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 02:47:39.515575 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501004    4496 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 02:47:39.515575 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501008    4496 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 02:47:39.515575 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501010    4496 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 02:47:39.515575 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501028    4496 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 02:47:39.515575 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501031    4496 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 02:47:39.515575 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501034    4496 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 02:47:39.515575 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501037    4496 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 02:47:39.515575 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501040    4496 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 02:47:39.515575 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501043    4496 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 02:47:39.515575 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501046    4496 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 02:47:39.515575 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501048    4496 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 02:47:39.515575 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501051    4496 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 02:47:39.515575 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501056    4496 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 02:47:39.515575 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501059    4496 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 02:47:39.516087 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501061    4496 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 02:47:39.516087 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501064    4496 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 02:47:39.516087 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501067    4496 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 02:47:39.516087 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501069    4496 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 02:47:39.516087 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501072    4496 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 02:47:39.516087 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501075    4496 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 02:47:39.516087 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501077    4496 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 02:47:39.516087 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501080    4496 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 02:47:39.516087 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501083    4496 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 02:47:39.516087 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501085    4496 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 02:47:39.516087 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501088    4496 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 02:47:39.516087 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501091    4496 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 02:47:39.516087 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501094    4496 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 02:47:39.516087 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501096    4496 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 02:47:39.516087 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501099    4496 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 02:47:39.516087 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501102    4496 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 02:47:39.516087 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501104    4496 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 02:47:39.516087 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501109    4496 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 02:47:39.516087 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501111    4496 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 02:47:39.516087 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501114    4496 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 02:47:39.516563 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501117    4496 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 02:47:39.516563 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501120    4496 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 02:47:39.516563 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501123    4496 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 02:47:39.516563 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501125    4496 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 02:47:39.516563 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501128    4496 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 02:47:39.516563 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501131    4496 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 02:47:39.516563 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501133    4496 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 02:47:39.516563 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501136    4496 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 02:47:39.516563 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501139    4496 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 02:47:39.516563 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501141    4496 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 02:47:39.516563 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501146    4496 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 02:47:39.516563 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501148    4496 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 02:47:39.516563 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501151    4496 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 02:47:39.516563 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501154    4496 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 02:47:39.516563 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501156    4496 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 02:47:39.516563 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501159    4496 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 02:47:39.516563 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501162    4496 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 02:47:39.516563 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501164    4496 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 02:47:39.516563 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501167    4496 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 02:47:39.522838 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.501190    4496 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 02:47:39.522838 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501222    4496 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 02:47:39.522838 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501226    4496 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 02:47:39.522838 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501229    4496 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 02:47:39.522838 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501232    4496 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 02:47:39.522838 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501235    4496 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 02:47:39.522838 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501237    4496 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 02:47:39.522838 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501240    4496 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 02:47:39.522838 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501243    4496 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 02:47:39.522838 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501246    4496 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 02:47:39.522838 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501250    4496 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 02:47:39.522838 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501252    4496 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 02:47:39.522838 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501255    4496 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 02:47:39.522838 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501258    4496 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 02:47:39.522838 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501261    4496 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 02:47:39.522838 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501264    4496 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 02:47:39.523352 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501266    4496 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 02:47:39.523352 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501269    4496 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 02:47:39.523352 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501271    4496 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 02:47:39.523352 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501274    4496 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 02:47:39.523352 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501277    4496 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 02:47:39.523352 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501279    4496 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 02:47:39.523352 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501282    4496 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 02:47:39.523352 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501284    4496 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 02:47:39.523352 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501288    4496 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 02:47:39.523352 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501291    4496 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 02:47:39.523352 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501294    4496 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 02:47:39.523352 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501297    4496 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 02:47:39.523352 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501299    4496 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 02:47:39.523352 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501302    4496 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 02:47:39.523352 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501304    4496 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 02:47:39.523352 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501307    4496 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 02:47:39.523352 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501310    4496 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 02:47:39.523352 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501312    4496 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 02:47:39.523352 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501315    4496 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 02:47:39.523352 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501317    4496 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 02:47:39.523878 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501320    4496 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 02:47:39.523878 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501322    4496 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 02:47:39.523878 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501325    4496 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 02:47:39.523878 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501327    4496 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 02:47:39.523878 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501330    4496 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 02:47:39.523878 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501333    4496 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 02:47:39.523878 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501337    4496 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 02:47:39.523878 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501339    4496 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 02:47:39.523878 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501342    4496 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 02:47:39.523878 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501345    4496 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 02:47:39.523878 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501347    4496 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 02:47:39.523878 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501351    4496 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 02:47:39.523878 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501353    4496 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 02:47:39.523878 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501356    4496 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 02:47:39.523878 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501358    4496 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 02:47:39.523878 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501361    4496 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 02:47:39.523878 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501364    4496 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 02:47:39.523878 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501366    4496 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 02:47:39.523878 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501369    4496 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 02:47:39.524434 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501371    4496 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 02:47:39.524434 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501377    4496 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 02:47:39.524434 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501380    4496 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 02:47:39.524434 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501382    4496 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 02:47:39.524434 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501385    4496 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 02:47:39.524434 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501387    4496 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 02:47:39.524434 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501390    4496 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 02:47:39.524434 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501392    4496 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 02:47:39.524434 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501395    4496 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 02:47:39.524434 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501398    4496 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 02:47:39.524434 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501400    4496 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 02:47:39.524434 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501403    4496 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 02:47:39.524434 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501406    4496 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 02:47:39.524434 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501408    4496 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 02:47:39.524434 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501411    4496 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 02:47:39.524434 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501413    4496 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 02:47:39.524434 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501417    4496 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 02:47:39.524434 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501421    4496 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 02:47:39.524434 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501424    4496 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 02:47:39.524951 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501428    4496 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 02:47:39.524951 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501431    4496 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 02:47:39.524951 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501433    4496 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 02:47:39.524951 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501436    4496 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 02:47:39.524951 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501438    4496 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 02:47:39.524951 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501441    4496 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 02:47:39.524951 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501444    4496 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 02:47:39.524951 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501447    4496 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 02:47:39.524951 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501449    4496 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 02:47:39.524951 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501452    4496 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 02:47:39.524951 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501454    4496 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 02:47:39.524951 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501457    4496 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 02:47:39.524951 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501460    4496 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 02:47:39.524951 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501462    4496 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 02:47:39.524951 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501466    4496 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 02:47:39.524951 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501469    4496 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 02:47:39.524951 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501472    4496 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 02:47:39.524951 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501474    4496 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 02:47:39.524951 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501477    4496 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 02:47:39.524951 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501480    4496 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 02:47:39.525509 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501482    4496 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 02:47:39.525509 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501485    4496 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 02:47:39.525509 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501487    4496 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 02:47:39.525509 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501490    4496 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 02:47:39.525509 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501500    4496 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 02:47:39.525509 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501504    4496 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 02:47:39.525509 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501506    4496 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 02:47:39.525509 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501509    4496 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 02:47:39.525509 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501511    4496 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 02:47:39.525509 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501515    4496 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 02:47:39.525509 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501520    4496 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 02:47:39.525509 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501522    4496 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 02:47:39.525509 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:39.501527    4496 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 02:47:39.525509 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.501575    4496 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 02 02:47:39.525509 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.503614    4496 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Nov 02 02:47:39.525509 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.503664    4496 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Nov 02 02:47:39.525509 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.503841    4496 server.go:1014] "Starting client certificate rotation"
Nov 02 02:47:39.525509 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.504007    4496 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 02 02:47:39.525984 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.504074    4496 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 02:42:00 +0000 UTC" deadline="2028-05-30 12:26:31.918803658 +0000 UTC"
Nov 02 02:47:39.525984 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.504110    4496 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13809h38m52.414696715s"
Nov 02 02:47:39.525984 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.504361    4496 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 02:47:39.525984 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.505383    4496 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 02:47:39.525984 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.506772    4496 log.go:25] "Validated CRI v1 runtime API"
Nov 02 02:47:39.525984 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.508305    4496 log.go:25] "Validated CRI v1 image API"
Nov 02 02:47:39.525984 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.509111    4496 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 02 02:47:39.525984 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.510591    4496 fs.go:135] Filesystem UUIDs: map[6b1b24d1-2508-478f-959d-02ff53ad6425:/dev/nvme0n1p3 74b93dce-6152-4166-9174-c1be2384bf9a:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Nov 02 02:47:39.526232 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.510606    4496 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0b526f283f6b1ad6466bec01d5bd889c9cbdfdd4d0cc5c47caca5e20a11608de/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0b526f283f6b1ad6466bec01d5bd889c9cbdfdd4d0cc5c47caca5e20a11608de/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/199cbd2b34394d4646dabbcc83851c90c1b35b513f8789a43d2ab8a72d0d41dd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/199cbd2b34394d4646dabbcc83851c90c1b35b513f8789a43d2ab8a72d0d41dd/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1d66575f918ec32ab11a4b703f24211fac7d25eed28e5dbe273c52b07dfc60af/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1d66575f918ec32ab11a4b703f24211fac7d25eed28e5dbe273c52b07dfc60af/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1da77b7031557bd08d918b40a19724d095489096dfeca493347206c5d46660b2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1da77b7031557bd08d918b40a19724d095489096dfeca493347206c5d46660b2/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2ed71fa3aff1cd56e1226d56dbe15d965fc44bee13d69b0cb27b95c8d7e6e2e4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2ed71fa3aff1cd56e1226d56dbe15d965fc44bee13d69b0cb27b95c8d7e6e2e4/userdata/shm major:0 minor:63 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5203b74b3926e29a5b3848e9d8aab0c1c47ed977b976ed8bd7e613c79b15d91e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5203b74b3926e29a5b3848e9d8aab0c1c47ed977b976ed8bd7e613c79b15d91e/userdata/shm major:0 minor:235 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6c4a6e1a0af8dafae71eada0be4fe75fc1bf6debb30f4ad76458efe124709def/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6c4a6e1a0af8dafae71eada0be4fe75fc1bf6debb30f4ad76458efe124709def/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/73ab21a3c4797d944b2b19c2abe98ef17171bfe56972b32ba8c542cb38144243/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/73ab21a3c4797d944b2b19c2abe98ef17171bfe56972b32ba8c542cb38144243/userdata/shm major:0 minor:225 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7e010f75b5a70d368f92ff60896e9c094444b76136d38786a4f0f7024c99eb40/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7e010f75b5a70d368f92ff60896e9c094444b76136d38786a4f0f7024c99eb40/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/85800559c3175d90ec68a3a769b750066c0d2a153ad689d1fca92627ac2c36da/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/85800559c3175d90ec68a3a769b750066c0d2a153ad689d1fca92627ac2c36da/userdata/shm major:0 minor:62 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8bba6a60c331d769827f2a8ce74bc143ee555be0b74c181fe482ca8b000ffce0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8bba6a60c331d769827f2a8ce74bc143ee555be0b74c181fe482ca8b000ffce0/userdata/shm major:0 minor:61 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9f13665adfc009e86a8e18b45db3c81b5fa09a6748c1eee67f7fb555e86bfa8b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9f13665adfc009e86a8e18b45db3c81b5fa09a6748c1eee67f7fb555e86bfa8b/userdata/shm major:0 minor:210 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cb72bec312a628bd7dcc49627f8e12decbd44378a591f97d2c08eac503ed16c1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cb72bec312a628bd7dcc49627f8e12decbd44378a591f97d2c08eac503ed16c1/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d960a2b9fc534555ae1a4989812606630fdb546f04750fe5c7a6129dd61abe4d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d960a2b9fc534555ae1a4989812606630fdb546f04750fe5c7a6129dd61abe4d/userdata/shm major:0 minor:227 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dd9841fb555d9d098d25de67000c4d6821e0d1c1602302738c214238cada1c41/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dd9841fb555d9d098d25de67000c4d6821e0d1c1602302738c214238cada1c41/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f227768a19caf7a40a90610cc586c26dfd19d06ff91053ac825736132ff9df09/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f227768a19caf7a40a90610cc586c26dfd19d06ff91053ac825736132ff9df09/userdata/shm major:0 minor:202 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/086e3504-d2fe-43ad-bc5e-dca409a4e695/volumes/kubernetes.io~projected/kube-api-access-984cg:{mountpoint:/var/lib/kubelet/pods/086e3504-d2fe-43ad-bc5e-dca409a4e695/volumes/kubernetes.io~projected/kube-api-access-984cg major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2d4b0e00-8bb2-4b4c-9805-af5c51c65405/volumes/kubernetes.io~projected/kube-api-access-4xrdb:{mountpoint:/var/lib/kubelet/pods/2d4b0e00-8bb2-4b4c-9805-af5c51c65405/volumes/kubernetes.io~projected/kube-api-access-4xrdb major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/39d09ab7-5724-44d4-a731-92eabb9280a6/volumes/kubernetes.io~projected/kube-api-access-thncs:{mountpoint:/var/lib/kubelet/pods/39d09ab7-5724-44d4-a731-92eabb9280a6/volumes/kubernetes.io~projected/kube-api-access-thncs major:0 minor:234 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/39d09ab7-5724-44d4-a731-92eabb9280a6/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/39d09ab7-5724-44d4-a731-92eabb9280a6/volumes/kubernetes.io~secret/signing-key major:0 minor:233 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/44f31bc6-bd01-4f16-ad6e-0d1ec90bfb4f/volumes/kubernetes.io~projected/kube-api-access-7rrzn:{mountpoint:/var/lib/kubelet/pods/44f31bc6-bd01-4f16-ad6e-0d1ec90bfb4f/volumes/kubernetes.io~projected/kube-api-access-7rrzn major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4fec139d-da70-4cf6-8730-b578df3bebcc/volumes/kubernetes.io~projected/kube-api-access-kzwzx:{mountpoint:/var/lib/kubelet/pods/4fec139d-da70-4cf6-8730-b578df3bebcc/volumes/kubernetes.io~projected/kube-api-access-kzwzx major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5c797b08-a314-44ed-8026-8a0ce1f6c36f/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/5c797b08-a314-44ed-8026-8a0ce1f6c36f/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5c797b08-a314-44ed-8026-8a0ce1f6c36f/volumes/kubernetes.io~projected/kube-api-access-b5dxg:{mountpoint:/var/lib/kubelet/pods/5c797b08-a314-44ed-8026-8a0ce1f6c36f/volumes/kubernetes.io~projected/kube-api-access-b5dxg major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5c797b08-a314-44ed-8026-8a0ce1f6c36f/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/5c797b08-a314-44ed-8026-8a0ce1f6c36f/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/797676e6-5545-41ad-a4d1-c15f26db122a/volumes/kubernetes.io~projected/kube-api-access-zmz88:{mountpoint:/var/lib/kubelet/pods/797676e6-5545-41ad-a4d1-c15f26db122a/volumes/kubernetes.io~projected/kube-api-access-zmz88 major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/797676e6-5545-41ad-a4d1-c15f26db122a/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/797676e6-5545-41ad-a4d1-c15f26db122a/volumes/kubernetes.io~secret/cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/93e3306c-8c14-4bd4-ae04-08bc4b01a5fd/volumes/kubernetes.io~projected/kube-api-access-b47mc:{mountpoint:/var/lib/kubelet/pods/93e3306c-8c14-4bd4-ae04-08bc4b01a5fd/volumes/kubernetes.io~projected/kube-api-access-b47mc major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/94fb7403-b6ef-4d42-be18-7c45b5c10692/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/94fb7403-b6ef-4d42-be18-7c45b5c10692/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/94fb7403-b6ef-4d42-be18-7c45b5c10692/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/94fb7403-b6ef-4d42-be18-7c45b5c10692/volumes/kubernetes.io~empty-dir/tmp major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/94fb7403-b6ef-4d42-be18-7c45b5c10692/volumes/kubernetes.io~projected/kube-api-access-cqtjv:{mountpoint:/var/lib/kubelet/pods/94fb7403-b6ef-4d42-be18-7c45b5c10692/volumes/kubernetes.io~projected/kube-api-access-cqtjv major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/95effc61-7dc3-42c1-9609-be607c3bc65f/volumes/kubernetes.io~projected/kube-api-access-2dl7x:{mountpoint:/var/lib/kubelet/pods/95effc61-7dc3-42c1-9609-be607c3bc65f/volumes/kubernetes.io~projected/kube-api-access-2dl7x major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a81017e4-d705-438e-833b-eedd6de0edde/volumes/kubernetes.io~projected/kube-api-access-68r2v:{mountpoint:/var/lib/kubelet/pods/a81017e4-d705-438e-833b-eedd6de0edde/volumes/kubernetes.io~projected/kube-api-access-68r2v major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bb7ec0c6-6988-45e6-b578-1b68b3a51d9d/volumes/kubernetes.io~projected/kube-api-access-ln99j:{mountpoint:/var/lib/kubelet/pods/bb7ec0c6-6988-45e6-b578-1b68b3a51d9d/volumes/kubernetes.io~projected/kube-api-access-ln99j major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c078a1ee-3179-45c9-9d3b-4d4a46f0c42f/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/c078a1ee-3179-45c9-9d3b-4d4a46f0c42f/volumes/kubernetes.io~secret/agent-certs major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c2f75412-4413-45d1-aa82-919a5206efae/volumes/kubernetes.io~projected/kube-api-access-5hclh:{mountpoint:/var/lib/kubelet/pods/c2f75412-4413-45d1-aa82-919a5206efae/volumes/kubernetes.io~projected/kube-api-access-5hclh major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d66bfee1-59f7-4ed9-90e1-31582a4c4784/volumes/kubernetes.io~projected/kube-api-access-vpg9j:{mountpoint:/var/lib/kubelet/pods/d66bfee1-59f7-4ed9-90e1-31582a4c4784/volumes/kubernetes.io~projected/kube-api-access-vpg9j major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e400bad6-59c3-4c43-aed0-40c731c967db/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/e400bad6-59c3-4c43-aed0-40c731c967db/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e701e783-5b43-45d7-b88b-83fd3c4af13b/volumes/kubernetes.io~projected/kube-api-access-kqsvj:{mountpoint:/var/lib/kubelet/pods/e701e783-5b43-45d7-b88b-83fd3c4af13b/volumes/kubernetes.io~projected/kube-api-access-kqsvj major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f2071fbb-7d2f-49d4-9366-34332e71c6bc/volumes/kubernetes.io~projected/kube-api-access-69zxc:{mountpoint:/var/lib/kubelet/pods/f2071fbb-7d2f-49d4-9366-34332e71c6bc/volumes/kubernetes.io~projected/kube-api-access-69zxc major:0 minor:56 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/9afeaf0f3c1dff6b94cd9c6a80c9d70ac4abf743fb6cd3df58e5b11ffcc3a8b6/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/703dc800951f0d20db41ed37a3204483626e5cf64bc9d5cf240ea348f4486f32/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/6b4239930b0c62470d52d17a2ae52512c80ba930ae996026accbddfb728f8f02/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/08edd11d9734347311a597034a9320ffb10bcd977064b3e7b6b39a1a682a0c84/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-113:{mountpoint:/var/lib/containers/storage/overlay/55c2a79e535c41c0bc533d4741c89a7d8e57cfc2ac3cd6af30980e5a0dd2069c/merged major:0 minor:113 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/1205afb030bb6a608ddb0758244329d855070f5c958cb2be8aee79bc55180157/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/4f37208fb903f0b2f869e789a82b37bd101f4f98704290e715f9117970d9f486/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/9a85b4d3bd43daf92ea95787f5cf09c071015258644a00dd2feaea5e7bac70ff/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-131:{mountpoint:/var/lib/containers/storage/overlay/ccfa4ec39f3419d645bc41077077dd317c79d1a595c7125a19138217a21377b6/merged major:0 minor:131 fsType:overlay blockSize:0} overlay_0-133:{mountpoint:/var/lib/containers/storage/overlay/775c87de3f9a5ac1125dbe60bf9bec715cddffd6a47d4f5ce977a42f342c0ffa/merged major:0 minor:133 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/278bf5fa8625fb8bb4d07c38c99b152ffb871fce7a40909b2cba0ee27f775a5d/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/0730df2bf93fc2fde02a2fb20e9af2566472bd1f2c02e3f6cbab0d0d117819af/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/a89472319e2d3677e84db7b042d402db93414b3bd44dad15a13eefd9cace3c51/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/732e84c896cfea2086ab077ba8aded4621e45177eab05da47d626da4333eb617/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/903eccec8c13edae4dc23d75321932ad5c2b02f775c30e21e0e7c4b63e1d1097/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/00a4ee2f804cd018657e3156f5ad182483ea2108f63f919293abf46ac90ec913/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/8008d56c964c36aba241b001ee2f5531e96d9f17cf3714ec444bbeb74f35d075/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/c3481e4beb4bd8bed2be51077ab2bd41394f59eb096d6a5f620deeb1439a8301/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/633e1128af3a2b9ed917b2296b2d68e03d7c45104e4c0e7d541cd613175203f2/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/95d0ed75d011cc312820cf053e80aa085976fa62e2c9d5736ddaa79260727a3e/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/e7c42e15266d60e2663f4384040b540a562b40ef05d8feab0ac24d149539c05b/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-204:{mountpoint:/var/lib/containers/storage/overlay/c083abb74700251eae7dd1cb9de3cc6933f345daf497394c80728a8ff21771e1/merged major:0 minor:204 fsType:overlay blockSize:0} overlay_0-212:{mountpoint:/var/lib/containers/storage/overlay/e2cc75d1502460fee0c98de56d0696d6243e2c386c4939418f80f9a04178d93f/merged major:0 minor:212 fsType:overlay blockSize:0} overlay_0-214:{mountpoint:/var/lib/containers/storage/overlay/cea4c299795d82f27d31e331362e14fdaef248d7aa64b1b9c282a064c895ba0f/merged major:0 minor:214 fsType:overlay blockSize:0} overlay_0-216:{mountpoint:/var/lib/containers/storage/overlay/05e4e703d45c1c91905a1186b45385f2067374556f789bd82f30e2e6f43de8a4/merged major:0 minor:216 fsType:overlay blockSize:0} overlay_0-229:{mountpoint:/var/lib/containers/storage/overlay/c182591eabccc7fb1ad422fb44ce24862dbc7f6ec307cfae891fa63fb0350384/merged major:0 minor:229 fsType:overlay blockSize:0} overlay_0-231:{mountpoint:/var/lib/containers/storage/overlay/48d544856e0a48a739e8d613a33b86ecaed2cc17c220eeeddf063895cb7a7cc1/merged major:0 minor:231 fsType:overlay blockSize:0} overlay_0-237:{mountpoint:/var/lib/containers/storage/overlay/2ef8ebb2353f7455773d535b3e20109bb74970af4f8fcde02e750275b140dcda/merged major:0 minor:237 fsType:overlay blockSize:0} overlay_0-239:{mountpoint:/var/lib/containers/storage/overlay/3de00f64f5f4bbe2af125ccf25695940617116de88f5ca75b5bb11bc40dd27b4/merged major:0 minor:239 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/c4690629a49b16aae7ceb581c97b89cb21a20dc6286bd12689424f200914c7e8/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-247:{mountpoint:/var/lib/containers/storage/overlay/a8d1267420bebdb4cff4e91f856f73808f40de001cef4ce03ec993a41d2e79b2/merged major:0 minor:247 fsType:overlay blockSize:0} overlay_0-253:{mountpoint:/var/lib/containers/storage/overlay/5ccb4f43be1d560d2d6fea34bb8d6584105efbed44732d7bb520bdefd8883160/merged major:0 minor:253 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/d744e6bd3cc4b9d1c47e9a67481440a02a4d25107ef8d170719a9b7fdfa5215e/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/a944932cb9c75687891ae7a6488dea1a92e7740b3787ec57004de0ccc3fcafb5/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/6f7c5b9e0a6beeacf184b1ed4ba8c2fade1bd3b33554019a8b89fb05b794e68d/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/bfd9480557c84811c6071211cc5448ef81a979619f1a225c04ad151df0862bf0/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/76028d8393765191d611685d78f6f3e52e5e276d5f067c455a51e551238f6042/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/5c70551abcd78eb22c37f2742bad54ae7556ec98e0064cf373e394257a2faaff/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/ff078a893bdfc6da33f0bef701f69fa450ab32216e6d46477308658acbb4c409/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/1988ebcb590d3519c66915ea636646cbfdac9c180ac1dbcd43f4bf94a1a2b533/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/d1c4c251f067e7685509e02bd4322632ae3620a0751de6bff7033fcb92578f12/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/0688b9c8430a088d4e9f9b6b657cbc9c648c0bf776cfdad082bf4276dcee07f2/merged major:0 minor:96 fsType:overlay blockSize:0}]
Nov 02 02:47:39.526394 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.517000    4496 manager.go:217] Machine: {Timestamp:2026-11-02 02:47:39.516543155 +0000 UTC m=+0.095150527 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099323 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ab0e6370e0a3b506477cf0d2b752e SystemUUID:ec2ab0e6-370e-0a3b-5064-77cf0d2b752e BootID:9465d3cb-d6fa-443e-a97e-2a8a74633ee2 Filesystems:[{Device:overlay_0-113 DeviceMajor:0 DeviceMinor:113 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/199cbd2b34394d4646dabbcc83851c90c1b35b513f8789a43d2ab8a72d0d41dd/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-212 DeviceMajor:0 DeviceMinor:212 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/086e3504-d2fe-43ad-bc5e-dca409a4e695/volumes/kubernetes.io~projected/kube-api-access-984cg DeviceMajor:0 DeviceMinor:51 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-133 DeviceMajor:0 DeviceMinor:133 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-216 DeviceMajor:0 DeviceMinor:216 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-247 DeviceMajor:0 DeviceMinor:247 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/e701e783-5b43-45d7-b88b-83fd3c4af13b/volumes/kubernetes.io~projected/kube-api-access-kqsvj DeviceMajor:0 DeviceMinor:57 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2ed71fa3aff1cd56e1226d56dbe15d965fc44bee13d69b0cb27b95c8d7e6e2e4/userdata/shm DeviceMajor:0 DeviceMinor:63 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0b526f283f6b1ad6466bec01d5bd889c9cbdfdd4d0cc5c47caca5e20a11608de/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5c797b08-a314-44ed-8026-8a0ce1f6c36f/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/a81017e4-d705-438e-833b-eedd6de0edde/volumes/kubernetes.io~projected/kube-api-access-68r2v DeviceMajor:0 DeviceMinor:54 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9f13665adfc009e86a8e18b45db3c81b5fa09a6748c1eee67f7fb555e86bfa8b/userdata/shm DeviceMajor:0 DeviceMinor:210 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/95effc61-7dc3-42c1-9609-be607c3bc65f/volumes/kubernetes.io~projected/kube-api-access-2dl7x DeviceMajor:0 DeviceMinor:223 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-239 DeviceMajor:0 DeviceMinor:239 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5c797b08-a314-44ed-8026-8a0ce1f6c36f/volumes/kubernetes.io~projected/kube-api-access-b5dxg DeviceMajor:0 DeviceMinor:59 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-204 DeviceMajor:0 DeviceMinor:204 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/39d09ab7-5724-44d4-a731-92eabb9280a6/volumes/kubernetes.io~projected/kube-api-access-thncs DeviceMajor:0 DeviceMinor:234 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5203b74b3926e29a5b3848e9d8aab0c1c47ed977b976ed8bd7e613c79b15d91e/userdata/shm DeviceMajor:0 DeviceMinor:235 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8bba6a60c331d769827f2a8ce74bc143ee555be0b74c181fe482ca8b000ffce0/userdata/shm DeviceMajor:0 DeviceMinor:61 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/797676e6-5545-41ad-a4d1-c15f26db122a/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:197 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/797676e6-5545-41ad-a4d1-c15f26db122a/volumes/kubernetes.io~projected/kube-api-access-zmz88 DeviceMajor:0 DeviceMinor:201 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-229 DeviceMajor:0 DeviceMinor:229 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/c078a1ee-3179-45c9-9d3b-4d4a46f0c42f/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:49 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/94fb7403-b6ef-4d42-be18-7c45b5c10692/volumes/kubernetes.io~projected/kube-api-access-cqtjv DeviceMajor:0 DeviceMinor:55 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-214 DeviceMajor:0 DeviceMinor:214 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1da77b7031557bd08d918b40a19724d095489096dfeca493347206c5d46660b2/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/4fec139d-da70-4cf6-8730-b578df3bebcc/volumes/kubernetes.io~projected/kube-api-access-kzwzx DeviceMajor:0 DeviceMinor:58 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cb72bec312a628bd7dcc49627f8e12decbd44378a591f97d2c08eac503ed16c1/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-131 DeviceMajor:0 DeviceMinor:131 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e400bad6-59c3-4c43-aed0-40c731c967db/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:206 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/94fb7403-b6ef-4d42-be18-7c45b5c10692/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:47 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6c4a6e1a0af8dafae71eada0be4fe75fc1bf6debb30f4ad76458efe124709def/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/85800559c3175d90ec68a3a769b750066c0d2a153ad689d1fca92627ac2c36da/userdata/shm DeviceMajor:0 DeviceMinor:62 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/39d09ab7-5724-44d4-a731-92eabb9280a6/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:233 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/44f31bc6-bd01-4f16-ad6e-0d1ec90bfb4f/volumes/kubernetes.io~projected/kube-api-access-7rrzn DeviceMajor:0 DeviceMinor:209 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dd9841fb555d9d098d25de67000c4d6821e0d1c1602302738c214238cada1c41/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/93e3306c-8c14-4bd4-ae04-08bc4b01a5fd/volumes/kubernetes.io~projected/kube-api-access-b47mc DeviceMajor:0 DeviceMinor:222 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/73ab21a3c4797d944b2b19c2abe98ef17171bfe56972b32ba8c542cb38144243/userdata/shm DeviceMajor:0 DeviceMinor:225 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d960a2b9fc534555ae1a4989812606630fdb546f04750fe5c7a6129dd61abe4d/userdata/shm DeviceMajor:0 DeviceMinor:227 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/94fb7403-b6ef-4d42-be18-7c45b5c10692/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:48 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/f2071fbb-7d2f-49d4-9366-34332e71c6bc/volumes/kubernetes.io~projected/kube-api-access-69zxc DeviceMajor:0 DeviceMinor:56 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f227768a19caf7a40a90610cc586c26dfd19d06ff91053ac825736132ff9df09/userdata/shm DeviceMajor:0 DeviceMinor:202 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/5c797b08-a314-44ed-8026-8a0ce1f6c36f/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:50 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-237 DeviceMajor:0 DeviceMinor:237 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c2f75412-4413-45d1-aa82-919a5206efae/volumes/kubernetes.io~projected/kube-api-access-5hclh DeviceMajor:0 DeviceMinor:52 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7e010f75b5a70d368f92ff60896e9c094444b76136d38786a4f0f7024c99eb40/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-231 DeviceMajor:0 DeviceMinor:231 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bb7ec0c6-6988-45e6-b578-1b68b3a51d9d/volumes/kubernetes.io~projected/kube-api-access-ln99j DeviceMajor:0 DeviceMinor:53 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1d66575f918ec32ab11a4b703f24211fac7d25eed28e5dbe273c52b07dfc60af/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2d4b0e00-8bb2-4b4c-9805-af5c51c65405/volumes/kubernetes.io~projected/kube-api-access-4xrdb DeviceMajor:0 DeviceMinor:45 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/d66bfee1-59f7-4ed9-90e1-31582a4c4784/volumes/kubernetes.io~projected/kube-api-access-vpg9j DeviceMajor:0 DeviceMinor:224 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-253 DeviceMajor:0 DeviceMinor:253 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:199cbd2b34394d4 MacAddress:b2:d6:8a:d1:b4:3c Speed:10000 Mtu:8901} {Name:5203b74b3926e29 MacAddress:76:2e:7b:fa:59:02 Speed:10000 Mtu:8901} {Name:73ab21a3c4797d9 MacAddress:4e:42:25:e5:7c:c2 Speed:10000 Mtu:8901} {Name:9f13665adfc009e MacAddress:ee:5c:3c:ab:eb:bc Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:3d:cb:48:41:27 Speed:0 Mtu:9001} {Name:br-int MacAddress:ee:cd:df:0b:5c:8f Speed:0 Mtu:8901} {Name:d960a2b9fc53455 MacAddress:da:9a:98:53:2b:a8 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:3d:cb:48:41:27 Speed:0 Mtu:9001} {Name:f227768a19caf7a MacAddress:5e:e2:69:6f:95:92 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:66:d2:2b:54:ee:d8 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:b6:74:6d:1c:58:eb Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 02 02:47:39.526394 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.517260    4496 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 02 02:47:39.526394 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.517300    4496 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 02 02:47:39.526394 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.517564    4496 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 02 02:47:39.526394 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.517589    4496 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-163.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 02 02:47:39.526394 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.517927    4496 topology_manager.go:143] "Creating topology manager with none policy"
Nov 02 02:47:39.526394 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.517939    4496 container_manager_linux.go:311] "Creating device plugin manager"
Nov 02 02:47:39.526394 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.517954    4496 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 02:47:39.526394 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.517983    4496 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 02:47:39.526394 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.518110    4496 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 02 02:47:39.526394 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.518363    4496 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 02 02:47:39.526394 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.518505    4496 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 02 02:47:39.526394 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.518548    4496 kubelet.go:493] "Attempting to sync node with API server"
Nov 02 02:47:39.526394 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.518565    4496 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 02 02:47:39.526394 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.518579    4496 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 02 02:47:39.526394 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.518593    4496 kubelet.go:405] "Adding apiserver pod source"
Nov 02 02:47:39.526394 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.518603    4496 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 02 02:47:39.526394 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.519668    4496 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 02 02:47:39.526394 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.519680    4496 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 02:47:39.526394 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.520110    4496 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 02 02:47:39.528355 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.528337    4496 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 02 02:47:39.528355 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.528353    4496 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 02:47:39.528515 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.528472    4496 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Nov 02 02:47:39.528695 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.528680    4496 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 02 02:47:39.528755 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.528733    4496 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 02 02:47:39.528856 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.528844    4496 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 02 02:47:39.528910 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.528864    4496 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 02 02:47:39.528910 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.528874    4496 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 02 02:47:39.528910 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.528884    4496 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 02 02:47:39.528910 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.528893    4496 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 02 02:47:39.528910 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.528903    4496 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 02 02:47:39.529153 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.528913    4496 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 02 02:47:39.529153 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.528921    4496 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 02 02:47:39.529153 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.528933    4496 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 02 02:47:39.529153 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.528942    4496 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 02 02:47:39.529153 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.528956    4496 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 02 02:47:39.529153 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.528969    4496 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 02 02:47:39.529153 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.529007    4496 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 02 02:47:39.529153 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.529037    4496 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 02 02:47:39.532359 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.532343    4496 server.go:1263] "Started kubelet"
Nov 02 02:47:39.532486 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.532456    4496 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 02 02:47:39.533103 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.532718    4496 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 02 02:47:39.533103 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.532789    4496 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 02 02:47:39.533103 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.533032    4496 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 02 02:47:39.532854 ip-10-0-133-163 systemd[1]: Started Kubernetes Kubelet.
Nov 02 02:47:39.534573 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.534006    4496 server.go:317] "Adding debug handlers to kubelet server"
Nov 02 02:47:39.535723 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.535700    4496 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 02 02:47:39.537825 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.535844    4496 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 02 02:47:39.537825 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.535967    4496 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 02:42:01 +0000 UTC" deadline="2028-04-08 12:05:46.293879979 +0000 UTC"
Nov 02 02:47:39.537825 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.536034    4496 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12561h18m6.757850188s"
Nov 02 02:47:39.537825 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.536174    4496 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 02 02:47:39.537825 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.536186    4496 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 02 02:47:39.537825 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.536287    4496 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 02 02:47:39.537825 ip-10-0-133-163 kubenswrapper[4496]: E1102 02:47:39.536462    4496 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-163.us-west-1.compute.internal\" not found"
Nov 02 02:47:39.537825 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.536947    4496 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 02 02:47:39.537825 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.536977    4496 factory.go:55] Registering systemd factory
Nov 02 02:47:39.537825 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.536988    4496 factory.go:223] Registration of the systemd container factory successfully
Nov 02 02:47:39.537825 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.537290    4496 factory.go:153] Registering CRI-O factory
Nov 02 02:47:39.537825 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.537303    4496 factory.go:223] Registration of the crio container factory successfully
Nov 02 02:47:39.537825 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.537327    4496 factory.go:103] Registering Raw factory
Nov 02 02:47:39.537825 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.537341    4496 manager.go:1199] Started watching for new ooms in manager
Nov 02 02:47:39.537825 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.537829    4496 manager.go:319] Starting recovery of all containers
Nov 02 02:47:39.539284 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.538477    4496 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="94fb7403-b6ef-4d42-be18-7c45b5c10692" volumeName="kubernetes.io/projected/94fb7403-b6ef-4d42-be18-7c45b5c10692-kube-api-access-cqtjv" seLinuxMountContext=""
Nov 02 02:47:39.539284 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.538524    4496 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a81017e4-d705-438e-833b-eedd6de0edde" volumeName="kubernetes.io/configmap/a81017e4-d705-438e-833b-eedd6de0edde-whereabouts-flatfile-configmap" seLinuxMountContext=""
Nov 02 02:47:39.539284 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.538540    4496 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="39d09ab7-5724-44d4-a731-92eabb9280a6" volumeName="kubernetes.io/configmap/39d09ab7-5724-44d4-a731-92eabb9280a6-signing-cabundle" seLinuxMountContext=""
Nov 02 02:47:39.539284 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.538554    4496 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="39d09ab7-5724-44d4-a731-92eabb9280a6" volumeName="kubernetes.io/secret/39d09ab7-5724-44d4-a731-92eabb9280a6-signing-key" seLinuxMountContext=""
Nov 02 02:47:39.539284 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.538573    4496 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5c797b08-a314-44ed-8026-8a0ce1f6c36f" volumeName="kubernetes.io/configmap/5c797b08-a314-44ed-8026-8a0ce1f6c36f-ovnkube-config" seLinuxMountContext=""
Nov 02 02:47:39.539284 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.538588    4496 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="797676e6-5545-41ad-a4d1-c15f26db122a" volumeName="kubernetes.io/projected/797676e6-5545-41ad-a4d1-c15f26db122a-kube-api-access-zmz88" seLinuxMountContext=""
Nov 02 02:47:39.539284 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.538607    4496 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="93e3306c-8c14-4bd4-ae04-08bc4b01a5fd" volumeName="kubernetes.io/projected/93e3306c-8c14-4bd4-ae04-08bc4b01a5fd-kube-api-access-b47mc" seLinuxMountContext=""
Nov 02 02:47:39.539284 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.538621    4496 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="94fb7403-b6ef-4d42-be18-7c45b5c10692" volumeName="kubernetes.io/empty-dir/94fb7403-b6ef-4d42-be18-7c45b5c10692-etc-tuned" seLinuxMountContext=""
Nov 02 02:47:39.539284 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.538643    4496 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="086e3504-d2fe-43ad-bc5e-dca409a4e695" volumeName="kubernetes.io/projected/086e3504-d2fe-43ad-bc5e-dca409a4e695-kube-api-access-984cg" seLinuxMountContext=""
Nov 02 02:47:39.539284 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.538657    4496 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2d4b0e00-8bb2-4b4c-9805-af5c51c65405" volumeName="kubernetes.io/configmap/2d4b0e00-8bb2-4b4c-9805-af5c51c65405-config-volume" seLinuxMountContext=""
Nov 02 02:47:39.539284 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.538671    4496 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4fec139d-da70-4cf6-8730-b578df3bebcc" volumeName="kubernetes.io/empty-dir/4fec139d-da70-4cf6-8730-b578df3bebcc-tmp-dir" seLinuxMountContext=""
Nov 02 02:47:39.539284 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.538690    4496 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4fec139d-da70-4cf6-8730-b578df3bebcc" volumeName="kubernetes.io/projected/4fec139d-da70-4cf6-8730-b578df3bebcc-kube-api-access-kzwzx" seLinuxMountContext=""
Nov 02 02:47:39.539284 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.538718    4496 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5c797b08-a314-44ed-8026-8a0ce1f6c36f" volumeName="kubernetes.io/configmap/5c797b08-a314-44ed-8026-8a0ce1f6c36f-env-overrides" seLinuxMountContext=""
Nov 02 02:47:39.539284 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.538736    4496 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="95effc61-7dc3-42c1-9609-be607c3bc65f" volumeName="kubernetes.io/empty-dir/95effc61-7dc3-42c1-9609-be607c3bc65f-archives-path" seLinuxMountContext=""
Nov 02 02:47:39.539284 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.538755    4496 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a81017e4-d705-438e-833b-eedd6de0edde" volumeName="kubernetes.io/configmap/a81017e4-d705-438e-833b-eedd6de0edde-cni-binary-copy" seLinuxMountContext=""
Nov 02 02:47:39.539284 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.538774    4496 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c078a1ee-3179-45c9-9d3b-4d4a46f0c42f" volumeName="kubernetes.io/secret/c078a1ee-3179-45c9-9d3b-4d4a46f0c42f-agent-certs" seLinuxMountContext=""
Nov 02 02:47:39.539284 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.538788    4496 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="797676e6-5545-41ad-a4d1-c15f26db122a" volumeName="kubernetes.io/secret/797676e6-5545-41ad-a4d1-c15f26db122a-cert" seLinuxMountContext=""
Nov 02 02:47:39.539284 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.538831    4496 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="95effc61-7dc3-42c1-9609-be607c3bc65f" volumeName="kubernetes.io/configmap/95effc61-7dc3-42c1-9609-be607c3bc65f-service-ca-bundle" seLinuxMountContext=""
Nov 02 02:47:39.539284 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.538852    4496 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a81017e4-d705-438e-833b-eedd6de0edde" volumeName="kubernetes.io/configmap/a81017e4-d705-438e-833b-eedd6de0edde-cni-sysctl-allowlist" seLinuxMountContext=""
Nov 02 02:47:39.539284 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.538868    4496 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bb7ec0c6-6988-45e6-b578-1b68b3a51d9d" volumeName="kubernetes.io/projected/bb7ec0c6-6988-45e6-b578-1b68b3a51d9d-kube-api-access-ln99j" seLinuxMountContext=""
Nov 02 02:47:39.539284 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.538888    4496 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f2071fbb-7d2f-49d4-9366-34332e71c6bc" volumeName="kubernetes.io/projected/f2071fbb-7d2f-49d4-9366-34332e71c6bc-kube-api-access-69zxc" seLinuxMountContext=""
Nov 02 02:47:39.539284 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.538902    4496 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c2f75412-4413-45d1-aa82-919a5206efae" volumeName="kubernetes.io/projected/c2f75412-4413-45d1-aa82-919a5206efae-kube-api-access-5hclh" seLinuxMountContext=""
Nov 02 02:47:39.539284 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.538916    4496 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e701e783-5b43-45d7-b88b-83fd3c4af13b" volumeName="kubernetes.io/projected/e701e783-5b43-45d7-b88b-83fd3c4af13b-kube-api-access-kqsvj" seLinuxMountContext=""
Nov 02 02:47:39.539284 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.538935    4496 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f2071fbb-7d2f-49d4-9366-34332e71c6bc" volumeName="kubernetes.io/configmap/f2071fbb-7d2f-49d4-9366-34332e71c6bc-iptables-alerter-script" seLinuxMountContext=""
Nov 02 02:47:39.539284 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.538950    4496 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="94fb7403-b6ef-4d42-be18-7c45b5c10692" volumeName="kubernetes.io/empty-dir/94fb7403-b6ef-4d42-be18-7c45b5c10692-tmp" seLinuxMountContext=""
Nov 02 02:47:39.539284 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.538968    4496 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="95effc61-7dc3-42c1-9609-be607c3bc65f" volumeName="kubernetes.io/configmap/95effc61-7dc3-42c1-9609-be607c3bc65f-trusted-ca-bundle" seLinuxMountContext=""
Nov 02 02:47:39.539284 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.538981    4496 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2d4b0e00-8bb2-4b4c-9805-af5c51c65405" volumeName="kubernetes.io/projected/2d4b0e00-8bb2-4b4c-9805-af5c51c65405-kube-api-access-4xrdb" seLinuxMountContext=""
Nov 02 02:47:39.539284 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.538999    4496 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5c797b08-a314-44ed-8026-8a0ce1f6c36f" volumeName="kubernetes.io/projected/5c797b08-a314-44ed-8026-8a0ce1f6c36f-kube-api-access-b5dxg" seLinuxMountContext=""
Nov 02 02:47:39.539284 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.539057    4496 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c078a1ee-3179-45c9-9d3b-4d4a46f0c42f" volumeName="kubernetes.io/configmap/c078a1ee-3179-45c9-9d3b-4d4a46f0c42f-konnectivity-ca" seLinuxMountContext=""
Nov 02 02:47:39.539284 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.539078    4496 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d66bfee1-59f7-4ed9-90e1-31582a4c4784" volumeName="kubernetes.io/projected/d66bfee1-59f7-4ed9-90e1-31582a4c4784-kube-api-access-vpg9j" seLinuxMountContext=""
Nov 02 02:47:39.539284 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.539092    4496 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="39d09ab7-5724-44d4-a731-92eabb9280a6" volumeName="kubernetes.io/projected/39d09ab7-5724-44d4-a731-92eabb9280a6-kube-api-access-thncs" seLinuxMountContext=""
Nov 02 02:47:39.539284 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.539110    4496 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="44f31bc6-bd01-4f16-ad6e-0d1ec90bfb4f" volumeName="kubernetes.io/projected/44f31bc6-bd01-4f16-ad6e-0d1ec90bfb4f-kube-api-access-7rrzn" seLinuxMountContext=""
Nov 02 02:47:39.539284 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.539124    4496 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5c797b08-a314-44ed-8026-8a0ce1f6c36f" volumeName="kubernetes.io/secret/5c797b08-a314-44ed-8026-8a0ce1f6c36f-ovn-node-metrics-cert" seLinuxMountContext=""
Nov 02 02:47:39.539284 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.539139    4496 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="93e3306c-8c14-4bd4-ae04-08bc4b01a5fd" volumeName="kubernetes.io/empty-dir/93e3306c-8c14-4bd4-ae04-08bc4b01a5fd-data-volume" seLinuxMountContext=""
Nov 02 02:47:39.539284 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.539157    4496 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a81017e4-d705-438e-833b-eedd6de0edde" volumeName="kubernetes.io/projected/a81017e4-d705-438e-833b-eedd6de0edde-kube-api-access-68r2v" seLinuxMountContext=""
Nov 02 02:47:39.539284 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.539171    4496 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c2f75412-4413-45d1-aa82-919a5206efae" volumeName="kubernetes.io/configmap/c2f75412-4413-45d1-aa82-919a5206efae-multus-daemon-config" seLinuxMountContext=""
Nov 02 02:47:39.539284 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.539190    4496 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c2f75412-4413-45d1-aa82-919a5206efae" volumeName="kubernetes.io/configmap/c2f75412-4413-45d1-aa82-919a5206efae-cni-binary-copy" seLinuxMountContext=""
Nov 02 02:47:39.539284 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.539204    4496 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e400bad6-59c3-4c43-aed0-40c731c967db" volumeName="kubernetes.io/secret/e400bad6-59c3-4c43-aed0-40c731c967db-original-pull-secret" seLinuxMountContext=""
Nov 02 02:47:39.539284 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.539217    4496 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2d4b0e00-8bb2-4b4c-9805-af5c51c65405" volumeName="kubernetes.io/empty-dir/2d4b0e00-8bb2-4b4c-9805-af5c51c65405-tmp-dir" seLinuxMountContext=""
Nov 02 02:47:39.539284 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.539235    4496 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5c797b08-a314-44ed-8026-8a0ce1f6c36f" volumeName="kubernetes.io/configmap/5c797b08-a314-44ed-8026-8a0ce1f6c36f-ovnkube-script-lib" seLinuxMountContext=""
Nov 02 02:47:39.539284 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.539249    4496 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e701e783-5b43-45d7-b88b-83fd3c4af13b" volumeName="kubernetes.io/configmap/e701e783-5b43-45d7-b88b-83fd3c4af13b-serviceca" seLinuxMountContext=""
Nov 02 02:47:39.539284 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.539282    4496 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="95effc61-7dc3-42c1-9609-be607c3bc65f" volumeName="kubernetes.io/projected/95effc61-7dc3-42c1-9609-be607c3bc65f-kube-api-access-2dl7x" seLinuxMountContext=""
Nov 02 02:47:39.539284 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.539295    4496 reconstruct.go:97] "Volume reconstruction finished"
Nov 02 02:47:39.539284 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.539304    4496 reconciler.go:29] "Reconciler: start to sync state"
Nov 02 02:47:39.558484 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.557972    4496 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hww8q_5c797b08-a314-44ed-8026-8a0ce1f6c36f/ovn-acl-logging/0.log"
Nov 02 02:47:39.558651 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.558625    4496 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hww8q_5c797b08-a314-44ed-8026-8a0ce1f6c36f/ovn-acl-logging/0.log"
Nov 02 02:47:39.563697 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.563660    4496 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 02 02:47:39.565242 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.565218    4496 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 02 02:47:39.565242 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.565241    4496 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 02 02:47:39.565368 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.565260    4496 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 02 02:47:39.565368 ip-10-0-133-163 kubenswrapper[4496]: E1102 02:47:39.565293    4496 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 02 02:47:39.578004 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.577986    4496 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hww8q_5c797b08-a314-44ed-8026-8a0ce1f6c36f/ovn-acl-logging/0.log"
Nov 02 02:47:39.578388 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.578365    4496 generic.go:356] "Generic (PLEG): container finished" podID="5c797b08-a314-44ed-8026-8a0ce1f6c36f" containerID="2b4fa01e32a51c3de62373eef442e0bcd848f8b0d41775d52ec2a9e7f2aeb60c" exitCode=1
Nov 02 02:47:39.588542 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.588514    4496 generic.go:356] "Generic (PLEG): container finished" podID="950fb6772a40be3e22d676c446372284" containerID="c8ea7ef81711de6aa7fae28c1bcb705f30c984de61919cd1d0eec71fe57a04f7" exitCode=0
Nov 02 02:47:39.594799 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.594778    4496 generic.go:356] "Generic (PLEG): container finished" podID="a81017e4-d705-438e-833b-eedd6de0edde" containerID="b739c5193007811120325da8e065a1e2330cdf710b065b3e7a1d1ba472d33a9e" exitCode=0
Nov 02 02:47:39.594799 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.594798    4496 generic.go:356] "Generic (PLEG): container finished" podID="a81017e4-d705-438e-833b-eedd6de0edde" containerID="576ac7e2a1977b3114cdcdfec404ceae7ebd6de8fcd9a05cb878c7deae995df6" exitCode=0
Nov 02 02:47:39.594799 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.594804    4496 generic.go:356] "Generic (PLEG): container finished" podID="a81017e4-d705-438e-833b-eedd6de0edde" containerID="a987233257be4a22cb3b47f7ec931f535bdb695368b4402526d64bb2310a1dd5" exitCode=0
Nov 02 02:47:39.594959 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.594810    4496 generic.go:356] "Generic (PLEG): container finished" podID="a81017e4-d705-438e-833b-eedd6de0edde" containerID="3cd13392d3a79feeedc18ebc385fc826fc62e5b93ff55e365b2f075ee0322611" exitCode=0
Nov 02 02:47:39.594959 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.594816    4496 generic.go:356] "Generic (PLEG): container finished" podID="a81017e4-d705-438e-833b-eedd6de0edde" containerID="f24f80957ce420ef3480bfde153c6bd24b0e4b39ac0ba29a059d95b5ad30f38d" exitCode=0
Nov 02 02:47:39.594959 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.594821    4496 generic.go:356] "Generic (PLEG): container finished" podID="a81017e4-d705-438e-833b-eedd6de0edde" containerID="b0dc03625aa435cf373a4f1913630511ce02ed069f9416d6f78287d1c4049270" exitCode=0
Nov 02 02:47:39.607855 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.607838    4496 manager.go:324] Recovery completed
Nov 02 02:47:39.619905 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.619890    4496 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 02:47:39.621099 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.621083    4496 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-163.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 02 02:47:39.621173 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.621111    4496 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-163.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 02:47:39.621173 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.621121    4496 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-163.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 02 02:47:39.622388 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.622369    4496 cpu_manager.go:226] "Starting" policy="none"
Nov 02 02:47:39.622388 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.622382    4496 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 02 02:47:39.622497 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.622391    4496 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 02:47:39.622497 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.622398    4496 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 02 02:47:39.622613 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.622599    4496 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Nov 02 02:47:39.622646 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.622614    4496 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Nov 02 02:47:39.622646 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.622638    4496 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Nov 02 02:47:39.622701 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.622647    4496 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Nov 02 02:47:39.622701 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.622654    4496 policy_none.go:50] "Start"
Nov 02 02:47:39.622701 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.622663    4496 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 02 02:47:39.622701 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.622672    4496 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 02 02:47:39.622798 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.622789    4496 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Nov 02 02:47:39.622798 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.622796    4496 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Nov 02 02:47:39.622851 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.622802    4496 policy_none.go:44] "Start"
Nov 02 02:47:39.623329 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.623313    4496 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-163.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 02 02:47:39.623403 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.623340    4496 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-163.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 02:47:39.623403 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.623353    4496 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-163.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 02 02:47:39.627388 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.627358    4496 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 02 02:47:39.629862 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.629846    4496 manager.go:341] "Starting Device Plugin manager"
Nov 02 02:47:39.629954 ip-10-0-133-163 kubenswrapper[4496]: E1102 02:47:39.629871    4496 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 02 02:47:39.629954 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.629880    4496 server.go:87] "Starting device plugin registration server"
Nov 02 02:47:39.630136 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.630124    4496 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 02 02:47:39.630180 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.630136    4496 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 02 02:47:39.630260 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.630245    4496 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 02 02:47:39.630475 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.630345    4496 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 02 02:47:39.630475 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.630384    4496 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 02 02:47:39.630475 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.630390    4496 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 02 02:47:39.630640 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.630425    4496 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T02:47:39.630374088Z","UUID":"98402631-6e38-4a0c-aea0-55d762a312a8","Handler":null,"Name":"","Endpoint":""}
Nov 02 02:47:39.631761 ip-10-0-133-163 kubenswrapper[4496]: E1102 02:47:39.631739    4496 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 02 02:47:39.631843 ip-10-0-133-163 kubenswrapper[4496]: E1102 02:47:39.631783    4496 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-163.us-west-1.compute.internal\" not found"
Nov 02 02:47:39.631843 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.631801    4496 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 02 02:47:39.631843 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.631822    4496 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 02 02:47:39.665432 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.665400    4496 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-163.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-163.us-west-1.compute.internal"]
Nov 02 02:47:39.665519 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.665508    4496 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 02:47:39.669846 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.669825    4496 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-163.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 02 02:47:39.669929 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.669851    4496 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-163.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 02:47:39.669929 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.669867    4496 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-163.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 02 02:47:39.670107 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.670097    4496 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 02:47:39.670246 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.670234    4496 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 02:47:39.673894 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.673876    4496 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-163.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 02 02:47:39.673960 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.673908    4496 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-163.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 02:47:39.673960 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.673919    4496 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-163.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 02 02:47:39.673960 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.673876    4496 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-163.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 02 02:47:39.674077 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.673997    4496 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-163.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 02:47:39.674077 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.674028    4496 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-163.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 02 02:47:39.674077 ip-10-0-133-163 kubenswrapper[4496]: E1102 02:47:39.674067    4496 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-163.us-west-1.compute.internal\" not found" node="ip-10-0-133-163.us-west-1.compute.internal"
Nov 02 02:47:39.674281 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.674255    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-163.us-west-1.compute.internal" event={"ID":"950fb6772a40be3e22d676c446372284","Type":"ContainerStarted","Data":"7a7d5b406ff4da9eb330b2cef992961555bdcd50227e029f325e53c0decce674"}
Nov 02 02:47:39.674315 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.674294    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-163.us-west-1.compute.internal" event={"ID":"950fb6772a40be3e22d676c446372284","Type":"ContainerDied","Data":"c8ea7ef81711de6aa7fae28c1bcb705f30c984de61919cd1d0eec71fe57a04f7"}
Nov 02 02:47:39.674315 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.674308    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-163.us-west-1.compute.internal" event={"ID":"950fb6772a40be3e22d676c446372284","Type":"ContainerStarted","Data":"85800559c3175d90ec68a3a769b750066c0d2a153ad689d1fca92627ac2c36da"}
Nov 02 02:47:39.674367 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.674327    4496 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 02:47:39.674367 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.674338    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-163.us-west-1.compute.internal" event={"ID":"562e4cfd5885e6864e00b6a01f2722f3","Type":"ContainerStarted","Data":"737e95dd26182223150ee9677fbc7f60f4030198402cbc59d5b1288d14df6aa3"}
Nov 02 02:47:39.674367 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.674353    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-163.us-west-1.compute.internal" event={"ID":"562e4cfd5885e6864e00b6a01f2722f3","Type":"ContainerStarted","Data":"2ed71fa3aff1cd56e1226d56dbe15d965fc44bee13d69b0cb27b95c8d7e6e2e4"}
Nov 02 02:47:39.675476 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.675462    4496 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-163.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 02 02:47:39.675526 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.675485    4496 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-163.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 02:47:39.675526 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.675495    4496 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-163.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 02 02:47:39.675615 ip-10-0-133-163 kubenswrapper[4496]: E1102 02:47:39.675604    4496 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-163.us-west-1.compute.internal\" not found" node="ip-10-0-133-163.us-west-1.compute.internal"
Nov 02 02:47:39.730347 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.730316    4496 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 02:47:39.730582 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.730562    4496 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 02:47:39.730637 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.730610    4496 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Nov 02 02:47:39.731324 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.731300    4496 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 02:47:39.731400 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.731350    4496 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 02:47:39.731400 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.731356    4496 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-163.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 02 02:47:39.731400 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.731379    4496 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-163.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 02:47:39.731400 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.731395    4496 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-163.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 02 02:47:39.731528 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.731424    4496 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-133-163.us-west-1.compute.internal"
Nov 02 02:47:39.736636 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.736622    4496 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-163.us-west-1.compute.internal"
Nov 02 02:47:39.782265 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.782240    4496 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 02:47:39.831549 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.831502    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/950fb6772a40be3e22d676c446372284-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-163.us-west-1.compute.internal\" (UID: \"950fb6772a40be3e22d676c446372284\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-163.us-west-1.compute.internal"
Nov 02 02:47:39.831549 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.831527    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/950fb6772a40be3e22d676c446372284-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-163.us-west-1.compute.internal\" (UID: \"950fb6772a40be3e22d676c446372284\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-163.us-west-1.compute.internal"
Nov 02 02:47:39.831549 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.831546    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/562e4cfd5885e6864e00b6a01f2722f3-config\") pod \"kube-apiserver-proxy-ip-10-0-133-163.us-west-1.compute.internal\" (UID: \"562e4cfd5885e6864e00b6a01f2722f3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-163.us-west-1.compute.internal"
Nov 02 02:47:39.841556 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.841532    4496 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 02:47:39.841611 ip-10-0-133-163 kubenswrapper[4496]: E1102 02:47:39.841573    4496 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-133-163.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-163.us-west-1.compute.internal"
Nov 02 02:47:39.841611 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.841584    4496 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-163.us-west-1.compute.internal"
Nov 02 02:47:39.893764 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.893746    4496 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-133-163.us-west-1.compute.internal"
Nov 02 02:47:39.893820 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.893803    4496 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-133-163.us-west-1.compute.internal"
Nov 02 02:47:39.894113 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.894092    4496 nodeinfomanager.go:471] existing CSINode "ip-10-0-133-163.us-west-1.compute.internal" is owned by different node (oldNodeID="c11ce92e-c71d-4b7a-b38d-a3a35d64822c", newNodeID=""), cleaning up...
Nov 02 02:47:39.894342 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.894326    4496 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-163.us-west-1.compute.internal" event="NodeNotReady"
Nov 02 02:47:39.894422 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.894359    4496 setters.go:546] "Node became not ready" node="ip-10-0-133-163.us-west-1.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-11-02T02:47:39Z","lastTransitionTime":"2026-11-02T02:47:39Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"}
Nov 02 02:47:39.899480 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.899463    4496 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 02:47:39.899555 ip-10-0-133-163 kubenswrapper[4496]: E1102 02:47:39.899494    4496 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-133-163.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-163.us-west-1.compute.internal"
Nov 02 02:47:39.932638 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.932615    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/950fb6772a40be3e22d676c446372284-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-163.us-west-1.compute.internal\" (UID: \"950fb6772a40be3e22d676c446372284\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-163.us-west-1.compute.internal"
Nov 02 02:47:39.932701 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.932677    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/950fb6772a40be3e22d676c446372284-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-163.us-west-1.compute.internal\" (UID: \"950fb6772a40be3e22d676c446372284\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-163.us-west-1.compute.internal"
Nov 02 02:47:39.932752 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.932724    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/950fb6772a40be3e22d676c446372284-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-163.us-west-1.compute.internal\" (UID: \"950fb6772a40be3e22d676c446372284\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-163.us-west-1.compute.internal"
Nov 02 02:47:39.932814 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.932754    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/562e4cfd5885e6864e00b6a01f2722f3-config\") pod \"kube-apiserver-proxy-ip-10-0-133-163.us-west-1.compute.internal\" (UID: \"562e4cfd5885e6864e00b6a01f2722f3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-163.us-west-1.compute.internal"
Nov 02 02:47:39.932814 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.932805    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/562e4cfd5885e6864e00b6a01f2722f3-config\") pod \"kube-apiserver-proxy-ip-10-0-133-163.us-west-1.compute.internal\" (UID: \"562e4cfd5885e6864e00b6a01f2722f3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-163.us-west-1.compute.internal"
Nov 02 02:47:39.932885 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:39.932815    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/950fb6772a40be3e22d676c446372284-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-163.us-west-1.compute.internal\" (UID: \"950fb6772a40be3e22d676c446372284\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-163.us-west-1.compute.internal"
Nov 02 02:47:40.164305 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.164246    4496 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-163.us-west-1.compute.internal" event="NodeReady"
Nov 02 02:47:40.164305 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.164308    4496 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Nov 02 02:47:40.519099 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.519063    4496 apiserver.go:52] "Watching apiserver"
Nov 02 02:47:40.603327 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.603300    4496 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-163.us-west-1.compute.internal"
Nov 02 02:47:40.603464 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.603435    4496 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-163.us-west-1.compute.internal"
Nov 02 02:47:40.624600 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.624572    4496 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 02 02:47:40.626381 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.626348    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zlplc","openshift-network-operator/iptables-alerter-8sxd5","openshift-service-ca/service-ca-5fffc97bdf-rspt2","openshift-cluster-node-tuning-operator/tuned-7qpcf","openshift-dns/dns-default-fbvxx","openshift-network-diagnostics/network-check-target-47f7x","openshift-insights/insights-runtime-extractor-r4zs9","openshift-insights/periodic-gathering-t9g7w-hqs6p","openshift-multus/multus-tt68t","openshift-kube-storage-version-migrator/migrator-66df5d496b-sp26d","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-163.us-west-1.compute.internal","openshift-dns/node-resolver-sztr6","kube-system/konnectivity-agent-z8sq6","kube-system/kube-apiserver-proxy-ip-10-0-133-163.us-west-1.compute.internal","openshift-image-registry/node-ca-kkqzw","openshift-ingress-canary/ingress-canary-zt7hz","openshift-ovn-kubernetes/ovnkube-node-hww8q","openshift-multus/network-metrics-daemon-zkjqm","kube-system/global-pull-secret-syncer-dxlld","openshift-multus/multus-additional-cni-plugins-69j4f"]
Nov 02 02:47:40.628003 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.627981    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zkjqm"
Nov 02 02:47:40.630795 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.630778    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fbvxx"
Nov 02 02:47:40.631849 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.631834    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-r4zs9"
Nov 02 02:47:40.637315 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.637296    4496 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 02 02:47:40.664942 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.664922    4496 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 02:47:40.665007 ip-10-0-133-163 kubenswrapper[4496]: E1102 02:47:40.664960    4496 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-133-163.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-163.us-west-1.compute.internal"
Nov 02 02:47:40.666244 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.666231    4496 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 02:47:40.666311 ip-10-0-133-163 kubenswrapper[4496]: E1102 02:47:40.666260    4496 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-133-163.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-163.us-west-1.compute.internal"
Nov 02 02:47:40.681745 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.681722    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 02 02:47:40.681745 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.681737    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 02 02:47:40.681881 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.681739    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 02 02:47:40.681881 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.681742    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 02 02:47:40.681881 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.681768    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 02 02:47:40.681881 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.681793    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 02 02:47:40.683107 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.683086    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 02 02:47:40.683193 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.683170    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 02 02:47:40.683366 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.683338    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-6md5s\""
Nov 02 02:47:40.683366 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.683353    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-5v9mr\""
Nov 02 02:47:40.683366 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.683353    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-7jkxs\""
Nov 02 02:47:40.683521 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.683394    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-x92pp\""
Nov 02 02:47:40.683757 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.683743    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-ghkrt\""
Nov 02 02:47:40.683834 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.683816    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 02 02:47:40.684088 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.684076    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-cjmh9\""
Nov 02 02:47:40.684420 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.684404    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-b8q6l\""
Nov 02 02:47:40.684457 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.684446    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-9gkth\""
Nov 02 02:47:40.708481 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.708464    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-wsjnf\""
Nov 02 02:47:40.708481 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.708462    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-pbbl9\""
Nov 02 02:47:40.709024 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.709004    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 02 02:47:40.709090 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.709076    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 02 02:47:40.709166 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.709142    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 02 02:47:40.709166 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.709162    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-5mjxc\""
Nov 02 02:47:40.709273 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.709181    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-xmln4\""
Nov 02 02:47:40.709273 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.709189    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 02 02:47:40.709273 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.709258    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Nov 02 02:47:40.709411 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.709273    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 02 02:47:40.709411 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.709323    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-gdqg4\""
Nov 02 02:47:40.709411 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.709356    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-cgnbb\""
Nov 02 02:47:40.709411 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.709383    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-vvbct\""
Nov 02 02:47:40.709769 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.709755    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-85n6h\""
Nov 02 02:47:40.709827 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.709816    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-dgnd5\""
Nov 02 02:47:40.709881 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.709870    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-nvmsr\""
Nov 02 02:47:40.737645 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.737619    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqsvj\" (UniqueName: \"kubernetes.io/projected/e701e783-5b43-45d7-b88b-83fd3c4af13b-kube-api-access-kqsvj\") pod \"node-ca-kkqzw\" (UID: \"e701e783-5b43-45d7-b88b-83fd3c4af13b\") " pod="openshift-image-registry/node-ca-kkqzw"
Nov 02 02:47:40.737735 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.737654    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a81017e4-d705-438e-833b-eedd6de0edde-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-69j4f\" (UID: \"a81017e4-d705-438e-833b-eedd6de0edde\") " pod="openshift-multus/multus-additional-cni-plugins-69j4f"
Nov 02 02:47:40.737735 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.737684    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c2f75412-4413-45d1-aa82-919a5206efae-host-run-k8s-cni-cncf-io\") pod \"multus-tt68t\" (UID: \"c2f75412-4413-45d1-aa82-919a5206efae\") " pod="openshift-multus/multus-tt68t"
Nov 02 02:47:40.737735 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.737707    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/94fb7403-b6ef-4d42-be18-7c45b5c10692-etc-modprobe-d\") pod \"tuned-7qpcf\" (UID: \"94fb7403-b6ef-4d42-be18-7c45b5c10692\") " pod="openshift-cluster-node-tuning-operator/tuned-7qpcf"
Nov 02 02:47:40.737864 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.737734    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/797676e6-5545-41ad-a4d1-c15f26db122a-cert\") pod \"ingress-canary-zt7hz\" (UID: \"797676e6-5545-41ad-a4d1-c15f26db122a\") " pod="openshift-ingress-canary/ingress-canary-zt7hz"
Nov 02 02:47:40.737864 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.737791    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d4b0e00-8bb2-4b4c-9805-af5c51c65405-tmp-dir\") pod \"dns-default-fbvxx\" (UID: \"2d4b0e00-8bb2-4b4c-9805-af5c51c65405\") " pod="openshift-dns/dns-default-fbvxx"
Nov 02 02:47:40.737864 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.737820    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5c797b08-a314-44ed-8026-8a0ce1f6c36f-node-log\") pod \"ovnkube-node-hww8q\" (UID: \"5c797b08-a314-44ed-8026-8a0ce1f6c36f\") " pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:40.737959 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.737865    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmz88\" (UniqueName: \"kubernetes.io/projected/797676e6-5545-41ad-a4d1-c15f26db122a-kube-api-access-zmz88\") pod \"ingress-canary-zt7hz\" (UID: \"797676e6-5545-41ad-a4d1-c15f26db122a\") " pod="openshift-ingress-canary/ingress-canary-zt7hz"
Nov 02 02:47:40.737959 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.737940    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xrdb\" (UniqueName: \"kubernetes.io/projected/2d4b0e00-8bb2-4b4c-9805-af5c51c65405-kube-api-access-4xrdb\") pod \"dns-default-fbvxx\" (UID: \"2d4b0e00-8bb2-4b4c-9805-af5c51c65405\") " pod="openshift-dns/dns-default-fbvxx"
Nov 02 02:47:40.738130 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.737946    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d4b0e00-8bb2-4b4c-9805-af5c51c65405-tmp-dir\") pod \"dns-default-fbvxx\" (UID: \"2d4b0e00-8bb2-4b4c-9805-af5c51c65405\") " pod="openshift-dns/dns-default-fbvxx"
Nov 02 02:47:40.738130 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.737984    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f2071fbb-7d2f-49d4-9366-34332e71c6bc-iptables-alerter-script\") pod \"iptables-alerter-8sxd5\" (UID: \"f2071fbb-7d2f-49d4-9366-34332e71c6bc\") " pod="openshift-network-operator/iptables-alerter-8sxd5"
Nov 02 02:47:40.738130 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.738009    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqtjv\" (UniqueName: \"kubernetes.io/projected/94fb7403-b6ef-4d42-be18-7c45b5c10692-kube-api-access-cqtjv\") pod \"tuned-7qpcf\" (UID: \"94fb7403-b6ef-4d42-be18-7c45b5c10692\") " pod="openshift-cluster-node-tuning-operator/tuned-7qpcf"
Nov 02 02:47:40.738130 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.738045    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5dxg\" (UniqueName: \"kubernetes.io/projected/5c797b08-a314-44ed-8026-8a0ce1f6c36f-kube-api-access-b5dxg\") pod \"ovnkube-node-hww8q\" (UID: \"5c797b08-a314-44ed-8026-8a0ce1f6c36f\") " pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:40.738130 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.738069    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bb7ec0c6-6988-45e6-b578-1b68b3a51d9d-socket-dir\") pod \"aws-ebs-csi-driver-node-zlplc\" (UID: \"bb7ec0c6-6988-45e6-b578-1b68b3a51d9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zlplc"
Nov 02 02:47:40.738130 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.738084    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bb7ec0c6-6988-45e6-b578-1b68b3a51d9d-device-dir\") pod \"aws-ebs-csi-driver-node-zlplc\" (UID: \"bb7ec0c6-6988-45e6-b578-1b68b3a51d9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zlplc"
Nov 02 02:47:40.738130 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.738100    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e400bad6-59c3-4c43-aed0-40c731c967db-original-pull-secret\") pod \"global-pull-secret-syncer-dxlld\" (UID: \"e400bad6-59c3-4c43-aed0-40c731c967db\") " pod="kube-system/global-pull-secret-syncer-dxlld"
Nov 02 02:47:40.738130 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.738114    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c2f75412-4413-45d1-aa82-919a5206efae-cnibin\") pod \"multus-tt68t\" (UID: \"c2f75412-4413-45d1-aa82-919a5206efae\") " pod="openshift-multus/multus-tt68t"
Nov 02 02:47:40.738130 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.738121    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/797676e6-5545-41ad-a4d1-c15f26db122a-cert\") pod \"ingress-canary-zt7hz\" (UID: \"797676e6-5545-41ad-a4d1-c15f26db122a\") " pod="openshift-ingress-canary/ingress-canary-zt7hz"
Nov 02 02:47:40.738539 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.738129    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hclh\" (UniqueName: \"kubernetes.io/projected/c2f75412-4413-45d1-aa82-919a5206efae-kube-api-access-5hclh\") pod \"multus-tt68t\" (UID: \"c2f75412-4413-45d1-aa82-919a5206efae\") " pod="openshift-multus/multus-tt68t"
Nov 02 02:47:40.738539 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.738220    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/94fb7403-b6ef-4d42-be18-7c45b5c10692-etc-sysctl-d\") pod \"tuned-7qpcf\" (UID: \"94fb7403-b6ef-4d42-be18-7c45b5c10692\") " pod="openshift-cluster-node-tuning-operator/tuned-7qpcf"
Nov 02 02:47:40.738539 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.738248    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/94fb7403-b6ef-4d42-be18-7c45b5c10692-etc-systemd\") pod \"tuned-7qpcf\" (UID: \"94fb7403-b6ef-4d42-be18-7c45b5c10692\") " pod="openshift-cluster-node-tuning-operator/tuned-7qpcf"
Nov 02 02:47:40.738539 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.738273    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/94fb7403-b6ef-4d42-be18-7c45b5c10692-tmp\") pod \"tuned-7qpcf\" (UID: \"94fb7403-b6ef-4d42-be18-7c45b5c10692\") " pod="openshift-cluster-node-tuning-operator/tuned-7qpcf"
Nov 02 02:47:40.738539 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.738370    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-984cg\" (UniqueName: \"kubernetes.io/projected/086e3504-d2fe-43ad-bc5e-dca409a4e695-kube-api-access-984cg\") pod \"network-metrics-daemon-zkjqm\" (UID: \"086e3504-d2fe-43ad-bc5e-dca409a4e695\") " pod="openshift-multus/network-metrics-daemon-zkjqm"
Nov 02 02:47:40.738539 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.738407    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/94fb7403-b6ef-4d42-be18-7c45b5c10692-tmp\") pod \"tuned-7qpcf\" (UID: \"94fb7403-b6ef-4d42-be18-7c45b5c10692\") " pod="openshift-cluster-node-tuning-operator/tuned-7qpcf"
Nov 02 02:47:40.738539 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.738473    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f2071fbb-7d2f-49d4-9366-34332e71c6bc-iptables-alerter-script\") pod \"iptables-alerter-8sxd5\" (UID: \"f2071fbb-7d2f-49d4-9366-34332e71c6bc\") " pod="openshift-network-operator/iptables-alerter-8sxd5"
Nov 02 02:47:40.738539 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.738491    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c2f75412-4413-45d1-aa82-919a5206efae-host-run-netns\") pod \"multus-tt68t\" (UID: \"c2f75412-4413-45d1-aa82-919a5206efae\") " pod="openshift-multus/multus-tt68t"
Nov 02 02:47:40.738539 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.738498    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e400bad6-59c3-4c43-aed0-40c731c967db-original-pull-secret\") pod \"global-pull-secret-syncer-dxlld\" (UID: \"e400bad6-59c3-4c43-aed0-40c731c967db\") " pod="kube-system/global-pull-secret-syncer-dxlld"
Nov 02 02:47:40.738539 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.738519    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c2f75412-4413-45d1-aa82-919a5206efae-multus-conf-dir\") pod \"multus-tt68t\" (UID: \"c2f75412-4413-45d1-aa82-919a5206efae\") " pod="openshift-multus/multus-tt68t"
Nov 02 02:47:40.738907 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.738550    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c2f75412-4413-45d1-aa82-919a5206efae-host-run-multus-certs\") pod \"multus-tt68t\" (UID: \"c2f75412-4413-45d1-aa82-919a5206efae\") " pod="openshift-multus/multus-tt68t"
Nov 02 02:47:40.738907 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.738574    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fec139d-da70-4cf6-8730-b578df3bebcc-tmp-dir\") pod \"node-resolver-sztr6\" (UID: \"4fec139d-da70-4cf6-8730-b578df3bebcc\") " pod="openshift-dns/node-resolver-sztr6"
Nov 02 02:47:40.738907 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.738604    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/94fb7403-b6ef-4d42-be18-7c45b5c10692-etc-sysctl-conf\") pod \"tuned-7qpcf\" (UID: \"94fb7403-b6ef-4d42-be18-7c45b5c10692\") " pod="openshift-cluster-node-tuning-operator/tuned-7qpcf"
Nov 02 02:47:40.738907 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.738661    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fec139d-da70-4cf6-8730-b578df3bebcc-tmp-dir\") pod \"node-resolver-sztr6\" (UID: \"4fec139d-da70-4cf6-8730-b578df3bebcc\") " pod="openshift-dns/node-resolver-sztr6"
Nov 02 02:47:40.738907 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.738683    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5c797b08-a314-44ed-8026-8a0ce1f6c36f-ovnkube-config\") pod \"ovnkube-node-hww8q\" (UID: \"5c797b08-a314-44ed-8026-8a0ce1f6c36f\") " pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:40.738907 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.738707    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/94fb7403-b6ef-4d42-be18-7c45b5c10692-sys\") pod \"tuned-7qpcf\" (UID: \"94fb7403-b6ef-4d42-be18-7c45b5c10692\") " pod="openshift-cluster-node-tuning-operator/tuned-7qpcf"
Nov 02 02:47:40.738907 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.738722    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5c797b08-a314-44ed-8026-8a0ce1f6c36f-host-kubelet\") pod \"ovnkube-node-hww8q\" (UID: \"5c797b08-a314-44ed-8026-8a0ce1f6c36f\") " pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:40.738907 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.738737    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5c797b08-a314-44ed-8026-8a0ce1f6c36f-ovn-node-metrics-cert\") pod \"ovnkube-node-hww8q\" (UID: \"5c797b08-a314-44ed-8026-8a0ce1f6c36f\") " pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:40.738907 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.738772    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95effc61-7dc3-42c1-9609-be607c3bc65f-service-ca-bundle\") pod \"periodic-gathering-t9g7w-hqs6p\" (UID: \"95effc61-7dc3-42c1-9609-be607c3bc65f\") " pod="openshift-insights/periodic-gathering-t9g7w-hqs6p"
Nov 02 02:47:40.738907 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.738796    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bb7ec0c6-6988-45e6-b578-1b68b3a51d9d-etc-selinux\") pod \"aws-ebs-csi-driver-node-zlplc\" (UID: \"bb7ec0c6-6988-45e6-b578-1b68b3a51d9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zlplc"
Nov 02 02:47:40.738907 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.738812    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ln99j\" (UniqueName: \"kubernetes.io/projected/bb7ec0c6-6988-45e6-b578-1b68b3a51d9d-kube-api-access-ln99j\") pod \"aws-ebs-csi-driver-node-zlplc\" (UID: \"bb7ec0c6-6988-45e6-b578-1b68b3a51d9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zlplc"
Nov 02 02:47:40.738907 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.738828    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a81017e4-d705-438e-833b-eedd6de0edde-cnibin\") pod \"multus-additional-cni-plugins-69j4f\" (UID: \"a81017e4-d705-438e-833b-eedd6de0edde\") " pod="openshift-multus/multus-additional-cni-plugins-69j4f"
Nov 02 02:47:40.738907 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.738844    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f2071fbb-7d2f-49d4-9366-34332e71c6bc-host-slash\") pod \"iptables-alerter-8sxd5\" (UID: \"f2071fbb-7d2f-49d4-9366-34332e71c6bc\") " pod="openshift-network-operator/iptables-alerter-8sxd5"
Nov 02 02:47:40.738907 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.738864    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e701e783-5b43-45d7-b88b-83fd3c4af13b-serviceca\") pod \"node-ca-kkqzw\" (UID: \"e701e783-5b43-45d7-b88b-83fd3c4af13b\") " pod="openshift-image-registry/node-ca-kkqzw"
Nov 02 02:47:40.739495 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.738977    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a81017e4-d705-438e-833b-eedd6de0edde-tuning-conf-dir\") pod \"multus-additional-cni-plugins-69j4f\" (UID: \"a81017e4-d705-438e-833b-eedd6de0edde\") " pod="openshift-multus/multus-additional-cni-plugins-69j4f"
Nov 02 02:47:40.739495 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.739003    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c2f75412-4413-45d1-aa82-919a5206efae-multus-cni-dir\") pod \"multus-tt68t\" (UID: \"c2f75412-4413-45d1-aa82-919a5206efae\") " pod="openshift-multus/multus-tt68t"
Nov 02 02:47:40.739495 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.739043    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c2f75412-4413-45d1-aa82-919a5206efae-cni-binary-copy\") pod \"multus-tt68t\" (UID: \"c2f75412-4413-45d1-aa82-919a5206efae\") " pod="openshift-multus/multus-tt68t"
Nov 02 02:47:40.739495 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.739057    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5c797b08-a314-44ed-8026-8a0ce1f6c36f-ovn-node-metrics-cert\") pod \"ovnkube-node-hww8q\" (UID: \"5c797b08-a314-44ed-8026-8a0ce1f6c36f\") " pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:40.739495 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.739105    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c2f75412-4413-45d1-aa82-919a5206efae-hostroot\") pod \"multus-tt68t\" (UID: \"c2f75412-4413-45d1-aa82-919a5206efae\") " pod="openshift-multus/multus-tt68t"
Nov 02 02:47:40.739495 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.739135    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4fec139d-da70-4cf6-8730-b578df3bebcc-hosts-file\") pod \"node-resolver-sztr6\" (UID: \"4fec139d-da70-4cf6-8730-b578df3bebcc\") " pod="openshift-dns/node-resolver-sztr6"
Nov 02 02:47:40.739495 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.739169    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/39d09ab7-5724-44d4-a731-92eabb9280a6-signing-cabundle\") pod \"service-ca-5fffc97bdf-rspt2\" (UID: \"39d09ab7-5724-44d4-a731-92eabb9280a6\") " pod="openshift-service-ca/service-ca-5fffc97bdf-rspt2"
Nov 02 02:47:40.739495 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.739240    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/95effc61-7dc3-42c1-9609-be607c3bc65f-archives-path\") pod \"periodic-gathering-t9g7w-hqs6p\" (UID: \"95effc61-7dc3-42c1-9609-be607c3bc65f\") " pod="openshift-insights/periodic-gathering-t9g7w-hqs6p"
Nov 02 02:47:40.739495 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.739260    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c2f75412-4413-45d1-aa82-919a5206efae-cni-binary-copy\") pod \"multus-tt68t\" (UID: \"c2f75412-4413-45d1-aa82-919a5206efae\") " pod="openshift-multus/multus-tt68t"
Nov 02 02:47:40.739495 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.739292    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c2f75412-4413-45d1-aa82-919a5206efae-multus-socket-dir-parent\") pod \"multus-tt68t\" (UID: \"c2f75412-4413-45d1-aa82-919a5206efae\") " pod="openshift-multus/multus-tt68t"
Nov 02 02:47:40.739495 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.739322    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5c797b08-a314-44ed-8026-8a0ce1f6c36f-run-systemd\") pod \"ovnkube-node-hww8q\" (UID: \"5c797b08-a314-44ed-8026-8a0ce1f6c36f\") " pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:40.739495 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.739346    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5c797b08-a314-44ed-8026-8a0ce1f6c36f-host-cni-bin\") pod \"ovnkube-node-hww8q\" (UID: \"5c797b08-a314-44ed-8026-8a0ce1f6c36f\") " pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:40.739495 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.739298    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/95effc61-7dc3-42c1-9609-be607c3bc65f-archives-path\") pod \"periodic-gathering-t9g7w-hqs6p\" (UID: \"95effc61-7dc3-42c1-9609-be607c3bc65f\") " pod="openshift-insights/periodic-gathering-t9g7w-hqs6p"
Nov 02 02:47:40.739495 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.739398    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c078a1ee-3179-45c9-9d3b-4d4a46f0c42f-konnectivity-ca\") pod \"konnectivity-agent-z8sq6\" (UID: \"c078a1ee-3179-45c9-9d3b-4d4a46f0c42f\") " pod="kube-system/konnectivity-agent-z8sq6"
Nov 02 02:47:40.739495 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.739425    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c2f75412-4413-45d1-aa82-919a5206efae-system-cni-dir\") pod \"multus-tt68t\" (UID: \"c2f75412-4413-45d1-aa82-919a5206efae\") " pod="openshift-multus/multus-tt68t"
Nov 02 02:47:40.739495 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.739448    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c2f75412-4413-45d1-aa82-919a5206efae-host-var-lib-kubelet\") pod \"multus-tt68t\" (UID: \"c2f75412-4413-45d1-aa82-919a5206efae\") " pod="openshift-multus/multus-tt68t"
Nov 02 02:47:40.739495 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.739475    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2d4b0e00-8bb2-4b4c-9805-af5c51c65405-config-volume\") pod \"dns-default-fbvxx\" (UID: \"2d4b0e00-8bb2-4b4c-9805-af5c51c65405\") " pod="openshift-dns/dns-default-fbvxx"
Nov 02 02:47:40.739495 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.739498    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2d4b0e00-8bb2-4b4c-9805-af5c51c65405-metrics-tls\") pod \"dns-default-fbvxx\" (UID: \"2d4b0e00-8bb2-4b4c-9805-af5c51c65405\") " pod="openshift-dns/dns-default-fbvxx"
Nov 02 02:47:40.740211 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.739521    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5c797b08-a314-44ed-8026-8a0ce1f6c36f-var-lib-openvswitch\") pod \"ovnkube-node-hww8q\" (UID: \"5c797b08-a314-44ed-8026-8a0ce1f6c36f\") " pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:40.740211 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.739547    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/94fb7403-b6ef-4d42-be18-7c45b5c10692-run\") pod \"tuned-7qpcf\" (UID: \"94fb7403-b6ef-4d42-be18-7c45b5c10692\") " pod="openshift-cluster-node-tuning-operator/tuned-7qpcf"
Nov 02 02:47:40.740211 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.739574    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/94fb7403-b6ef-4d42-be18-7c45b5c10692-host\") pod \"tuned-7qpcf\" (UID: \"94fb7403-b6ef-4d42-be18-7c45b5c10692\") " pod="openshift-cluster-node-tuning-operator/tuned-7qpcf"
Nov 02 02:47:40.740211 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.739629    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/94fb7403-b6ef-4d42-be18-7c45b5c10692-etc-tuned\") pod \"tuned-7qpcf\" (UID: \"94fb7403-b6ef-4d42-be18-7c45b5c10692\") " pod="openshift-cluster-node-tuning-operator/tuned-7qpcf"
Nov 02 02:47:40.740211 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.739683    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/94fb7403-b6ef-4d42-be18-7c45b5c10692-etc-tuned\") pod \"tuned-7qpcf\" (UID: \"94fb7403-b6ef-4d42-be18-7c45b5c10692\") " pod="openshift-cluster-node-tuning-operator/tuned-7qpcf"
Nov 02 02:47:40.740211 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.739713    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rrzn\" (UniqueName: \"kubernetes.io/projected/44f31bc6-bd01-4f16-ad6e-0d1ec90bfb4f-kube-api-access-7rrzn\") pod \"network-check-target-47f7x\" (UID: \"44f31bc6-bd01-4f16-ad6e-0d1ec90bfb4f\") " pod="openshift-network-diagnostics/network-check-target-47f7x"
Nov 02 02:47:40.740211 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.739742    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5c797b08-a314-44ed-8026-8a0ce1f6c36f-host-run-ovn-kubernetes\") pod \"ovnkube-node-hww8q\" (UID: \"5c797b08-a314-44ed-8026-8a0ce1f6c36f\") " pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:40.740211 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.739830    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95effc61-7dc3-42c1-9609-be607c3bc65f-trusted-ca-bundle\") pod \"periodic-gathering-t9g7w-hqs6p\" (UID: \"95effc61-7dc3-42c1-9609-be607c3bc65f\") " pod="openshift-insights/periodic-gathering-t9g7w-hqs6p"
Nov 02 02:47:40.740211 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.739860    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dl7x\" (UniqueName: \"kubernetes.io/projected/95effc61-7dc3-42c1-9609-be607c3bc65f-kube-api-access-2dl7x\") pod \"periodic-gathering-t9g7w-hqs6p\" (UID: \"95effc61-7dc3-42c1-9609-be607c3bc65f\") " pod="openshift-insights/periodic-gathering-t9g7w-hqs6p"
Nov 02 02:47:40.740211 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.739913    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/93e3306c-8c14-4bd4-ae04-08bc4b01a5fd-crio-socket\") pod \"insights-runtime-extractor-r4zs9\" (UID: \"93e3306c-8c14-4bd4-ae04-08bc4b01a5fd\") " pod="openshift-insights/insights-runtime-extractor-r4zs9"
Nov 02 02:47:40.740211 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.739931    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e701e783-5b43-45d7-b88b-83fd3c4af13b-host\") pod \"node-ca-kkqzw\" (UID: \"e701e783-5b43-45d7-b88b-83fd3c4af13b\") " pod="openshift-image-registry/node-ca-kkqzw"
Nov 02 02:47:40.740211 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.739972    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/94fb7403-b6ef-4d42-be18-7c45b5c10692-etc-sysconfig\") pod \"tuned-7qpcf\" (UID: \"94fb7403-b6ef-4d42-be18-7c45b5c10692\") " pod="openshift-cluster-node-tuning-operator/tuned-7qpcf"
Nov 02 02:47:40.740211 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.740032    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/94fb7403-b6ef-4d42-be18-7c45b5c10692-etc-kubernetes\") pod \"tuned-7qpcf\" (UID: \"94fb7403-b6ef-4d42-be18-7c45b5c10692\") " pod="openshift-cluster-node-tuning-operator/tuned-7qpcf"
Nov 02 02:47:40.740211 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.740062    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5c797b08-a314-44ed-8026-8a0ce1f6c36f-run-openvswitch\") pod \"ovnkube-node-hww8q\" (UID: \"5c797b08-a314-44ed-8026-8a0ce1f6c36f\") " pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:40.740211 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.740085    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bb7ec0c6-6988-45e6-b578-1b68b3a51d9d-registration-dir\") pod \"aws-ebs-csi-driver-node-zlplc\" (UID: \"bb7ec0c6-6988-45e6-b578-1b68b3a51d9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zlplc"
Nov 02 02:47:40.740211 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.740103    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a81017e4-d705-438e-833b-eedd6de0edde-system-cni-dir\") pod \"multus-additional-cni-plugins-69j4f\" (UID: \"a81017e4-d705-438e-833b-eedd6de0edde\") " pod="openshift-multus/multus-additional-cni-plugins-69j4f"
Nov 02 02:47:40.740211 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.740157    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a81017e4-d705-438e-833b-eedd6de0edde-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-69j4f\" (UID: \"a81017e4-d705-438e-833b-eedd6de0edde\") " pod="openshift-multus/multus-additional-cni-plugins-69j4f"
Nov 02 02:47:40.740211 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.740199    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b47mc\" (UniqueName: \"kubernetes.io/projected/93e3306c-8c14-4bd4-ae04-08bc4b01a5fd-kube-api-access-b47mc\") pod \"insights-runtime-extractor-r4zs9\" (UID: \"93e3306c-8c14-4bd4-ae04-08bc4b01a5fd\") " pod="openshift-insights/insights-runtime-extractor-r4zs9"
Nov 02 02:47:40.740819 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.740228    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bb7ec0c6-6988-45e6-b578-1b68b3a51d9d-sys-fs\") pod \"aws-ebs-csi-driver-node-zlplc\" (UID: \"bb7ec0c6-6988-45e6-b578-1b68b3a51d9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zlplc"
Nov 02 02:47:40.740819 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.740243    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e400bad6-59c3-4c43-aed0-40c731c967db-kubelet-config\") pod \"global-pull-secret-syncer-dxlld\" (UID: \"e400bad6-59c3-4c43-aed0-40c731c967db\") " pod="kube-system/global-pull-secret-syncer-dxlld"
Nov 02 02:47:40.740819 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.740263    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c2f75412-4413-45d1-aa82-919a5206efae-host-var-lib-cni-multus\") pod \"multus-tt68t\" (UID: \"c2f75412-4413-45d1-aa82-919a5206efae\") " pod="openshift-multus/multus-tt68t"
Nov 02 02:47:40.740819 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.740308    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c2f75412-4413-45d1-aa82-919a5206efae-etc-kubernetes\") pod \"multus-tt68t\" (UID: \"c2f75412-4413-45d1-aa82-919a5206efae\") " pod="openshift-multus/multus-tt68t"
Nov 02 02:47:40.740819 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.740350    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thncs\" (UniqueName: \"kubernetes.io/projected/39d09ab7-5724-44d4-a731-92eabb9280a6-kube-api-access-thncs\") pod \"service-ca-5fffc97bdf-rspt2\" (UID: \"39d09ab7-5724-44d4-a731-92eabb9280a6\") " pod="openshift-service-ca/service-ca-5fffc97bdf-rspt2"
Nov 02 02:47:40.740819 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.740378    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69zxc\" (UniqueName: \"kubernetes.io/projected/f2071fbb-7d2f-49d4-9366-34332e71c6bc-kube-api-access-69zxc\") pod \"iptables-alerter-8sxd5\" (UID: \"f2071fbb-7d2f-49d4-9366-34332e71c6bc\") " pod="openshift-network-operator/iptables-alerter-8sxd5"
Nov 02 02:47:40.740819 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.740519    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5c797b08-a314-44ed-8026-8a0ce1f6c36f-etc-openvswitch\") pod \"ovnkube-node-hww8q\" (UID: \"5c797b08-a314-44ed-8026-8a0ce1f6c36f\") " pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:40.740819 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.740543    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5c797b08-a314-44ed-8026-8a0ce1f6c36f-env-overrides\") pod \"ovnkube-node-hww8q\" (UID: \"5c797b08-a314-44ed-8026-8a0ce1f6c36f\") " pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:40.740819 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.740594    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5c797b08-a314-44ed-8026-8a0ce1f6c36f-ovnkube-script-lib\") pod \"ovnkube-node-hww8q\" (UID: \"5c797b08-a314-44ed-8026-8a0ce1f6c36f\") " pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:40.740819 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.740613    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68r2v\" (UniqueName: \"kubernetes.io/projected/a81017e4-d705-438e-833b-eedd6de0edde-kube-api-access-68r2v\") pod \"multus-additional-cni-plugins-69j4f\" (UID: \"a81017e4-d705-438e-833b-eedd6de0edde\") " pod="openshift-multus/multus-additional-cni-plugins-69j4f"
Nov 02 02:47:40.740819 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.740636    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/086e3504-d2fe-43ad-bc5e-dca409a4e695-metrics-certs\") pod \"network-metrics-daemon-zkjqm\" (UID: \"086e3504-d2fe-43ad-bc5e-dca409a4e695\") " pod="openshift-multus/network-metrics-daemon-zkjqm"
Nov 02 02:47:40.740819 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.740652    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c078a1ee-3179-45c9-9d3b-4d4a46f0c42f-agent-certs\") pod \"konnectivity-agent-z8sq6\" (UID: \"c078a1ee-3179-45c9-9d3b-4d4a46f0c42f\") " pod="kube-system/konnectivity-agent-z8sq6"
Nov 02 02:47:40.740819 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.740667    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c2f75412-4413-45d1-aa82-919a5206efae-host-var-lib-cni-bin\") pod \"multus-tt68t\" (UID: \"c2f75412-4413-45d1-aa82-919a5206efae\") " pod="openshift-multus/multus-tt68t"
Nov 02 02:47:40.740819 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.740683    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/94fb7403-b6ef-4d42-be18-7c45b5c10692-var-lib-kubelet\") pod \"tuned-7qpcf\" (UID: \"94fb7403-b6ef-4d42-be18-7c45b5c10692\") " pod="openshift-cluster-node-tuning-operator/tuned-7qpcf"
Nov 02 02:47:40.740819 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.740794    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5c797b08-a314-44ed-8026-8a0ce1f6c36f-host-run-netns\") pod \"ovnkube-node-hww8q\" (UID: \"5c797b08-a314-44ed-8026-8a0ce1f6c36f\") " pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:40.741359 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.740834    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpg9j\" (UniqueName: \"kubernetes.io/projected/d66bfee1-59f7-4ed9-90e1-31582a4c4784-kube-api-access-vpg9j\") pod \"migrator-66df5d496b-sp26d\" (UID: \"d66bfee1-59f7-4ed9-90e1-31582a4c4784\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-sp26d"
Nov 02 02:47:40.741359 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.740877    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c2f75412-4413-45d1-aa82-919a5206efae-multus-daemon-config\") pod \"multus-tt68t\" (UID: \"c2f75412-4413-45d1-aa82-919a5206efae\") " pod="openshift-multus/multus-tt68t"
Nov 02 02:47:40.741359 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.740931    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzwzx\" (UniqueName: \"kubernetes.io/projected/4fec139d-da70-4cf6-8730-b578df3bebcc-kube-api-access-kzwzx\") pod \"node-resolver-sztr6\" (UID: \"4fec139d-da70-4cf6-8730-b578df3bebcc\") " pod="openshift-dns/node-resolver-sztr6"
Nov 02 02:47:40.741359 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.741038    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/39d09ab7-5724-44d4-a731-92eabb9280a6-signing-key\") pod \"service-ca-5fffc97bdf-rspt2\" (UID: \"39d09ab7-5724-44d4-a731-92eabb9280a6\") " pod="openshift-service-ca/service-ca-5fffc97bdf-rspt2"
Nov 02 02:47:40.741359 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.741059    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c078a1ee-3179-45c9-9d3b-4d4a46f0c42f-agent-certs\") pod \"konnectivity-agent-z8sq6\" (UID: \"c078a1ee-3179-45c9-9d3b-4d4a46f0c42f\") " pod="kube-system/konnectivity-agent-z8sq6"
Nov 02 02:47:40.741359 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.741109    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5c797b08-a314-44ed-8026-8a0ce1f6c36f-systemd-units\") pod \"ovnkube-node-hww8q\" (UID: \"5c797b08-a314-44ed-8026-8a0ce1f6c36f\") " pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:40.741359 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.741128    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5c797b08-a314-44ed-8026-8a0ce1f6c36f-host-slash\") pod \"ovnkube-node-hww8q\" (UID: \"5c797b08-a314-44ed-8026-8a0ce1f6c36f\") " pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:40.741359 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.741148    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5c797b08-a314-44ed-8026-8a0ce1f6c36f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hww8q\" (UID: \"5c797b08-a314-44ed-8026-8a0ce1f6c36f\") " pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:40.741359 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.741192    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/93e3306c-8c14-4bd4-ae04-08bc4b01a5fd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r4zs9\" (UID: \"93e3306c-8c14-4bd4-ae04-08bc4b01a5fd\") " pod="openshift-insights/insights-runtime-extractor-r4zs9"
Nov 02 02:47:40.741359 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.741255    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/93e3306c-8c14-4bd4-ae04-08bc4b01a5fd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r4zs9\" (UID: \"93e3306c-8c14-4bd4-ae04-08bc4b01a5fd\") " pod="openshift-insights/insights-runtime-extractor-r4zs9"
Nov 02 02:47:40.741359 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.741273    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bb7ec0c6-6988-45e6-b578-1b68b3a51d9d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zlplc\" (UID: \"bb7ec0c6-6988-45e6-b578-1b68b3a51d9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zlplc"
Nov 02 02:47:40.741359 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.741277    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/39d09ab7-5724-44d4-a731-92eabb9280a6-signing-key\") pod \"service-ca-5fffc97bdf-rspt2\" (UID: \"39d09ab7-5724-44d4-a731-92eabb9280a6\") " pod="openshift-service-ca/service-ca-5fffc97bdf-rspt2"
Nov 02 02:47:40.741359 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.741289    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a81017e4-d705-438e-833b-eedd6de0edde-cni-binary-copy\") pod \"multus-additional-cni-plugins-69j4f\" (UID: \"a81017e4-d705-438e-833b-eedd6de0edde\") " pod="openshift-multus/multus-additional-cni-plugins-69j4f"
Nov 02 02:47:40.741359 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.741318    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e400bad6-59c3-4c43-aed0-40c731c967db-dbus\") pod \"global-pull-secret-syncer-dxlld\" (UID: \"e400bad6-59c3-4c43-aed0-40c731c967db\") " pod="kube-system/global-pull-secret-syncer-dxlld"
Nov 02 02:47:40.741359 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.741362    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/94fb7403-b6ef-4d42-be18-7c45b5c10692-lib-modules\") pod \"tuned-7qpcf\" (UID: \"94fb7403-b6ef-4d42-be18-7c45b5c10692\") " pod="openshift-cluster-node-tuning-operator/tuned-7qpcf"
Nov 02 02:47:40.741760 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.741389    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5c797b08-a314-44ed-8026-8a0ce1f6c36f-run-ovn\") pod \"ovnkube-node-hww8q\" (UID: \"5c797b08-a314-44ed-8026-8a0ce1f6c36f\") " pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:40.741760 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.741434    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5c797b08-a314-44ed-8026-8a0ce1f6c36f-log-socket\") pod \"ovnkube-node-hww8q\" (UID: \"5c797b08-a314-44ed-8026-8a0ce1f6c36f\") " pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:40.741760 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.741466    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5c797b08-a314-44ed-8026-8a0ce1f6c36f-host-cni-netd\") pod \"ovnkube-node-hww8q\" (UID: \"5c797b08-a314-44ed-8026-8a0ce1f6c36f\") " pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:40.741760 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.741479    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a81017e4-d705-438e-833b-eedd6de0edde-cni-binary-copy\") pod \"multus-additional-cni-plugins-69j4f\" (UID: \"a81017e4-d705-438e-833b-eedd6de0edde\") " pod="openshift-multus/multus-additional-cni-plugins-69j4f"
Nov 02 02:47:40.741760 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.741501    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/93e3306c-8c14-4bd4-ae04-08bc4b01a5fd-data-volume\") pod \"insights-runtime-extractor-r4zs9\" (UID: \"93e3306c-8c14-4bd4-ae04-08bc4b01a5fd\") " pod="openshift-insights/insights-runtime-extractor-r4zs9"
Nov 02 02:47:40.741760 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.741580    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/93e3306c-8c14-4bd4-ae04-08bc4b01a5fd-data-volume\") pod \"insights-runtime-extractor-r4zs9\" (UID: \"93e3306c-8c14-4bd4-ae04-08bc4b01a5fd\") " pod="openshift-insights/insights-runtime-extractor-r4zs9"
Nov 02 02:47:40.782381 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.782323    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Nov 02 02:47:40.782381 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.782324    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 02 02:47:40.782511 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.782383    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 02 02:47:40.782511 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.782389    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Nov 02 02:47:40.782511 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.782465    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 02 02:47:40.782511 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.782466    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 02 02:47:40.782511 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.782480    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 02 02:47:40.782511 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.782480    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 02 02:47:40.782765 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.782476    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Nov 02 02:47:40.782765 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.782615    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 02 02:47:40.782765 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.782635    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Nov 02 02:47:40.782765 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.782703    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 02 02:47:40.782765 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.782760    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 02 02:47:40.782971 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.782861    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 02 02:47:40.782971 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.782942    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 02 02:47:40.783094 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.782982    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 02 02:47:40.783094 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.783001    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 02 02:47:40.783094 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.783025    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 02 02:47:40.783094 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.783037    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 02 02:47:40.783094 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.783040    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 02 02:47:40.783094 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.783069    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 02 02:47:40.783094 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.783074    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 02 02:47:40.783094 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.783085    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 02 02:47:40.783445 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.783130    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 02 02:47:40.783445 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.783134    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 02 02:47:40.783445 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.783149    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 02 02:47:40.783445 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.783159    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 02 02:47:40.783445 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.783209    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Nov 02 02:47:40.783445 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.783217    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 02 02:47:40.783445 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.783252    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 02 02:47:40.783651 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.783491    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 02 02:47:40.783651 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.783551    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 02 02:47:40.783722 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.783708    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 02 02:47:40.788818 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.788800    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a81017e4-d705-438e-833b-eedd6de0edde-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-69j4f\" (UID: \"a81017e4-d705-438e-833b-eedd6de0edde\") " pod="openshift-multus/multus-additional-cni-plugins-69j4f"
Nov 02 02:47:40.789960 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.789945    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2d4b0e00-8bb2-4b4c-9805-af5c51c65405-config-volume\") pod \"dns-default-fbvxx\" (UID: \"2d4b0e00-8bb2-4b4c-9805-af5c51c65405\") " pod="openshift-dns/dns-default-fbvxx"
Nov 02 02:47:40.790062 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.789996    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e701e783-5b43-45d7-b88b-83fd3c4af13b-serviceca\") pod \"node-ca-kkqzw\" (UID: \"e701e783-5b43-45d7-b88b-83fd3c4af13b\") " pod="openshift-image-registry/node-ca-kkqzw"
Nov 02 02:47:40.790062 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.790004    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5c797b08-a314-44ed-8026-8a0ce1f6c36f-ovnkube-config\") pod \"ovnkube-node-hww8q\" (UID: \"5c797b08-a314-44ed-8026-8a0ce1f6c36f\") " pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:40.790062 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.790004    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95effc61-7dc3-42c1-9609-be607c3bc65f-service-ca-bundle\") pod \"periodic-gathering-t9g7w-hqs6p\" (UID: \"95effc61-7dc3-42c1-9609-be607c3bc65f\") " pod="openshift-insights/periodic-gathering-t9g7w-hqs6p"
Nov 02 02:47:40.790062 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.790050    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c078a1ee-3179-45c9-9d3b-4d4a46f0c42f-konnectivity-ca\") pod \"konnectivity-agent-z8sq6\" (UID: \"c078a1ee-3179-45c9-9d3b-4d4a46f0c42f\") " pod="kube-system/konnectivity-agent-z8sq6"
Nov 02 02:47:40.790260 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.790086    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/39d09ab7-5724-44d4-a731-92eabb9280a6-signing-cabundle\") pod \"service-ca-5fffc97bdf-rspt2\" (UID: \"39d09ab7-5724-44d4-a731-92eabb9280a6\") " pod="openshift-service-ca/service-ca-5fffc97bdf-rspt2"
Nov 02 02:47:40.790260 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.790223    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95effc61-7dc3-42c1-9609-be607c3bc65f-trusted-ca-bundle\") pod \"periodic-gathering-t9g7w-hqs6p\" (UID: \"95effc61-7dc3-42c1-9609-be607c3bc65f\") " pod="openshift-insights/periodic-gathering-t9g7w-hqs6p"
Nov 02 02:47:40.790410 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.790394    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a81017e4-d705-438e-833b-eedd6de0edde-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-69j4f\" (UID: \"a81017e4-d705-438e-833b-eedd6de0edde\") " pod="openshift-multus/multus-additional-cni-plugins-69j4f"
Nov 02 02:47:40.790717 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.790703    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5c797b08-a314-44ed-8026-8a0ce1f6c36f-env-overrides\") pod \"ovnkube-node-hww8q\" (UID: \"5c797b08-a314-44ed-8026-8a0ce1f6c36f\") " pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:40.790835 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.790820    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5c797b08-a314-44ed-8026-8a0ce1f6c36f-ovnkube-script-lib\") pod \"ovnkube-node-hww8q\" (UID: \"5c797b08-a314-44ed-8026-8a0ce1f6c36f\") " pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:40.791865 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.791850    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c2f75412-4413-45d1-aa82-919a5206efae-multus-daemon-config\") pod \"multus-tt68t\" (UID: \"c2f75412-4413-45d1-aa82-919a5206efae\") " pod="openshift-multus/multus-tt68t"
Nov 02 02:47:40.793764 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.793745    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqsvj\" (UniqueName: \"kubernetes.io/projected/e701e783-5b43-45d7-b88b-83fd3c4af13b-kube-api-access-kqsvj\") pod \"node-ca-kkqzw\" (UID: \"e701e783-5b43-45d7-b88b-83fd3c4af13b\") " pod="openshift-image-registry/node-ca-kkqzw"
Nov 02 02:47:40.794129 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.794112    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmz88\" (UniqueName: \"kubernetes.io/projected/797676e6-5545-41ad-a4d1-c15f26db122a-kube-api-access-zmz88\") pod \"ingress-canary-zt7hz\" (UID: \"797676e6-5545-41ad-a4d1-c15f26db122a\") " pod="openshift-ingress-canary/ingress-canary-zt7hz"
Nov 02 02:47:40.794487 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.794471    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xrdb\" (UniqueName: \"kubernetes.io/projected/2d4b0e00-8bb2-4b4c-9805-af5c51c65405-kube-api-access-4xrdb\") pod \"dns-default-fbvxx\" (UID: \"2d4b0e00-8bb2-4b4c-9805-af5c51c65405\") " pod="openshift-dns/dns-default-fbvxx"
Nov 02 02:47:40.794581 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.794565    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqtjv\" (UniqueName: \"kubernetes.io/projected/94fb7403-b6ef-4d42-be18-7c45b5c10692-kube-api-access-cqtjv\") pod \"tuned-7qpcf\" (UID: \"94fb7403-b6ef-4d42-be18-7c45b5c10692\") " pod="openshift-cluster-node-tuning-operator/tuned-7qpcf"
Nov 02 02:47:40.795001 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.794986    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5dxg\" (UniqueName: \"kubernetes.io/projected/5c797b08-a314-44ed-8026-8a0ce1f6c36f-kube-api-access-b5dxg\") pod \"ovnkube-node-hww8q\" (UID: \"5c797b08-a314-44ed-8026-8a0ce1f6c36f\") " pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:40.795266 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.795252    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-984cg\" (UniqueName: \"kubernetes.io/projected/086e3504-d2fe-43ad-bc5e-dca409a4e695-kube-api-access-984cg\") pod \"network-metrics-daemon-zkjqm\" (UID: \"086e3504-d2fe-43ad-bc5e-dca409a4e695\") " pod="openshift-multus/network-metrics-daemon-zkjqm"
Nov 02 02:47:40.795616 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.795601    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hclh\" (UniqueName: \"kubernetes.io/projected/c2f75412-4413-45d1-aa82-919a5206efae-kube-api-access-5hclh\") pod \"multus-tt68t\" (UID: \"c2f75412-4413-45d1-aa82-919a5206efae\") " pod="openshift-multus/multus-tt68t"
Nov 02 02:47:40.799200 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.799179    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ln99j\" (UniqueName: \"kubernetes.io/projected/bb7ec0c6-6988-45e6-b578-1b68b3a51d9d-kube-api-access-ln99j\") pod \"aws-ebs-csi-driver-node-zlplc\" (UID: \"bb7ec0c6-6988-45e6-b578-1b68b3a51d9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zlplc"
Nov 02 02:47:40.799200 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.799191    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rrzn\" (UniqueName: \"kubernetes.io/projected/44f31bc6-bd01-4f16-ad6e-0d1ec90bfb4f-kube-api-access-7rrzn\") pod \"network-check-target-47f7x\" (UID: \"44f31bc6-bd01-4f16-ad6e-0d1ec90bfb4f\") " pod="openshift-network-diagnostics/network-check-target-47f7x"
Nov 02 02:47:40.799417 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.799400    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dl7x\" (UniqueName: \"kubernetes.io/projected/95effc61-7dc3-42c1-9609-be607c3bc65f-kube-api-access-2dl7x\") pod \"periodic-gathering-t9g7w-hqs6p\" (UID: \"95effc61-7dc3-42c1-9609-be607c3bc65f\") " pod="openshift-insights/periodic-gathering-t9g7w-hqs6p"
Nov 02 02:47:40.800056 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.800039    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69zxc\" (UniqueName: \"kubernetes.io/projected/f2071fbb-7d2f-49d4-9366-34332e71c6bc-kube-api-access-69zxc\") pod \"iptables-alerter-8sxd5\" (UID: \"f2071fbb-7d2f-49d4-9366-34332e71c6bc\") " pod="openshift-network-operator/iptables-alerter-8sxd5"
Nov 02 02:47:40.800201 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.800182    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpg9j\" (UniqueName: \"kubernetes.io/projected/d66bfee1-59f7-4ed9-90e1-31582a4c4784-kube-api-access-vpg9j\") pod \"migrator-66df5d496b-sp26d\" (UID: \"d66bfee1-59f7-4ed9-90e1-31582a4c4784\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-sp26d"
Nov 02 02:47:40.800342 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.800323    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thncs\" (UniqueName: \"kubernetes.io/projected/39d09ab7-5724-44d4-a731-92eabb9280a6-kube-api-access-thncs\") pod \"service-ca-5fffc97bdf-rspt2\" (UID: \"39d09ab7-5724-44d4-a731-92eabb9280a6\") " pod="openshift-service-ca/service-ca-5fffc97bdf-rspt2"
Nov 02 02:47:40.800670 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.800651    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68r2v\" (UniqueName: \"kubernetes.io/projected/a81017e4-d705-438e-833b-eedd6de0edde-kube-api-access-68r2v\") pod \"multus-additional-cni-plugins-69j4f\" (UID: \"a81017e4-d705-438e-833b-eedd6de0edde\") " pod="openshift-multus/multus-additional-cni-plugins-69j4f"
Nov 02 02:47:40.800749 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.800696    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b47mc\" (UniqueName: \"kubernetes.io/projected/93e3306c-8c14-4bd4-ae04-08bc4b01a5fd-kube-api-access-b47mc\") pod \"insights-runtime-extractor-r4zs9\" (UID: \"93e3306c-8c14-4bd4-ae04-08bc4b01a5fd\") " pod="openshift-insights/insights-runtime-extractor-r4zs9"
Nov 02 02:47:40.800804 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.800761    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzwzx\" (UniqueName: \"kubernetes.io/projected/4fec139d-da70-4cf6-8730-b578df3bebcc-kube-api-access-kzwzx\") pod \"node-resolver-sztr6\" (UID: \"4fec139d-da70-4cf6-8730-b578df3bebcc\") " pod="openshift-dns/node-resolver-sztr6"
Nov 02 02:47:40.841789 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.841768    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/93e3306c-8c14-4bd4-ae04-08bc4b01a5fd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r4zs9\" (UID: \"93e3306c-8c14-4bd4-ae04-08bc4b01a5fd\") " pod="openshift-insights/insights-runtime-extractor-r4zs9"
Nov 02 02:47:40.841871 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.841800    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/93e3306c-8c14-4bd4-ae04-08bc4b01a5fd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r4zs9\" (UID: \"93e3306c-8c14-4bd4-ae04-08bc4b01a5fd\") " pod="openshift-insights/insights-runtime-extractor-r4zs9"
Nov 02 02:47:40.841871 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.841824    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bb7ec0c6-6988-45e6-b578-1b68b3a51d9d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zlplc\" (UID: \"bb7ec0c6-6988-45e6-b578-1b68b3a51d9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zlplc"
Nov 02 02:47:40.841871 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.841847    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e400bad6-59c3-4c43-aed0-40c731c967db-dbus\") pod \"global-pull-secret-syncer-dxlld\" (UID: \"e400bad6-59c3-4c43-aed0-40c731c967db\") " pod="kube-system/global-pull-secret-syncer-dxlld"
Nov 02 02:47:40.842007 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.841869    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/94fb7403-b6ef-4d42-be18-7c45b5c10692-lib-modules\") pod \"tuned-7qpcf\" (UID: \"94fb7403-b6ef-4d42-be18-7c45b5c10692\") " pod="openshift-cluster-node-tuning-operator/tuned-7qpcf"
Nov 02 02:47:40.842007 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.841895    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5c797b08-a314-44ed-8026-8a0ce1f6c36f-run-ovn\") pod \"ovnkube-node-hww8q\" (UID: \"5c797b08-a314-44ed-8026-8a0ce1f6c36f\") " pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:40.842007 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.841909    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bb7ec0c6-6988-45e6-b578-1b68b3a51d9d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zlplc\" (UID: \"bb7ec0c6-6988-45e6-b578-1b68b3a51d9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zlplc"
Nov 02 02:47:40.842007 ip-10-0-133-163 kubenswrapper[4496]: E1102 02:47:40.841926    4496 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 02:47:40.842007 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.841918    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5c797b08-a314-44ed-8026-8a0ce1f6c36f-log-socket\") pod \"ovnkube-node-hww8q\" (UID: \"5c797b08-a314-44ed-8026-8a0ce1f6c36f\") " pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:40.842007 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.841959    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5c797b08-a314-44ed-8026-8a0ce1f6c36f-log-socket\") pod \"ovnkube-node-hww8q\" (UID: \"5c797b08-a314-44ed-8026-8a0ce1f6c36f\") " pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:40.842007 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.841985    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5c797b08-a314-44ed-8026-8a0ce1f6c36f-run-ovn\") pod \"ovnkube-node-hww8q\" (UID: \"5c797b08-a314-44ed-8026-8a0ce1f6c36f\") " pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:40.842007 ip-10-0-133-163 kubenswrapper[4496]: E1102 02:47:40.842007    4496 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93e3306c-8c14-4bd4-ae04-08bc4b01a5fd-insights-runtime-extractor-tls podName:93e3306c-8c14-4bd4-ae04-08bc4b01a5fd nodeName:}" failed. No retries permitted until 2026-11-02 02:47:41.341993158 +0000 UTC m=+1.920600534 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/93e3306c-8c14-4bd4-ae04-08bc4b01a5fd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-r4zs9" (UID: "93e3306c-8c14-4bd4-ae04-08bc4b01a5fd") : secret "insights-runtime-extractor-tls" not found
Nov 02 02:47:40.842007 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.841925    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e400bad6-59c3-4c43-aed0-40c731c967db-dbus\") pod \"global-pull-secret-syncer-dxlld\" (UID: \"e400bad6-59c3-4c43-aed0-40c731c967db\") " pod="kube-system/global-pull-secret-syncer-dxlld"
Nov 02 02:47:40.842303 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.842040    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5c797b08-a314-44ed-8026-8a0ce1f6c36f-host-cni-netd\") pod \"ovnkube-node-hww8q\" (UID: \"5c797b08-a314-44ed-8026-8a0ce1f6c36f\") " pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:40.842303 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.842042    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/94fb7403-b6ef-4d42-be18-7c45b5c10692-lib-modules\") pod \"tuned-7qpcf\" (UID: \"94fb7403-b6ef-4d42-be18-7c45b5c10692\") " pod="openshift-cluster-node-tuning-operator/tuned-7qpcf"
Nov 02 02:47:40.842303 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.842061    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c2f75412-4413-45d1-aa82-919a5206efae-host-run-k8s-cni-cncf-io\") pod \"multus-tt68t\" (UID: \"c2f75412-4413-45d1-aa82-919a5206efae\") " pod="openshift-multus/multus-tt68t"
Nov 02 02:47:40.842303 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.842069    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5c797b08-a314-44ed-8026-8a0ce1f6c36f-host-cni-netd\") pod \"ovnkube-node-hww8q\" (UID: \"5c797b08-a314-44ed-8026-8a0ce1f6c36f\") " pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:40.842303 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.842077    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/94fb7403-b6ef-4d42-be18-7c45b5c10692-etc-modprobe-d\") pod \"tuned-7qpcf\" (UID: \"94fb7403-b6ef-4d42-be18-7c45b5c10692\") " pod="openshift-cluster-node-tuning-operator/tuned-7qpcf"
Nov 02 02:47:40.842303 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.842094    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c2f75412-4413-45d1-aa82-919a5206efae-host-run-k8s-cni-cncf-io\") pod \"multus-tt68t\" (UID: \"c2f75412-4413-45d1-aa82-919a5206efae\") " pod="openshift-multus/multus-tt68t"
Nov 02 02:47:40.842303 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.842104    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5c797b08-a314-44ed-8026-8a0ce1f6c36f-node-log\") pod \"ovnkube-node-hww8q\" (UID: \"5c797b08-a314-44ed-8026-8a0ce1f6c36f\") " pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:40.842303 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.842129    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bb7ec0c6-6988-45e6-b578-1b68b3a51d9d-socket-dir\") pod \"aws-ebs-csi-driver-node-zlplc\" (UID: \"bb7ec0c6-6988-45e6-b578-1b68b3a51d9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zlplc"
Nov 02 02:47:40.842303 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.842134    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5c797b08-a314-44ed-8026-8a0ce1f6c36f-node-log\") pod \"ovnkube-node-hww8q\" (UID: \"5c797b08-a314-44ed-8026-8a0ce1f6c36f\") " pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:40.842303 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.842145    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bb7ec0c6-6988-45e6-b578-1b68b3a51d9d-device-dir\") pod \"aws-ebs-csi-driver-node-zlplc\" (UID: \"bb7ec0c6-6988-45e6-b578-1b68b3a51d9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zlplc"
Nov 02 02:47:40.842303 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.842165    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c2f75412-4413-45d1-aa82-919a5206efae-cnibin\") pod \"multus-tt68t\" (UID: \"c2f75412-4413-45d1-aa82-919a5206efae\") " pod="openshift-multus/multus-tt68t"
Nov 02 02:47:40.842303 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.842174    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bb7ec0c6-6988-45e6-b578-1b68b3a51d9d-socket-dir\") pod \"aws-ebs-csi-driver-node-zlplc\" (UID: \"bb7ec0c6-6988-45e6-b578-1b68b3a51d9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zlplc"
Nov 02 02:47:40.842303 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.842144    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/94fb7403-b6ef-4d42-be18-7c45b5c10692-etc-modprobe-d\") pod \"tuned-7qpcf\" (UID: \"94fb7403-b6ef-4d42-be18-7c45b5c10692\") " pod="openshift-cluster-node-tuning-operator/tuned-7qpcf"
Nov 02 02:47:40.842303 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.842181    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bb7ec0c6-6988-45e6-b578-1b68b3a51d9d-device-dir\") pod \"aws-ebs-csi-driver-node-zlplc\" (UID: \"bb7ec0c6-6988-45e6-b578-1b68b3a51d9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zlplc"
Nov 02 02:47:40.842303 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.842192    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/94fb7403-b6ef-4d42-be18-7c45b5c10692-etc-sysctl-d\") pod \"tuned-7qpcf\" (UID: \"94fb7403-b6ef-4d42-be18-7c45b5c10692\") " pod="openshift-cluster-node-tuning-operator/tuned-7qpcf"
Nov 02 02:47:40.842303 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.842214    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c2f75412-4413-45d1-aa82-919a5206efae-cnibin\") pod \"multus-tt68t\" (UID: \"c2f75412-4413-45d1-aa82-919a5206efae\") " pod="openshift-multus/multus-tt68t"
Nov 02 02:47:40.842303 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.842217    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/94fb7403-b6ef-4d42-be18-7c45b5c10692-etc-systemd\") pod \"tuned-7qpcf\" (UID: \"94fb7403-b6ef-4d42-be18-7c45b5c10692\") " pod="openshift-cluster-node-tuning-operator/tuned-7qpcf"
Nov 02 02:47:40.842303 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.842251    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/94fb7403-b6ef-4d42-be18-7c45b5c10692-etc-sysctl-d\") pod \"tuned-7qpcf\" (UID: \"94fb7403-b6ef-4d42-be18-7c45b5c10692\") " pod="openshift-cluster-node-tuning-operator/tuned-7qpcf"
Nov 02 02:47:40.842303 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.842254    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c2f75412-4413-45d1-aa82-919a5206efae-host-run-netns\") pod \"multus-tt68t\" (UID: \"c2f75412-4413-45d1-aa82-919a5206efae\") " pod="openshift-multus/multus-tt68t"
Nov 02 02:47:40.842303 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.842261    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/94fb7403-b6ef-4d42-be18-7c45b5c10692-etc-systemd\") pod \"tuned-7qpcf\" (UID: \"94fb7403-b6ef-4d42-be18-7c45b5c10692\") " pod="openshift-cluster-node-tuning-operator/tuned-7qpcf"
Nov 02 02:47:40.842303 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.842283    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c2f75412-4413-45d1-aa82-919a5206efae-multus-conf-dir\") pod \"multus-tt68t\" (UID: \"c2f75412-4413-45d1-aa82-919a5206efae\") " pod="openshift-multus/multus-tt68t"
Nov 02 02:47:40.842303 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.842300    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/93e3306c-8c14-4bd4-ae04-08bc4b01a5fd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r4zs9\" (UID: \"93e3306c-8c14-4bd4-ae04-08bc4b01a5fd\") " pod="openshift-insights/insights-runtime-extractor-r4zs9"
Nov 02 02:47:40.842303 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.842301    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c2f75412-4413-45d1-aa82-919a5206efae-host-run-multus-certs\") pod \"multus-tt68t\" (UID: \"c2f75412-4413-45d1-aa82-919a5206efae\") " pod="openshift-multus/multus-tt68t"
Nov 02 02:47:40.843218 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.842321    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c2f75412-4413-45d1-aa82-919a5206efae-host-run-multus-certs\") pod \"multus-tt68t\" (UID: \"c2f75412-4413-45d1-aa82-919a5206efae\") " pod="openshift-multus/multus-tt68t"
Nov 02 02:47:40.843218 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.842341    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c2f75412-4413-45d1-aa82-919a5206efae-multus-conf-dir\") pod \"multus-tt68t\" (UID: \"c2f75412-4413-45d1-aa82-919a5206efae\") " pod="openshift-multus/multus-tt68t"
Nov 02 02:47:40.843218 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.842345    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/94fb7403-b6ef-4d42-be18-7c45b5c10692-etc-sysctl-conf\") pod \"tuned-7qpcf\" (UID: \"94fb7403-b6ef-4d42-be18-7c45b5c10692\") " pod="openshift-cluster-node-tuning-operator/tuned-7qpcf"
Nov 02 02:47:40.843218 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.842347    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c2f75412-4413-45d1-aa82-919a5206efae-host-run-netns\") pod \"multus-tt68t\" (UID: \"c2f75412-4413-45d1-aa82-919a5206efae\") " pod="openshift-multus/multus-tt68t"
Nov 02 02:47:40.843218 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.842363    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/94fb7403-b6ef-4d42-be18-7c45b5c10692-sys\") pod \"tuned-7qpcf\" (UID: \"94fb7403-b6ef-4d42-be18-7c45b5c10692\") " pod="openshift-cluster-node-tuning-operator/tuned-7qpcf"
Nov 02 02:47:40.843218 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.842377    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5c797b08-a314-44ed-8026-8a0ce1f6c36f-host-kubelet\") pod \"ovnkube-node-hww8q\" (UID: \"5c797b08-a314-44ed-8026-8a0ce1f6c36f\") " pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:40.843218 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.842391    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bb7ec0c6-6988-45e6-b578-1b68b3a51d9d-etc-selinux\") pod \"aws-ebs-csi-driver-node-zlplc\" (UID: \"bb7ec0c6-6988-45e6-b578-1b68b3a51d9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zlplc"
Nov 02 02:47:40.843218 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.842405    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a81017e4-d705-438e-833b-eedd6de0edde-cnibin\") pod \"multus-additional-cni-plugins-69j4f\" (UID: \"a81017e4-d705-438e-833b-eedd6de0edde\") " pod="openshift-multus/multus-additional-cni-plugins-69j4f"
Nov 02 02:47:40.843218 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.842415    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/94fb7403-b6ef-4d42-be18-7c45b5c10692-etc-sysctl-conf\") pod \"tuned-7qpcf\" (UID: \"94fb7403-b6ef-4d42-be18-7c45b5c10692\") " pod="openshift-cluster-node-tuning-operator/tuned-7qpcf"
Nov 02 02:47:40.843218 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.842419    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f2071fbb-7d2f-49d4-9366-34332e71c6bc-host-slash\") pod \"iptables-alerter-8sxd5\" (UID: \"f2071fbb-7d2f-49d4-9366-34332e71c6bc\") " pod="openshift-network-operator/iptables-alerter-8sxd5"
Nov 02 02:47:40.843218 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.842435    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a81017e4-d705-438e-833b-eedd6de0edde-tuning-conf-dir\") pod \"multus-additional-cni-plugins-69j4f\" (UID: \"a81017e4-d705-438e-833b-eedd6de0edde\") " pod="openshift-multus/multus-additional-cni-plugins-69j4f"
Nov 02 02:47:40.843218 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.842445    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/94fb7403-b6ef-4d42-be18-7c45b5c10692-sys\") pod \"tuned-7qpcf\" (UID: \"94fb7403-b6ef-4d42-be18-7c45b5c10692\") " pod="openshift-cluster-node-tuning-operator/tuned-7qpcf"
Nov 02 02:47:40.843218 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.842420    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5c797b08-a314-44ed-8026-8a0ce1f6c36f-host-kubelet\") pod \"ovnkube-node-hww8q\" (UID: \"5c797b08-a314-44ed-8026-8a0ce1f6c36f\") " pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:40.843218 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.842447    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bb7ec0c6-6988-45e6-b578-1b68b3a51d9d-etc-selinux\") pod \"aws-ebs-csi-driver-node-zlplc\" (UID: \"bb7ec0c6-6988-45e6-b578-1b68b3a51d9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zlplc"
Nov 02 02:47:40.843218 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.842456    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c2f75412-4413-45d1-aa82-919a5206efae-multus-cni-dir\") pod \"multus-tt68t\" (UID: \"c2f75412-4413-45d1-aa82-919a5206efae\") " pod="openshift-multus/multus-tt68t"
Nov 02 02:47:40.843218 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.842482    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f2071fbb-7d2f-49d4-9366-34332e71c6bc-host-slash\") pod \"iptables-alerter-8sxd5\" (UID: \"f2071fbb-7d2f-49d4-9366-34332e71c6bc\") " pod="openshift-network-operator/iptables-alerter-8sxd5"
Nov 02 02:47:40.843218 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.842484    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a81017e4-d705-438e-833b-eedd6de0edde-cnibin\") pod \"multus-additional-cni-plugins-69j4f\" (UID: \"a81017e4-d705-438e-833b-eedd6de0edde\") " pod="openshift-multus/multus-additional-cni-plugins-69j4f"
Nov 02 02:47:40.843218 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.842510    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c2f75412-4413-45d1-aa82-919a5206efae-multus-cni-dir\") pod \"multus-tt68t\" (UID: \"c2f75412-4413-45d1-aa82-919a5206efae\") " pod="openshift-multus/multus-tt68t"
Nov 02 02:47:40.843218 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.842505    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c2f75412-4413-45d1-aa82-919a5206efae-hostroot\") pod \"multus-tt68t\" (UID: \"c2f75412-4413-45d1-aa82-919a5206efae\") " pod="openshift-multus/multus-tt68t"
Nov 02 02:47:40.843218 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.842560    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4fec139d-da70-4cf6-8730-b578df3bebcc-hosts-file\") pod \"node-resolver-sztr6\" (UID: \"4fec139d-da70-4cf6-8730-b578df3bebcc\") " pod="openshift-dns/node-resolver-sztr6"
Nov 02 02:47:40.843218 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.842530    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a81017e4-d705-438e-833b-eedd6de0edde-tuning-conf-dir\") pod \"multus-additional-cni-plugins-69j4f\" (UID: \"a81017e4-d705-438e-833b-eedd6de0edde\") " pod="openshift-multus/multus-additional-cni-plugins-69j4f"
Nov 02 02:47:40.843218 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.842576    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c2f75412-4413-45d1-aa82-919a5206efae-multus-socket-dir-parent\") pod \"multus-tt68t\" (UID: \"c2f75412-4413-45d1-aa82-919a5206efae\") " pod="openshift-multus/multus-tt68t"
Nov 02 02:47:40.843218 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.842596    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c2f75412-4413-45d1-aa82-919a5206efae-hostroot\") pod \"multus-tt68t\" (UID: \"c2f75412-4413-45d1-aa82-919a5206efae\") " pod="openshift-multus/multus-tt68t"
Nov 02 02:47:40.843218 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.842607    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4fec139d-da70-4cf6-8730-b578df3bebcc-hosts-file\") pod \"node-resolver-sztr6\" (UID: \"4fec139d-da70-4cf6-8730-b578df3bebcc\") " pod="openshift-dns/node-resolver-sztr6"
Nov 02 02:47:40.843218 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.842609    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c2f75412-4413-45d1-aa82-919a5206efae-multus-socket-dir-parent\") pod \"multus-tt68t\" (UID: \"c2f75412-4413-45d1-aa82-919a5206efae\") " pod="openshift-multus/multus-tt68t"
Nov 02 02:47:40.843218 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.842631    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5c797b08-a314-44ed-8026-8a0ce1f6c36f-run-systemd\") pod \"ovnkube-node-hww8q\" (UID: \"5c797b08-a314-44ed-8026-8a0ce1f6c36f\") " pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:40.843218 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.842645    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5c797b08-a314-44ed-8026-8a0ce1f6c36f-host-cni-bin\") pod \"ovnkube-node-hww8q\" (UID: \"5c797b08-a314-44ed-8026-8a0ce1f6c36f\") " pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:40.843218 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.842668    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5c797b08-a314-44ed-8026-8a0ce1f6c36f-host-cni-bin\") pod \"ovnkube-node-hww8q\" (UID: \"5c797b08-a314-44ed-8026-8a0ce1f6c36f\") " pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:40.843218 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.842681    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5c797b08-a314-44ed-8026-8a0ce1f6c36f-run-systemd\") pod \"ovnkube-node-hww8q\" (UID: \"5c797b08-a314-44ed-8026-8a0ce1f6c36f\") " pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:40.843218 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.842716    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c2f75412-4413-45d1-aa82-919a5206efae-system-cni-dir\") pod \"multus-tt68t\" (UID: \"c2f75412-4413-45d1-aa82-919a5206efae\") " pod="openshift-multus/multus-tt68t"
Nov 02 02:47:40.843218 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.842741    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c2f75412-4413-45d1-aa82-919a5206efae-host-var-lib-kubelet\") pod \"multus-tt68t\" (UID: \"c2f75412-4413-45d1-aa82-919a5206efae\") " pod="openshift-multus/multus-tt68t"
Nov 02 02:47:40.843218 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.842760    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c2f75412-4413-45d1-aa82-919a5206efae-system-cni-dir\") pod \"multus-tt68t\" (UID: \"c2f75412-4413-45d1-aa82-919a5206efae\") " pod="openshift-multus/multus-tt68t"
Nov 02 02:47:40.843218 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.842764    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2d4b0e00-8bb2-4b4c-9805-af5c51c65405-metrics-tls\") pod \"dns-default-fbvxx\" (UID: \"2d4b0e00-8bb2-4b4c-9805-af5c51c65405\") " pod="openshift-dns/dns-default-fbvxx"
Nov 02 02:47:40.843218 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.842783    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c2f75412-4413-45d1-aa82-919a5206efae-host-var-lib-kubelet\") pod \"multus-tt68t\" (UID: \"c2f75412-4413-45d1-aa82-919a5206efae\") " pod="openshift-multus/multus-tt68t"
Nov 02 02:47:40.843218 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.842789    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5c797b08-a314-44ed-8026-8a0ce1f6c36f-var-lib-openvswitch\") pod \"ovnkube-node-hww8q\" (UID: \"5c797b08-a314-44ed-8026-8a0ce1f6c36f\") " pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:40.843218 ip-10-0-133-163 kubenswrapper[4496]: E1102 02:47:40.842831    4496 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 02:47:40.843218 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.842837    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5c797b08-a314-44ed-8026-8a0ce1f6c36f-var-lib-openvswitch\") pod \"ovnkube-node-hww8q\" (UID: \"5c797b08-a314-44ed-8026-8a0ce1f6c36f\") " pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:40.843218 ip-10-0-133-163 kubenswrapper[4496]: E1102 02:47:40.842870    4496 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d4b0e00-8bb2-4b4c-9805-af5c51c65405-metrics-tls podName:2d4b0e00-8bb2-4b4c-9805-af5c51c65405 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:41.342859185 +0000 UTC m=+1.921466545 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2d4b0e00-8bb2-4b4c-9805-af5c51c65405-metrics-tls") pod "dns-default-fbvxx" (UID: "2d4b0e00-8bb2-4b4c-9805-af5c51c65405") : secret "dns-default-metrics-tls" not found
Nov 02 02:47:40.843218 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.842880    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/94fb7403-b6ef-4d42-be18-7c45b5c10692-run\") pod \"tuned-7qpcf\" (UID: \"94fb7403-b6ef-4d42-be18-7c45b5c10692\") " pod="openshift-cluster-node-tuning-operator/tuned-7qpcf"
Nov 02 02:47:40.843218 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.842894    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/94fb7403-b6ef-4d42-be18-7c45b5c10692-host\") pod \"tuned-7qpcf\" (UID: \"94fb7403-b6ef-4d42-be18-7c45b5c10692\") " pod="openshift-cluster-node-tuning-operator/tuned-7qpcf"
Nov 02 02:47:40.843218 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.842909    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5c797b08-a314-44ed-8026-8a0ce1f6c36f-host-run-ovn-kubernetes\") pod \"ovnkube-node-hww8q\" (UID: \"5c797b08-a314-44ed-8026-8a0ce1f6c36f\") " pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:40.843218 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.842951    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/94fb7403-b6ef-4d42-be18-7c45b5c10692-host\") pod \"tuned-7qpcf\" (UID: \"94fb7403-b6ef-4d42-be18-7c45b5c10692\") " pod="openshift-cluster-node-tuning-operator/tuned-7qpcf"
Nov 02 02:47:40.843218 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.842956    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/94fb7403-b6ef-4d42-be18-7c45b5c10692-run\") pod \"tuned-7qpcf\" (UID: \"94fb7403-b6ef-4d42-be18-7c45b5c10692\") " pod="openshift-cluster-node-tuning-operator/tuned-7qpcf"
Nov 02 02:47:40.843218 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.842959    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5c797b08-a314-44ed-8026-8a0ce1f6c36f-host-run-ovn-kubernetes\") pod \"ovnkube-node-hww8q\" (UID: \"5c797b08-a314-44ed-8026-8a0ce1f6c36f\") " pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:40.843218 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.842971    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/93e3306c-8c14-4bd4-ae04-08bc4b01a5fd-crio-socket\") pod \"insights-runtime-extractor-r4zs9\" (UID: \"93e3306c-8c14-4bd4-ae04-08bc4b01a5fd\") " pod="openshift-insights/insights-runtime-extractor-r4zs9"
Nov 02 02:47:40.843218 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.842995    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/93e3306c-8c14-4bd4-ae04-08bc4b01a5fd-crio-socket\") pod \"insights-runtime-extractor-r4zs9\" (UID: \"93e3306c-8c14-4bd4-ae04-08bc4b01a5fd\") " pod="openshift-insights/insights-runtime-extractor-r4zs9"
Nov 02 02:47:40.843218 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.842999    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e701e783-5b43-45d7-b88b-83fd3c4af13b-host\") pod \"node-ca-kkqzw\" (UID: \"e701e783-5b43-45d7-b88b-83fd3c4af13b\") " pod="openshift-image-registry/node-ca-kkqzw"
Nov 02 02:47:40.843218 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.843036    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/94fb7403-b6ef-4d42-be18-7c45b5c10692-etc-sysconfig\") pod \"tuned-7qpcf\" (UID: \"94fb7403-b6ef-4d42-be18-7c45b5c10692\") " pod="openshift-cluster-node-tuning-operator/tuned-7qpcf"
Nov 02 02:47:40.843218 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.843058    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/94fb7403-b6ef-4d42-be18-7c45b5c10692-etc-kubernetes\") pod \"tuned-7qpcf\" (UID: \"94fb7403-b6ef-4d42-be18-7c45b5c10692\") " pod="openshift-cluster-node-tuning-operator/tuned-7qpcf"
Nov 02 02:47:40.843218 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.843081    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5c797b08-a314-44ed-8026-8a0ce1f6c36f-run-openvswitch\") pod \"ovnkube-node-hww8q\" (UID: \"5c797b08-a314-44ed-8026-8a0ce1f6c36f\") " pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:40.843218 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.843103    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bb7ec0c6-6988-45e6-b578-1b68b3a51d9d-registration-dir\") pod \"aws-ebs-csi-driver-node-zlplc\" (UID: \"bb7ec0c6-6988-45e6-b578-1b68b3a51d9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zlplc"
Nov 02 02:47:40.843218 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.843110    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/94fb7403-b6ef-4d42-be18-7c45b5c10692-etc-kubernetes\") pod \"tuned-7qpcf\" (UID: \"94fb7403-b6ef-4d42-be18-7c45b5c10692\") " pod="openshift-cluster-node-tuning-operator/tuned-7qpcf"
Nov 02 02:47:40.843218 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.843113    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/94fb7403-b6ef-4d42-be18-7c45b5c10692-etc-sysconfig\") pod \"tuned-7qpcf\" (UID: \"94fb7403-b6ef-4d42-be18-7c45b5c10692\") " pod="openshift-cluster-node-tuning-operator/tuned-7qpcf"
Nov 02 02:47:40.843218 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.843112    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e701e783-5b43-45d7-b88b-83fd3c4af13b-host\") pod \"node-ca-kkqzw\" (UID: \"e701e783-5b43-45d7-b88b-83fd3c4af13b\") " pod="openshift-image-registry/node-ca-kkqzw"
Nov 02 02:47:40.843218 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.843126    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a81017e4-d705-438e-833b-eedd6de0edde-system-cni-dir\") pod \"multus-additional-cni-plugins-69j4f\" (UID: \"a81017e4-d705-438e-833b-eedd6de0edde\") " pod="openshift-multus/multus-additional-cni-plugins-69j4f"
Nov 02 02:47:40.843218 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.843148    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5c797b08-a314-44ed-8026-8a0ce1f6c36f-run-openvswitch\") pod \"ovnkube-node-hww8q\" (UID: \"5c797b08-a314-44ed-8026-8a0ce1f6c36f\") " pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:40.843218 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.843167    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bb7ec0c6-6988-45e6-b578-1b68b3a51d9d-registration-dir\") pod \"aws-ebs-csi-driver-node-zlplc\" (UID: \"bb7ec0c6-6988-45e6-b578-1b68b3a51d9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zlplc"
Nov 02 02:47:40.843218 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.843152    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a81017e4-d705-438e-833b-eedd6de0edde-system-cni-dir\") pod \"multus-additional-cni-plugins-69j4f\" (UID: \"a81017e4-d705-438e-833b-eedd6de0edde\") " pod="openshift-multus/multus-additional-cni-plugins-69j4f"
Nov 02 02:47:40.843218 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.843153    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bb7ec0c6-6988-45e6-b578-1b68b3a51d9d-sys-fs\") pod \"aws-ebs-csi-driver-node-zlplc\" (UID: \"bb7ec0c6-6988-45e6-b578-1b68b3a51d9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zlplc"
Nov 02 02:47:40.843218 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.843205    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bb7ec0c6-6988-45e6-b578-1b68b3a51d9d-sys-fs\") pod \"aws-ebs-csi-driver-node-zlplc\" (UID: \"bb7ec0c6-6988-45e6-b578-1b68b3a51d9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zlplc"
Nov 02 02:47:40.843218 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.843206    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e400bad6-59c3-4c43-aed0-40c731c967db-kubelet-config\") pod \"global-pull-secret-syncer-dxlld\" (UID: \"e400bad6-59c3-4c43-aed0-40c731c967db\") " pod="kube-system/global-pull-secret-syncer-dxlld"
Nov 02 02:47:40.843218 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.843236    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c2f75412-4413-45d1-aa82-919a5206efae-host-var-lib-cni-multus\") pod \"multus-tt68t\" (UID: \"c2f75412-4413-45d1-aa82-919a5206efae\") " pod="openshift-multus/multus-tt68t"
Nov 02 02:47:40.843218 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.843243    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e400bad6-59c3-4c43-aed0-40c731c967db-kubelet-config\") pod \"global-pull-secret-syncer-dxlld\" (UID: \"e400bad6-59c3-4c43-aed0-40c731c967db\") " pod="kube-system/global-pull-secret-syncer-dxlld"
Nov 02 02:47:40.844894 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.843262    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c2f75412-4413-45d1-aa82-919a5206efae-etc-kubernetes\") pod \"multus-tt68t\" (UID: \"c2f75412-4413-45d1-aa82-919a5206efae\") " pod="openshift-multus/multus-tt68t"
Nov 02 02:47:40.844894 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.843283    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5c797b08-a314-44ed-8026-8a0ce1f6c36f-etc-openvswitch\") pod \"ovnkube-node-hww8q\" (UID: \"5c797b08-a314-44ed-8026-8a0ce1f6c36f\") " pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:40.844894 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.843301    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/086e3504-d2fe-43ad-bc5e-dca409a4e695-metrics-certs\") pod \"network-metrics-daemon-zkjqm\" (UID: \"086e3504-d2fe-43ad-bc5e-dca409a4e695\") " pod="openshift-multus/network-metrics-daemon-zkjqm"
Nov 02 02:47:40.844894 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.843307    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c2f75412-4413-45d1-aa82-919a5206efae-host-var-lib-cni-multus\") pod \"multus-tt68t\" (UID: \"c2f75412-4413-45d1-aa82-919a5206efae\") " pod="openshift-multus/multus-tt68t"
Nov 02 02:47:40.844894 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.843323    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c2f75412-4413-45d1-aa82-919a5206efae-host-var-lib-cni-bin\") pod \"multus-tt68t\" (UID: \"c2f75412-4413-45d1-aa82-919a5206efae\") " pod="openshift-multus/multus-tt68t"
Nov 02 02:47:40.844894 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.843335    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c2f75412-4413-45d1-aa82-919a5206efae-etc-kubernetes\") pod \"multus-tt68t\" (UID: \"c2f75412-4413-45d1-aa82-919a5206efae\") " pod="openshift-multus/multus-tt68t"
Nov 02 02:47:40.844894 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.843344    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5c797b08-a314-44ed-8026-8a0ce1f6c36f-etc-openvswitch\") pod \"ovnkube-node-hww8q\" (UID: \"5c797b08-a314-44ed-8026-8a0ce1f6c36f\") " pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:40.844894 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.843348    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/94fb7403-b6ef-4d42-be18-7c45b5c10692-var-lib-kubelet\") pod \"tuned-7qpcf\" (UID: \"94fb7403-b6ef-4d42-be18-7c45b5c10692\") " pod="openshift-cluster-node-tuning-operator/tuned-7qpcf"
Nov 02 02:47:40.844894 ip-10-0-133-163 kubenswrapper[4496]: E1102 02:47:40.843373    4496 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 02 02:47:40.844894 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.843377    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c2f75412-4413-45d1-aa82-919a5206efae-host-var-lib-cni-bin\") pod \"multus-tt68t\" (UID: \"c2f75412-4413-45d1-aa82-919a5206efae\") " pod="openshift-multus/multus-tt68t"
Nov 02 02:47:40.844894 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.843379    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5c797b08-a314-44ed-8026-8a0ce1f6c36f-host-run-netns\") pod \"ovnkube-node-hww8q\" (UID: \"5c797b08-a314-44ed-8026-8a0ce1f6c36f\") " pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:40.844894 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.843387    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/94fb7403-b6ef-4d42-be18-7c45b5c10692-var-lib-kubelet\") pod \"tuned-7qpcf\" (UID: \"94fb7403-b6ef-4d42-be18-7c45b5c10692\") " pod="openshift-cluster-node-tuning-operator/tuned-7qpcf"
Nov 02 02:47:40.844894 ip-10-0-133-163 kubenswrapper[4496]: E1102 02:47:40.843405    4496 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/086e3504-d2fe-43ad-bc5e-dca409a4e695-metrics-certs podName:086e3504-d2fe-43ad-bc5e-dca409a4e695 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:41.343396352 +0000 UTC m=+1.922003710 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/086e3504-d2fe-43ad-bc5e-dca409a4e695-metrics-certs") pod "network-metrics-daemon-zkjqm" (UID: "086e3504-d2fe-43ad-bc5e-dca409a4e695") : secret "metrics-daemon-secret" not found
Nov 02 02:47:40.844894 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.843405    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5c797b08-a314-44ed-8026-8a0ce1f6c36f-host-run-netns\") pod \"ovnkube-node-hww8q\" (UID: \"5c797b08-a314-44ed-8026-8a0ce1f6c36f\") " pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:40.844894 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.843418    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5c797b08-a314-44ed-8026-8a0ce1f6c36f-systemd-units\") pod \"ovnkube-node-hww8q\" (UID: \"5c797b08-a314-44ed-8026-8a0ce1f6c36f\") " pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:40.844894 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.843433    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5c797b08-a314-44ed-8026-8a0ce1f6c36f-host-slash\") pod \"ovnkube-node-hww8q\" (UID: \"5c797b08-a314-44ed-8026-8a0ce1f6c36f\") " pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:40.844894 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.843449    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5c797b08-a314-44ed-8026-8a0ce1f6c36f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hww8q\" (UID: \"5c797b08-a314-44ed-8026-8a0ce1f6c36f\") " pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:40.844894 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.843488    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5c797b08-a314-44ed-8026-8a0ce1f6c36f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hww8q\" (UID: \"5c797b08-a314-44ed-8026-8a0ce1f6c36f\") " pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:40.844894 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.843494    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5c797b08-a314-44ed-8026-8a0ce1f6c36f-host-slash\") pod \"ovnkube-node-hww8q\" (UID: \"5c797b08-a314-44ed-8026-8a0ce1f6c36f\") " pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:40.844894 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.843500    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5c797b08-a314-44ed-8026-8a0ce1f6c36f-systemd-units\") pod \"ovnkube-node-hww8q\" (UID: \"5c797b08-a314-44ed-8026-8a0ce1f6c36f\") " pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:40.933610 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:40.933595    4496 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 02:47:41.058646 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:41.058572    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:41.348987 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:41.348923    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2d4b0e00-8bb2-4b4c-9805-af5c51c65405-metrics-tls\") pod \"dns-default-fbvxx\" (UID: \"2d4b0e00-8bb2-4b4c-9805-af5c51c65405\") " pod="openshift-dns/dns-default-fbvxx"
Nov 02 02:47:41.348987 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:41.348962    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/086e3504-d2fe-43ad-bc5e-dca409a4e695-metrics-certs\") pod \"network-metrics-daemon-zkjqm\" (UID: \"086e3504-d2fe-43ad-bc5e-dca409a4e695\") " pod="openshift-multus/network-metrics-daemon-zkjqm"
Nov 02 02:47:41.349158 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:41.348995    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/93e3306c-8c14-4bd4-ae04-08bc4b01a5fd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r4zs9\" (UID: \"93e3306c-8c14-4bd4-ae04-08bc4b01a5fd\") " pod="openshift-insights/insights-runtime-extractor-r4zs9"
Nov 02 02:47:41.349158 ip-10-0-133-163 kubenswrapper[4496]: E1102 02:47:41.349080    4496 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 02:47:41.349158 ip-10-0-133-163 kubenswrapper[4496]: E1102 02:47:41.349091    4496 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 02 02:47:41.349158 ip-10-0-133-163 kubenswrapper[4496]: E1102 02:47:41.349106    4496 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 02:47:41.349158 ip-10-0-133-163 kubenswrapper[4496]: E1102 02:47:41.349137    4496 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d4b0e00-8bb2-4b4c-9805-af5c51c65405-metrics-tls podName:2d4b0e00-8bb2-4b4c-9805-af5c51c65405 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:42.349123556 +0000 UTC m=+2.927730914 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2d4b0e00-8bb2-4b4c-9805-af5c51c65405-metrics-tls") pod "dns-default-fbvxx" (UID: "2d4b0e00-8bb2-4b4c-9805-af5c51c65405") : secret "dns-default-metrics-tls" not found
Nov 02 02:47:41.349158 ip-10-0-133-163 kubenswrapper[4496]: E1102 02:47:41.349150    4496 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/086e3504-d2fe-43ad-bc5e-dca409a4e695-metrics-certs podName:086e3504-d2fe-43ad-bc5e-dca409a4e695 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:42.349144474 +0000 UTC m=+2.927751831 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/086e3504-d2fe-43ad-bc5e-dca409a4e695-metrics-certs") pod "network-metrics-daemon-zkjqm" (UID: "086e3504-d2fe-43ad-bc5e-dca409a4e695") : secret "metrics-daemon-secret" not found
Nov 02 02:47:41.349361 ip-10-0-133-163 kubenswrapper[4496]: E1102 02:47:41.349167    4496 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93e3306c-8c14-4bd4-ae04-08bc4b01a5fd-insights-runtime-extractor-tls podName:93e3306c-8c14-4bd4-ae04-08bc4b01a5fd nodeName:}" failed. No retries permitted until 2026-11-02 02:47:42.349154503 +0000 UTC m=+2.927761864 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/93e3306c-8c14-4bd4-ae04-08bc4b01a5fd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-r4zs9" (UID: "93e3306c-8c14-4bd4-ae04-08bc4b01a5fd") : secret "insights-runtime-extractor-tls" not found
Nov 02 02:47:41.606633 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:41.606553    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-t9g7w-hqs6p" event={"ID":"95effc61-7dc3-42c1-9609-be607c3bc65f","Type":"ContainerStarted","Data":"684b7419e6f30d5e3c07e887c1abdf27a11c764d68a8a6944a8901e7659ff795"}
Nov 02 02:47:42.357646 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:42.357606    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2d4b0e00-8bb2-4b4c-9805-af5c51c65405-metrics-tls\") pod \"dns-default-fbvxx\" (UID: \"2d4b0e00-8bb2-4b4c-9805-af5c51c65405\") " pod="openshift-dns/dns-default-fbvxx"
Nov 02 02:47:42.357825 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:42.357660    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/086e3504-d2fe-43ad-bc5e-dca409a4e695-metrics-certs\") pod \"network-metrics-daemon-zkjqm\" (UID: \"086e3504-d2fe-43ad-bc5e-dca409a4e695\") " pod="openshift-multus/network-metrics-daemon-zkjqm"
Nov 02 02:47:42.357825 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:42.357696    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/93e3306c-8c14-4bd4-ae04-08bc4b01a5fd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r4zs9\" (UID: \"93e3306c-8c14-4bd4-ae04-08bc4b01a5fd\") " pod="openshift-insights/insights-runtime-extractor-r4zs9"
Nov 02 02:47:42.357825 ip-10-0-133-163 kubenswrapper[4496]: E1102 02:47:42.357762    4496 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 02:47:42.357825 ip-10-0-133-163 kubenswrapper[4496]: E1102 02:47:42.357801    4496 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 02 02:47:42.358047 ip-10-0-133-163 kubenswrapper[4496]: E1102 02:47:42.357853    4496 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d4b0e00-8bb2-4b4c-9805-af5c51c65405-metrics-tls podName:2d4b0e00-8bb2-4b4c-9805-af5c51c65405 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:44.357830957 +0000 UTC m=+4.936438322 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2d4b0e00-8bb2-4b4c-9805-af5c51c65405-metrics-tls") pod "dns-default-fbvxx" (UID: "2d4b0e00-8bb2-4b4c-9805-af5c51c65405") : secret "dns-default-metrics-tls" not found
Nov 02 02:47:42.358047 ip-10-0-133-163 kubenswrapper[4496]: E1102 02:47:42.357874    4496 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 02:47:42.358047 ip-10-0-133-163 kubenswrapper[4496]: E1102 02:47:42.357968    4496 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/086e3504-d2fe-43ad-bc5e-dca409a4e695-metrics-certs podName:086e3504-d2fe-43ad-bc5e-dca409a4e695 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:44.357935237 +0000 UTC m=+4.936542613 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/086e3504-d2fe-43ad-bc5e-dca409a4e695-metrics-certs") pod "network-metrics-daemon-zkjqm" (UID: "086e3504-d2fe-43ad-bc5e-dca409a4e695") : secret "metrics-daemon-secret" not found
Nov 02 02:47:42.358047 ip-10-0-133-163 kubenswrapper[4496]: E1102 02:47:42.357996    4496 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93e3306c-8c14-4bd4-ae04-08bc4b01a5fd-insights-runtime-extractor-tls podName:93e3306c-8c14-4bd4-ae04-08bc4b01a5fd nodeName:}" failed. No retries permitted until 2026-11-02 02:47:44.35798725 +0000 UTC m=+4.936594615 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/93e3306c-8c14-4bd4-ae04-08bc4b01a5fd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-r4zs9" (UID: "93e3306c-8c14-4bd4-ae04-08bc4b01a5fd") : secret "insights-runtime-extractor-tls" not found
Nov 02 02:47:42.647075 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:42.646995    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:42.647450 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:42.647189    4496 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 02:47:43.558072 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:43.558041    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-z8sq6"
Nov 02 02:47:43.728450 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:43.728428    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-47f7x"
Nov 02 02:47:44.373354 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:44.373314    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2d4b0e00-8bb2-4b4c-9805-af5c51c65405-metrics-tls\") pod \"dns-default-fbvxx\" (UID: \"2d4b0e00-8bb2-4b4c-9805-af5c51c65405\") " pod="openshift-dns/dns-default-fbvxx"
Nov 02 02:47:44.373592 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:44.373367    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/086e3504-d2fe-43ad-bc5e-dca409a4e695-metrics-certs\") pod \"network-metrics-daemon-zkjqm\" (UID: \"086e3504-d2fe-43ad-bc5e-dca409a4e695\") " pod="openshift-multus/network-metrics-daemon-zkjqm"
Nov 02 02:47:44.373592 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:44.373400    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/93e3306c-8c14-4bd4-ae04-08bc4b01a5fd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r4zs9\" (UID: \"93e3306c-8c14-4bd4-ae04-08bc4b01a5fd\") " pod="openshift-insights/insights-runtime-extractor-r4zs9"
Nov 02 02:47:44.373592 ip-10-0-133-163 kubenswrapper[4496]: E1102 02:47:44.373460    4496 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 02:47:44.373592 ip-10-0-133-163 kubenswrapper[4496]: E1102 02:47:44.373516    4496 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 02 02:47:44.373592 ip-10-0-133-163 kubenswrapper[4496]: E1102 02:47:44.373547    4496 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d4b0e00-8bb2-4b4c-9805-af5c51c65405-metrics-tls podName:2d4b0e00-8bb2-4b4c-9805-af5c51c65405 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:48.373529966 +0000 UTC m=+8.952137326 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2d4b0e00-8bb2-4b4c-9805-af5c51c65405-metrics-tls") pod "dns-default-fbvxx" (UID: "2d4b0e00-8bb2-4b4c-9805-af5c51c65405") : secret "dns-default-metrics-tls" not found
Nov 02 02:47:44.373592 ip-10-0-133-163 kubenswrapper[4496]: E1102 02:47:44.373567    4496 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/086e3504-d2fe-43ad-bc5e-dca409a4e695-metrics-certs podName:086e3504-d2fe-43ad-bc5e-dca409a4e695 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:48.373554972 +0000 UTC m=+8.952162330 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/086e3504-d2fe-43ad-bc5e-dca409a4e695-metrics-certs") pod "network-metrics-daemon-zkjqm" (UID: "086e3504-d2fe-43ad-bc5e-dca409a4e695") : secret "metrics-daemon-secret" not found
Nov 02 02:47:44.373592 ip-10-0-133-163 kubenswrapper[4496]: E1102 02:47:44.373516    4496 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 02:47:44.373592 ip-10-0-133-163 kubenswrapper[4496]: E1102 02:47:44.373596    4496 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93e3306c-8c14-4bd4-ae04-08bc4b01a5fd-insights-runtime-extractor-tls podName:93e3306c-8c14-4bd4-ae04-08bc4b01a5fd nodeName:}" failed. No retries permitted until 2026-11-02 02:47:48.373587706 +0000 UTC m=+8.952195064 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/93e3306c-8c14-4bd4-ae04-08bc4b01a5fd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-r4zs9" (UID: "93e3306c-8c14-4bd4-ae04-08bc4b01a5fd") : secret "insights-runtime-extractor-tls" not found
Nov 02 02:47:44.615953 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:44.615912    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-rspt2" event={"ID":"39d09ab7-5724-44d4-a731-92eabb9280a6","Type":"ContainerStarted","Data":"b6563d20b830bdc11abaf1cac4c465b70b8d5bdc77ee501e516e7df3bb5d8751"}
Nov 02 02:47:44.907134 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:44.907106    4496 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-sztr6_4fec139d-da70-4cf6-8730-b578df3bebcc/dns-node-resolver/0.log"
Nov 02 02:47:45.401318 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:45.401286    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hww8q"
Nov 02 02:47:45.907536 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:45.907511    4496 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-kkqzw_e701e783-5b43-45d7-b88b-83fd3c4af13b/node-ca/0.log"
Nov 02 02:47:46.905540 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:46.905512    4496 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-zt7hz_797676e6-5545-41ad-a4d1-c15f26db122a/serve-healthcheck-canary/0.log"
Nov 02 02:47:47.305795 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:47.305763    4496 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-sp26d_d66bfee1-59f7-4ed9-90e1-31582a4c4784/migrator/0.log"
Nov 02 02:47:48.401730 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:48.401691    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/086e3504-d2fe-43ad-bc5e-dca409a4e695-metrics-certs\") pod \"network-metrics-daemon-zkjqm\" (UID: \"086e3504-d2fe-43ad-bc5e-dca409a4e695\") " pod="openshift-multus/network-metrics-daemon-zkjqm"
Nov 02 02:47:48.401730 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:48.401735    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/93e3306c-8c14-4bd4-ae04-08bc4b01a5fd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r4zs9\" (UID: \"93e3306c-8c14-4bd4-ae04-08bc4b01a5fd\") " pod="openshift-insights/insights-runtime-extractor-r4zs9"
Nov 02 02:47:48.402261 ip-10-0-133-163 kubenswrapper[4496]: E1102 02:47:48.401844    4496 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 02:47:48.402261 ip-10-0-133-163 kubenswrapper[4496]: E1102 02:47:48.401892    4496 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93e3306c-8c14-4bd4-ae04-08bc4b01a5fd-insights-runtime-extractor-tls podName:93e3306c-8c14-4bd4-ae04-08bc4b01a5fd nodeName:}" failed. No retries permitted until 2026-11-02 02:47:56.401879588 +0000 UTC m=+16.980486946 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/93e3306c-8c14-4bd4-ae04-08bc4b01a5fd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-r4zs9" (UID: "93e3306c-8c14-4bd4-ae04-08bc4b01a5fd") : secret "insights-runtime-extractor-tls" not found
Nov 02 02:47:48.402261 ip-10-0-133-163 kubenswrapper[4496]: E1102 02:47:48.401885    4496 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 02 02:47:48.402261 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:48.401938    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2d4b0e00-8bb2-4b4c-9805-af5c51c65405-metrics-tls\") pod \"dns-default-fbvxx\" (UID: \"2d4b0e00-8bb2-4b4c-9805-af5c51c65405\") " pod="openshift-dns/dns-default-fbvxx"
Nov 02 02:47:48.402261 ip-10-0-133-163 kubenswrapper[4496]: E1102 02:47:48.401958    4496 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/086e3504-d2fe-43ad-bc5e-dca409a4e695-metrics-certs podName:086e3504-d2fe-43ad-bc5e-dca409a4e695 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:56.401940869 +0000 UTC m=+16.980548233 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/086e3504-d2fe-43ad-bc5e-dca409a4e695-metrics-certs") pod "network-metrics-daemon-zkjqm" (UID: "086e3504-d2fe-43ad-bc5e-dca409a4e695") : secret "metrics-daemon-secret" not found
Nov 02 02:47:48.402261 ip-10-0-133-163 kubenswrapper[4496]: E1102 02:47:48.402033    4496 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 02:47:48.402261 ip-10-0-133-163 kubenswrapper[4496]: E1102 02:47:48.402082    4496 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d4b0e00-8bb2-4b4c-9805-af5c51c65405-metrics-tls podName:2d4b0e00-8bb2-4b4c-9805-af5c51c65405 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:56.402072661 +0000 UTC m=+16.980680040 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2d4b0e00-8bb2-4b4c-9805-af5c51c65405-metrics-tls") pod "dns-default-fbvxx" (UID: "2d4b0e00-8bb2-4b4c-9805-af5c51c65405") : secret "dns-default-metrics-tls" not found
Nov 02 02:47:55.613088 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:55.613055    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-zrssn"]
Nov 02 02:47:55.614754 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:55.614738    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-zrssn"
Nov 02 02:47:55.656485 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:55.656461    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52wzs\" (UniqueName: \"kubernetes.io/projected/87db08be-0446-48e1-8012-623b96e7c646-kube-api-access-52wzs\") pod \"downloads-6dd789c8b7-zrssn\" (UID: \"87db08be-0446-48e1-8012-623b96e7c646\") " pod="openshift-console/downloads-6dd789c8b7-zrssn"
Nov 02 02:47:55.656607 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:55.656495    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/87db08be-0446-48e1-8012-623b96e7c646-tmp\") pod \"downloads-6dd789c8b7-zrssn\" (UID: \"87db08be-0446-48e1-8012-623b96e7c646\") " pod="openshift-console/downloads-6dd789c8b7-zrssn"
Nov 02 02:47:55.662395 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:55.662370    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-zrssn"]
Nov 02 02:47:55.670604 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:55.670579    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Nov 02 02:47:55.670700 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:55.670613    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Nov 02 02:47:55.688705 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:55.688686    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-kp24t\""
Nov 02 02:47:55.757517 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:55.757488    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52wzs\" (UniqueName: \"kubernetes.io/projected/87db08be-0446-48e1-8012-623b96e7c646-kube-api-access-52wzs\") pod \"downloads-6dd789c8b7-zrssn\" (UID: \"87db08be-0446-48e1-8012-623b96e7c646\") " pod="openshift-console/downloads-6dd789c8b7-zrssn"
Nov 02 02:47:55.757653 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:55.757535    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/87db08be-0446-48e1-8012-623b96e7c646-tmp\") pod \"downloads-6dd789c8b7-zrssn\" (UID: \"87db08be-0446-48e1-8012-623b96e7c646\") " pod="openshift-console/downloads-6dd789c8b7-zrssn"
Nov 02 02:47:55.757890 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:55.757874    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/87db08be-0446-48e1-8012-623b96e7c646-tmp\") pod \"downloads-6dd789c8b7-zrssn\" (UID: \"87db08be-0446-48e1-8012-623b96e7c646\") " pod="openshift-console/downloads-6dd789c8b7-zrssn"
Nov 02 02:47:55.814709 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:55.814682    4496 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 02 02:47:55.816805 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:55.816780    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52wzs\" (UniqueName: \"kubernetes.io/projected/87db08be-0446-48e1-8012-623b96e7c646-kube-api-access-52wzs\") pod \"downloads-6dd789c8b7-zrssn\" (UID: \"87db08be-0446-48e1-8012-623b96e7c646\") " pod="openshift-console/downloads-6dd789c8b7-zrssn"
Nov 02 02:47:55.924266 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:55.924204    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-zrssn"
Nov 02 02:47:56.338822 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:56.338791    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-zrssn"]
Nov 02 02:47:56.341906 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:56.341872    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87db08be_0446_48e1_8012_623b96e7c646.slice/crio-54d14796a5e70190951eae1f15e029723f0f32d63a46122cdafbcd2d4bd920ec WatchSource:0}: Error finding container 54d14796a5e70190951eae1f15e029723f0f32d63a46122cdafbcd2d4bd920ec: Status 404 returned error can't find the container with id 54d14796a5e70190951eae1f15e029723f0f32d63a46122cdafbcd2d4bd920ec
Nov 02 02:47:56.463096 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:56.463068    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2d4b0e00-8bb2-4b4c-9805-af5c51c65405-metrics-tls\") pod \"dns-default-fbvxx\" (UID: \"2d4b0e00-8bb2-4b4c-9805-af5c51c65405\") " pod="openshift-dns/dns-default-fbvxx"
Nov 02 02:47:56.463209 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:56.463117    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/086e3504-d2fe-43ad-bc5e-dca409a4e695-metrics-certs\") pod \"network-metrics-daemon-zkjqm\" (UID: \"086e3504-d2fe-43ad-bc5e-dca409a4e695\") " pod="openshift-multus/network-metrics-daemon-zkjqm"
Nov 02 02:47:56.463209 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:56.463141    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/93e3306c-8c14-4bd4-ae04-08bc4b01a5fd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r4zs9\" (UID: \"93e3306c-8c14-4bd4-ae04-08bc4b01a5fd\") " pod="openshift-insights/insights-runtime-extractor-r4zs9"
Nov 02 02:47:56.465482 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:56.465454    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/086e3504-d2fe-43ad-bc5e-dca409a4e695-metrics-certs\") pod \"network-metrics-daemon-zkjqm\" (UID: \"086e3504-d2fe-43ad-bc5e-dca409a4e695\") " pod="openshift-multus/network-metrics-daemon-zkjqm"
Nov 02 02:47:56.465602 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:56.465499    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/93e3306c-8c14-4bd4-ae04-08bc4b01a5fd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r4zs9\" (UID: \"93e3306c-8c14-4bd4-ae04-08bc4b01a5fd\") " pod="openshift-insights/insights-runtime-extractor-r4zs9"
Nov 02 02:47:56.465602 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:56.465499    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2d4b0e00-8bb2-4b4c-9805-af5c51c65405-metrics-tls\") pod \"dns-default-fbvxx\" (UID: \"2d4b0e00-8bb2-4b4c-9805-af5c51c65405\") " pod="openshift-dns/dns-default-fbvxx"
Nov 02 02:47:56.529253 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:56.529228    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zkjqm"
Nov 02 02:47:56.531304 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:56.531283    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fbvxx"
Nov 02 02:47:56.532579 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:56.532547    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-r4zs9"
Nov 02 02:47:56.646749 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:56.646721    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-zrssn" event={"ID":"87db08be-0446-48e1-8012-623b96e7c646","Type":"ContainerStarted","Data":"54d14796a5e70190951eae1f15e029723f0f32d63a46122cdafbcd2d4bd920ec"}
Nov 02 02:47:56.907808 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:56.907738    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-r4zs9"]
Nov 02 02:47:56.948814 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:56.948787    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93e3306c_8c14_4bd4_ae04_08bc4b01a5fd.slice/crio-6254dce2e0918be97990dc73456aad6ac18ddfa5a018f12859ee62769b43643c WatchSource:0}: Error finding container 6254dce2e0918be97990dc73456aad6ac18ddfa5a018f12859ee62769b43643c: Status 404 returned error can't find the container with id 6254dce2e0918be97990dc73456aad6ac18ddfa5a018f12859ee62769b43643c
Nov 02 02:47:56.956631 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:56.956603    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zkjqm"]
Nov 02 02:47:56.956860 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:56.956841    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-fbvxx"]
Nov 02 02:47:56.960631 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:56.960605    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod086e3504_d2fe_43ad_bc5e_dca409a4e695.slice/crio-c7a9f068e49597c6a5276b2b796a11a613d76a1984126bb1fae5f975e78e3ae1 WatchSource:0}: Error finding container c7a9f068e49597c6a5276b2b796a11a613d76a1984126bb1fae5f975e78e3ae1: Status 404 returned error can't find the container with id c7a9f068e49597c6a5276b2b796a11a613d76a1984126bb1fae5f975e78e3ae1
Nov 02 02:47:56.961387 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:47:56.961365    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d4b0e00_8bb2_4b4c_9805_af5c51c65405.slice/crio-522e841578c1cb0388e780d0ca94565a6ea1d224252650868780d02389d3bad0 WatchSource:0}: Error finding container 522e841578c1cb0388e780d0ca94565a6ea1d224252650868780d02389d3bad0: Status 404 returned error can't find the container with id 522e841578c1cb0388e780d0ca94565a6ea1d224252650868780d02389d3bad0
Nov 02 02:47:57.652256 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:57.652214    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fbvxx" event={"ID":"2d4b0e00-8bb2-4b4c-9805-af5c51c65405","Type":"ContainerStarted","Data":"522e841578c1cb0388e780d0ca94565a6ea1d224252650868780d02389d3bad0"}
Nov 02 02:47:57.653672 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:57.653641    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-r4zs9" event={"ID":"93e3306c-8c14-4bd4-ae04-08bc4b01a5fd","Type":"ContainerStarted","Data":"eb54a134090ac4e59267f1a6493d1f5b94bd4e3071ecc5f37b52103fb2ba9a91"}
Nov 02 02:47:57.653797 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:57.653693    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-r4zs9" event={"ID":"93e3306c-8c14-4bd4-ae04-08bc4b01a5fd","Type":"ContainerStarted","Data":"6254dce2e0918be97990dc73456aad6ac18ddfa5a018f12859ee62769b43643c"}
Nov 02 02:47:57.654855 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:57.654830    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zkjqm" event={"ID":"086e3504-d2fe-43ad-bc5e-dca409a4e695","Type":"ContainerStarted","Data":"c7a9f068e49597c6a5276b2b796a11a613d76a1984126bb1fae5f975e78e3ae1"}
Nov 02 02:47:58.664047 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:58.663991    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-r4zs9" event={"ID":"93e3306c-8c14-4bd4-ae04-08bc4b01a5fd","Type":"ContainerStarted","Data":"185671856a951cc234fc24a66197c66ca6b9ad71e9089d9881681b6476a2e369"}
Nov 02 02:47:59.572618 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:59.571615    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-88c54d969-g7zvl"]
Nov 02 02:47:59.573703 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:59.573679    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-88c54d969-g7zvl"
Nov 02 02:47:59.628677 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:59.628646    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-88c54d969-g7zvl"]
Nov 02 02:47:59.660282 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:59.660251    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Nov 02 02:47:59.660452 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:59.660327    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Nov 02 02:47:59.660452 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:59.660381    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Nov 02 02:47:59.660452 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:59.660438    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Nov 02 02:47:59.671032 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:59.670983    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zkjqm" event={"ID":"086e3504-d2fe-43ad-bc5e-dca409a4e695","Type":"ContainerStarted","Data":"897f8f877a6d2bc13752884a035e0c3af03c822caa08967228b80f61e950f4a5"}
Nov 02 02:47:59.671429 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:59.671080    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zkjqm" event={"ID":"086e3504-d2fe-43ad-bc5e-dca409a4e695","Type":"ContainerStarted","Data":"46e2abd7046b4ff17a5487390608314dedc53b08c5e2efb359481eed51fcefc7"}
Nov 02 02:47:59.673703 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:59.673249    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fbvxx" event={"ID":"2d4b0e00-8bb2-4b4c-9805-af5c51c65405","Type":"ContainerStarted","Data":"4233a053aeaff4694faf9da9947f30f19bcdd0b1c67a58fa2a0a9a91994d123c"}
Nov 02 02:47:59.673703 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:59.673298    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fbvxx" event={"ID":"2d4b0e00-8bb2-4b4c-9805-af5c51c65405","Type":"ContainerStarted","Data":"ccef85f5940bd4e7cf2fa6e5928a70e1ff8ef469fd8eb242ae670fa594a41836"}
Nov 02 02:47:59.673703 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:59.673328    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-fbvxx"
Nov 02 02:47:59.679010 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:59.678990    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Nov 02 02:47:59.679276 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:59.679249    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-rs9jf\""
Nov 02 02:47:59.688499 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:59.688467    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/66d05c06-04c1-4ecf-8003-4bd1b75c6c5f-console-serving-cert\") pod \"console-88c54d969-g7zvl\" (UID: \"66d05c06-04c1-4ecf-8003-4bd1b75c6c5f\") " pod="openshift-console/console-88c54d969-g7zvl"
Nov 02 02:47:59.688604 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:59.688510    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/66d05c06-04c1-4ecf-8003-4bd1b75c6c5f-console-config\") pod \"console-88c54d969-g7zvl\" (UID: \"66d05c06-04c1-4ecf-8003-4bd1b75c6c5f\") " pod="openshift-console/console-88c54d969-g7zvl"
Nov 02 02:47:59.688604 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:59.688540    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/66d05c06-04c1-4ecf-8003-4bd1b75c6c5f-oauth-serving-cert\") pod \"console-88c54d969-g7zvl\" (UID: \"66d05c06-04c1-4ecf-8003-4bd1b75c6c5f\") " pod="openshift-console/console-88c54d969-g7zvl"
Nov 02 02:47:59.688604 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:59.688573    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/66d05c06-04c1-4ecf-8003-4bd1b75c6c5f-console-oauth-config\") pod \"console-88c54d969-g7zvl\" (UID: \"66d05c06-04c1-4ecf-8003-4bd1b75c6c5f\") " pod="openshift-console/console-88c54d969-g7zvl"
Nov 02 02:47:59.688604 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:59.688591    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/66d05c06-04c1-4ecf-8003-4bd1b75c6c5f-service-ca\") pod \"console-88c54d969-g7zvl\" (UID: \"66d05c06-04c1-4ecf-8003-4bd1b75c6c5f\") " pod="openshift-console/console-88c54d969-g7zvl"
Nov 02 02:47:59.688604 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:59.688604    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qx2s\" (UniqueName: \"kubernetes.io/projected/66d05c06-04c1-4ecf-8003-4bd1b75c6c5f-kube-api-access-6qx2s\") pod \"console-88c54d969-g7zvl\" (UID: \"66d05c06-04c1-4ecf-8003-4bd1b75c6c5f\") " pod="openshift-console/console-88c54d969-g7zvl"
Nov 02 02:47:59.688831 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:59.688647    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/66d05c06-04c1-4ecf-8003-4bd1b75c6c5f-tmp\") pod \"console-88c54d969-g7zvl\" (UID: \"66d05c06-04c1-4ecf-8003-4bd1b75c6c5f\") " pod="openshift-console/console-88c54d969-g7zvl"
Nov 02 02:47:59.789764 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:59.789729    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/66d05c06-04c1-4ecf-8003-4bd1b75c6c5f-tmp\") pod \"console-88c54d969-g7zvl\" (UID: \"66d05c06-04c1-4ecf-8003-4bd1b75c6c5f\") " pod="openshift-console/console-88c54d969-g7zvl"
Nov 02 02:47:59.789933 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:59.789786    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/66d05c06-04c1-4ecf-8003-4bd1b75c6c5f-console-serving-cert\") pod \"console-88c54d969-g7zvl\" (UID: \"66d05c06-04c1-4ecf-8003-4bd1b75c6c5f\") " pod="openshift-console/console-88c54d969-g7zvl"
Nov 02 02:47:59.789933 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:59.789851    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/66d05c06-04c1-4ecf-8003-4bd1b75c6c5f-console-config\") pod \"console-88c54d969-g7zvl\" (UID: \"66d05c06-04c1-4ecf-8003-4bd1b75c6c5f\") " pod="openshift-console/console-88c54d969-g7zvl"
Nov 02 02:47:59.789933 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:59.789903    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/66d05c06-04c1-4ecf-8003-4bd1b75c6c5f-oauth-serving-cert\") pod \"console-88c54d969-g7zvl\" (UID: \"66d05c06-04c1-4ecf-8003-4bd1b75c6c5f\") " pod="openshift-console/console-88c54d969-g7zvl"
Nov 02 02:47:59.790140 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:59.789980    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/66d05c06-04c1-4ecf-8003-4bd1b75c6c5f-console-oauth-config\") pod \"console-88c54d969-g7zvl\" (UID: \"66d05c06-04c1-4ecf-8003-4bd1b75c6c5f\") " pod="openshift-console/console-88c54d969-g7zvl"
Nov 02 02:47:59.790140 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:59.789998    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/66d05c06-04c1-4ecf-8003-4bd1b75c6c5f-service-ca\") pod \"console-88c54d969-g7zvl\" (UID: \"66d05c06-04c1-4ecf-8003-4bd1b75c6c5f\") " pod="openshift-console/console-88c54d969-g7zvl"
Nov 02 02:47:59.790140 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:59.790042    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qx2s\" (UniqueName: \"kubernetes.io/projected/66d05c06-04c1-4ecf-8003-4bd1b75c6c5f-kube-api-access-6qx2s\") pod \"console-88c54d969-g7zvl\" (UID: \"66d05c06-04c1-4ecf-8003-4bd1b75c6c5f\") " pod="openshift-console/console-88c54d969-g7zvl"
Nov 02 02:47:59.790286 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:59.790198    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/66d05c06-04c1-4ecf-8003-4bd1b75c6c5f-tmp\") pod \"console-88c54d969-g7zvl\" (UID: \"66d05c06-04c1-4ecf-8003-4bd1b75c6c5f\") " pod="openshift-console/console-88c54d969-g7zvl"
Nov 02 02:47:59.791038 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:59.790996    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/66d05c06-04c1-4ecf-8003-4bd1b75c6c5f-oauth-serving-cert\") pod \"console-88c54d969-g7zvl\" (UID: \"66d05c06-04c1-4ecf-8003-4bd1b75c6c5f\") " pod="openshift-console/console-88c54d969-g7zvl"
Nov 02 02:47:59.791222 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:59.791204    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/66d05c06-04c1-4ecf-8003-4bd1b75c6c5f-service-ca\") pod \"console-88c54d969-g7zvl\" (UID: \"66d05c06-04c1-4ecf-8003-4bd1b75c6c5f\") " pod="openshift-console/console-88c54d969-g7zvl"
Nov 02 02:47:59.791562 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:59.791541    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/66d05c06-04c1-4ecf-8003-4bd1b75c6c5f-console-config\") pod \"console-88c54d969-g7zvl\" (UID: \"66d05c06-04c1-4ecf-8003-4bd1b75c6c5f\") " pod="openshift-console/console-88c54d969-g7zvl"
Nov 02 02:47:59.793870 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:59.793532    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/66d05c06-04c1-4ecf-8003-4bd1b75c6c5f-console-serving-cert\") pod \"console-88c54d969-g7zvl\" (UID: \"66d05c06-04c1-4ecf-8003-4bd1b75c6c5f\") " pod="openshift-console/console-88c54d969-g7zvl"
Nov 02 02:47:59.796444 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:59.796420    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/66d05c06-04c1-4ecf-8003-4bd1b75c6c5f-console-oauth-config\") pod \"console-88c54d969-g7zvl\" (UID: \"66d05c06-04c1-4ecf-8003-4bd1b75c6c5f\") " pod="openshift-console/console-88c54d969-g7zvl"
Nov 02 02:47:59.848490 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:59.848423    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qx2s\" (UniqueName: \"kubernetes.io/projected/66d05c06-04c1-4ecf-8003-4bd1b75c6c5f-kube-api-access-6qx2s\") pod \"console-88c54d969-g7zvl\" (UID: \"66d05c06-04c1-4ecf-8003-4bd1b75c6c5f\") " pod="openshift-console/console-88c54d969-g7zvl"
Nov 02 02:47:59.887205 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:47:59.887171    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-88c54d969-g7zvl"
Nov 02 02:48:00.266067 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:00.265911    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-88c54d969-g7zvl"]
Nov 02 02:48:00.271051 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:48:00.270995    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66d05c06_04c1_4ecf_8003_4bd1b75c6c5f.slice/crio-b1bd8c8d6d3e355458f62c9aa8694abb7271563a6bdf1629fe4c4bc55e99aca7 WatchSource:0}: Error finding container b1bd8c8d6d3e355458f62c9aa8694abb7271563a6bdf1629fe4c4bc55e99aca7: Status 404 returned error can't find the container with id b1bd8c8d6d3e355458f62c9aa8694abb7271563a6bdf1629fe4c4bc55e99aca7
Nov 02 02:48:00.679597 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:00.679512    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-r4zs9" event={"ID":"93e3306c-8c14-4bd4-ae04-08bc4b01a5fd","Type":"ContainerStarted","Data":"7b039020144dd8b290473b6dbe61fa8afec156b3f2daeb22ef34c8e822c2e973"}
Nov 02 02:48:00.680786 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:00.680762    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-88c54d969-g7zvl" event={"ID":"66d05c06-04c1-4ecf-8003-4bd1b75c6c5f","Type":"ContainerStarted","Data":"b1bd8c8d6d3e355458f62c9aa8694abb7271563a6bdf1629fe4c4bc55e99aca7"}
Nov 02 02:48:01.013859 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:01.013828    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-cv2tt"]
Nov 02 02:48:01.016301 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:01.016273    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-cv2tt"
Nov 02 02:48:01.072204 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:01.072165    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Nov 02 02:48:01.072360 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:01.072234    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-tg9tg\""
Nov 02 02:48:01.076648 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:01.076618    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-cv2tt"]
Nov 02 02:48:01.100689 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:01.100655    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/56e6d06c-317d-43db-90a1-5c758a6d5d24-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-cv2tt\" (UID: \"56e6d06c-317d-43db-90a1-5c758a6d5d24\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-cv2tt"
Nov 02 02:48:01.201330 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:01.201287    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/56e6d06c-317d-43db-90a1-5c758a6d5d24-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-cv2tt\" (UID: \"56e6d06c-317d-43db-90a1-5c758a6d5d24\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-cv2tt"
Nov 02 02:48:01.204099 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:01.204068    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/56e6d06c-317d-43db-90a1-5c758a6d5d24-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-cv2tt\" (UID: \"56e6d06c-317d-43db-90a1-5c758a6d5d24\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-cv2tt"
Nov 02 02:48:01.329464 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:01.329378    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-cv2tt"
Nov 02 02:48:01.710331 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:01.710215    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-cv2tt"]
Nov 02 02:48:02.690536 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:02.690495    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-cv2tt" event={"ID":"56e6d06c-317d-43db-90a1-5c758a6d5d24","Type":"ContainerStarted","Data":"a753489c0d1e7c9900478b42babf602414d54cda8197b0a1b92b22f800b05571"}
Nov 02 02:48:03.533734 ip-10-0-133-163 kubenswrapper[4496]: E1102 02:48:03.533693    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 02:48:03.696176 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:03.696095    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-cv2tt" event={"ID":"56e6d06c-317d-43db-90a1-5c758a6d5d24","Type":"ContainerStarted","Data":"2fb4766fe158b9694dd40bf42242f1a43a1012ba1fad088f65e58a23f49055ea"}
Nov 02 02:48:03.696329 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:03.696255    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-cv2tt"
Nov 02 02:48:03.702404 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:03.702381    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-cv2tt"
Nov 02 02:48:03.814372 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:03.814323    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-cv2tt" podStartSLOduration=2.178712827 podStartE2EDuration="3.814309715s" totalImagesPullingTime="1.635596888s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:48:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:48:03.81400861 +0000 UTC m=+24.392615990" watchObservedRunningTime="2026-11-02 02:48:03.814309715 +0000 UTC m=+24.392917092"
Nov 02 02:48:04.618701 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:04.618665    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-bk94k"]
Nov 02 02:48:04.620931 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:04.620898    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-bk94k"
Nov 02 02:48:04.701586 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:04.701548    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-88c54d969-g7zvl" event={"ID":"66d05c06-04c1-4ecf-8003-4bd1b75c6c5f","Type":"ContainerStarted","Data":"41eb0fac565a14deb2282f290abaffcd420cd3704ea0e62bb330c2a2332fa7e5"}
Nov 02 02:48:04.725594 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:04.725526    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Nov 02 02:48:04.725769 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:04.725750    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Nov 02 02:48:04.725831 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:04.725751    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 02 02:48:04.725831 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:04.725750    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 02 02:48:04.725831 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:04.725823    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 02 02:48:04.725971 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:04.725859    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-nzmzw\""
Nov 02 02:48:04.727086 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:04.727061    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/51e4a664-eed1-49a4-bfb8-49112a42b99c-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-bk94k\" (UID: \"51e4a664-eed1-49a4-bfb8-49112a42b99c\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-bk94k"
Nov 02 02:48:04.727201 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:04.727104    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/51e4a664-eed1-49a4-bfb8-49112a42b99c-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-bk94k\" (UID: \"51e4a664-eed1-49a4-bfb8-49112a42b99c\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-bk94k"
Nov 02 02:48:04.727201 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:04.727132    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnk2h\" (UniqueName: \"kubernetes.io/projected/51e4a664-eed1-49a4-bfb8-49112a42b99c-kube-api-access-gnk2h\") pod \"prometheus-operator-9cd89b74c-bk94k\" (UID: \"51e4a664-eed1-49a4-bfb8-49112a42b99c\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-bk94k"
Nov 02 02:48:04.727316 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:04.727260    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/51e4a664-eed1-49a4-bfb8-49112a42b99c-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-bk94k\" (UID: \"51e4a664-eed1-49a4-bfb8-49112a42b99c\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-bk94k"
Nov 02 02:48:04.777342 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:04.777311    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-bk94k"]
Nov 02 02:48:04.827797 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:04.827768    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/51e4a664-eed1-49a4-bfb8-49112a42b99c-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-bk94k\" (UID: \"51e4a664-eed1-49a4-bfb8-49112a42b99c\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-bk94k"
Nov 02 02:48:04.827952 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:04.827825    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/51e4a664-eed1-49a4-bfb8-49112a42b99c-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-bk94k\" (UID: \"51e4a664-eed1-49a4-bfb8-49112a42b99c\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-bk94k"
Nov 02 02:48:04.827952 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:04.827913    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/51e4a664-eed1-49a4-bfb8-49112a42b99c-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-bk94k\" (UID: \"51e4a664-eed1-49a4-bfb8-49112a42b99c\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-bk94k"
Nov 02 02:48:04.827952 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:04.827943    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnk2h\" (UniqueName: \"kubernetes.io/projected/51e4a664-eed1-49a4-bfb8-49112a42b99c-kube-api-access-gnk2h\") pod \"prometheus-operator-9cd89b74c-bk94k\" (UID: \"51e4a664-eed1-49a4-bfb8-49112a42b99c\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-bk94k"
Nov 02 02:48:04.828712 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:04.828682    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/51e4a664-eed1-49a4-bfb8-49112a42b99c-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-bk94k\" (UID: \"51e4a664-eed1-49a4-bfb8-49112a42b99c\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-bk94k"
Nov 02 02:48:04.830823 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:04.830798    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/51e4a664-eed1-49a4-bfb8-49112a42b99c-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-bk94k\" (UID: \"51e4a664-eed1-49a4-bfb8-49112a42b99c\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-bk94k"
Nov 02 02:48:04.830823 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:04.830815    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/51e4a664-eed1-49a4-bfb8-49112a42b99c-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-bk94k\" (UID: \"51e4a664-eed1-49a4-bfb8-49112a42b99c\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-bk94k"
Nov 02 02:48:04.887142 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:04.887084    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnk2h\" (UniqueName: \"kubernetes.io/projected/51e4a664-eed1-49a4-bfb8-49112a42b99c-kube-api-access-gnk2h\") pod \"prometheus-operator-9cd89b74c-bk94k\" (UID: \"51e4a664-eed1-49a4-bfb8-49112a42b99c\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-bk94k"
Nov 02 02:48:04.905857 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:04.905795    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-88c54d969-g7zvl" podStartSLOduration=2.467351618 podStartE2EDuration="5.905777869s" totalImagesPullingTime="3.438426251s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:47:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:48:04.904607326 +0000 UTC m=+25.483214703" watchObservedRunningTime="2026-11-02 02:48:04.905777869 +0000 UTC m=+25.484385248"
Nov 02 02:48:04.933851 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:04.933822    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-bk94k"
Nov 02 02:48:05.305341 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:05.305305    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-bk94k"]
Nov 02 02:48:05.308280 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:48:05.308251    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51e4a664_eed1_49a4_bfb8_49112a42b99c.slice/crio-d2c5c49d391bc1415c630e42d8e01a9111694d9558d80dba0fcbad76fdeda8bb WatchSource:0}: Error finding container d2c5c49d391bc1415c630e42d8e01a9111694d9558d80dba0fcbad76fdeda8bb: Status 404 returned error can't find the container with id d2c5c49d391bc1415c630e42d8e01a9111694d9558d80dba0fcbad76fdeda8bb
Nov 02 02:48:05.707693 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:05.707603    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-bk94k" event={"ID":"51e4a664-eed1-49a4-bfb8-49112a42b99c","Type":"ContainerStarted","Data":"d2c5c49d391bc1415c630e42d8e01a9111694d9558d80dba0fcbad76fdeda8bb"}
Nov 02 02:48:07.718205 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:07.718166    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-bk94k" event={"ID":"51e4a664-eed1-49a4-bfb8-49112a42b99c","Type":"ContainerStarted","Data":"eebb22c398cb3da86a4a7f9b9ecda1281970daf36c17c51085b930368e5a7538"}
Nov 02 02:48:07.718684 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:07.718226    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-bk94k" event={"ID":"51e4a664-eed1-49a4-bfb8-49112a42b99c","Type":"ContainerStarted","Data":"a7cc8808fb328ea4477ec1e324c5322809486105452743deba5e418bc80ec6b0"}
Nov 02 02:48:07.833736 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:07.833678    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-bk94k" podStartSLOduration=1.949445171 podStartE2EDuration="3.833660564s" totalImagesPullingTime="1.884215393s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:48:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:48:07.833186382 +0000 UTC m=+28.411793762" watchObservedRunningTime="2026-11-02 02:48:07.833660564 +0000 UTC m=+28.412267945"
Nov 02 02:48:09.683667 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:09.683638    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-fbvxx"
Nov 02 02:48:09.887459 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:09.887412    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-88c54d969-g7zvl"
Nov 02 02:48:09.887611 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:09.887514    4496 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-88c54d969-g7zvl"
Nov 02 02:48:09.892603 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:09.892579    4496 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-88c54d969-g7zvl"
Nov 02 02:48:10.732794 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:10.732763    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-88c54d969-g7zvl"
Nov 02 02:48:11.263132 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:11.263093    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-q2bzg"]
Nov 02 02:48:11.293901 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:11.293865    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-sdtnc"]
Nov 02 02:48:11.294067 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:11.293990    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-q2bzg"
Nov 02 02:48:11.307967 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:11.307944    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sdtnc"
Nov 02 02:48:11.327192 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:11.327169    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-q2bzg"]
Nov 02 02:48:11.351965 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:11.351938    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-sdtnc"]
Nov 02 02:48:11.376883 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:11.376856    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Nov 02 02:48:11.377264 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:11.377212    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Nov 02 02:48:11.377358 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:11.377276    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-4xg8t\""
Nov 02 02:48:11.395179 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:11.382694    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2d589c8b-5c19-4cee-a17e-c8bc46c6fa35-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-q2bzg\" (UID: \"2d589c8b-5c19-4cee-a17e-c8bc46c6fa35\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-q2bzg"
Nov 02 02:48:11.395179 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:11.382742    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/9cf4b8c4-e1ad-4880-a501-53d6a5970ac8-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-sdtnc\" (UID: \"9cf4b8c4-e1ad-4880-a501-53d6a5970ac8\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sdtnc"
Nov 02 02:48:11.395179 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:11.382771    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/9cf4b8c4-e1ad-4880-a501-53d6a5970ac8-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-sdtnc\" (UID: \"9cf4b8c4-e1ad-4880-a501-53d6a5970ac8\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sdtnc"
Nov 02 02:48:11.395179 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:11.382796    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2d589c8b-5c19-4cee-a17e-c8bc46c6fa35-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-q2bzg\" (UID: \"2d589c8b-5c19-4cee-a17e-c8bc46c6fa35\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-q2bzg"
Nov 02 02:48:11.395179 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:11.382824    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2d589c8b-5c19-4cee-a17e-c8bc46c6fa35-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-q2bzg\" (UID: \"2d589c8b-5c19-4cee-a17e-c8bc46c6fa35\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-q2bzg"
Nov 02 02:48:11.395179 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:11.382875    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9cf4b8c4-e1ad-4880-a501-53d6a5970ac8-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-sdtnc\" (UID: \"9cf4b8c4-e1ad-4880-a501-53d6a5970ac8\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sdtnc"
Nov 02 02:48:11.395179 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:11.382899    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llw5n\" (UniqueName: \"kubernetes.io/projected/9cf4b8c4-e1ad-4880-a501-53d6a5970ac8-kube-api-access-llw5n\") pod \"kube-state-metrics-5dc8d7d89b-sdtnc\" (UID: \"9cf4b8c4-e1ad-4880-a501-53d6a5970ac8\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sdtnc"
Nov 02 02:48:11.395179 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:11.382932    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jn4t\" (UniqueName: \"kubernetes.io/projected/2d589c8b-5c19-4cee-a17e-c8bc46c6fa35-kube-api-access-8jn4t\") pod \"openshift-state-metrics-7bd7489957-q2bzg\" (UID: \"2d589c8b-5c19-4cee-a17e-c8bc46c6fa35\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-q2bzg"
Nov 02 02:48:11.395179 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:11.382969    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9cf4b8c4-e1ad-4880-a501-53d6a5970ac8-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-sdtnc\" (UID: \"9cf4b8c4-e1ad-4880-a501-53d6a5970ac8\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sdtnc"
Nov 02 02:48:11.395179 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:11.383001    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9cf4b8c4-e1ad-4880-a501-53d6a5970ac8-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-sdtnc\" (UID: \"9cf4b8c4-e1ad-4880-a501-53d6a5970ac8\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sdtnc"
Nov 02 02:48:11.401408 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:11.401385    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Nov 02 02:48:11.402326 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:11.402307    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-d2wl9"]
Nov 02 02:48:11.418208 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:11.418176    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-d2wl9"
Nov 02 02:48:11.420407 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:11.420388    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-6ff565b847-nzkql"]
Nov 02 02:48:11.438242 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:11.438217    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6ff565b847-nzkql"
Nov 02 02:48:11.459579 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:11.459559    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Nov 02 02:48:11.460328 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:11.460310    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-gz6f9\""
Nov 02 02:48:11.462995 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:11.462977    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Nov 02 02:48:11.473585 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:11.473562    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 02 02:48:11.483580 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:11.483559    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/9cf4b8c4-e1ad-4880-a501-53d6a5970ac8-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-sdtnc\" (UID: \"9cf4b8c4-e1ad-4880-a501-53d6a5970ac8\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sdtnc"
Nov 02 02:48:11.483662 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:11.483598    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/9cf4b8c4-e1ad-4880-a501-53d6a5970ac8-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-sdtnc\" (UID: \"9cf4b8c4-e1ad-4880-a501-53d6a5970ac8\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sdtnc"
Nov 02 02:48:11.483814 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:11.483787    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2d589c8b-5c19-4cee-a17e-c8bc46c6fa35-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-q2bzg\" (UID: \"2d589c8b-5c19-4cee-a17e-c8bc46c6fa35\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-q2bzg"
Nov 02 02:48:11.483921 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:11.483838    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2d589c8b-5c19-4cee-a17e-c8bc46c6fa35-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-q2bzg\" (UID: \"2d589c8b-5c19-4cee-a17e-c8bc46c6fa35\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-q2bzg"
Nov 02 02:48:11.483921 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:11.483873    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/55c81f55-e3b2-4006-ad91-c2a843a426ad-node-exporter-textfile\") pod \"node-exporter-d2wl9\" (UID: \"55c81f55-e3b2-4006-ad91-c2a843a426ad\") " pod="openshift-monitoring/node-exporter-d2wl9"
Nov 02 02:48:11.483921 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:11.483891    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/55c81f55-e3b2-4006-ad91-c2a843a426ad-node-exporter-accelerators-collector-config\") pod \"node-exporter-d2wl9\" (UID: \"55c81f55-e3b2-4006-ad91-c2a843a426ad\") " pod="openshift-monitoring/node-exporter-d2wl9"
Nov 02 02:48:11.483921 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:11.483918    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9cf4b8c4-e1ad-4880-a501-53d6a5970ac8-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-sdtnc\" (UID: \"9cf4b8c4-e1ad-4880-a501-53d6a5970ac8\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sdtnc"
Nov 02 02:48:11.484113 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:11.483934    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llw5n\" (UniqueName: \"kubernetes.io/projected/9cf4b8c4-e1ad-4880-a501-53d6a5970ac8-kube-api-access-llw5n\") pod \"kube-state-metrics-5dc8d7d89b-sdtnc\" (UID: \"9cf4b8c4-e1ad-4880-a501-53d6a5970ac8\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sdtnc"
Nov 02 02:48:11.484113 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:11.483963    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jn4t\" (UniqueName: \"kubernetes.io/projected/2d589c8b-5c19-4cee-a17e-c8bc46c6fa35-kube-api-access-8jn4t\") pod \"openshift-state-metrics-7bd7489957-q2bzg\" (UID: \"2d589c8b-5c19-4cee-a17e-c8bc46c6fa35\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-q2bzg"
Nov 02 02:48:11.484113 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:11.483969    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/9cf4b8c4-e1ad-4880-a501-53d6a5970ac8-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-sdtnc\" (UID: \"9cf4b8c4-e1ad-4880-a501-53d6a5970ac8\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sdtnc"
Nov 02 02:48:11.484113 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:11.483986    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/55c81f55-e3b2-4006-ad91-c2a843a426ad-metrics-client-ca\") pod \"node-exporter-d2wl9\" (UID: \"55c81f55-e3b2-4006-ad91-c2a843a426ad\") " pod="openshift-monitoring/node-exporter-d2wl9"
Nov 02 02:48:11.484366 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:11.484343    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/9cf4b8c4-e1ad-4880-a501-53d6a5970ac8-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-sdtnc\" (UID: \"9cf4b8c4-e1ad-4880-a501-53d6a5970ac8\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sdtnc"
Nov 02 02:48:11.484441 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:11.484401    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9cf4b8c4-e1ad-4880-a501-53d6a5970ac8-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-sdtnc\" (UID: \"9cf4b8c4-e1ad-4880-a501-53d6a5970ac8\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sdtnc"
Nov 02 02:48:11.484498 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:11.484442    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/55c81f55-e3b2-4006-ad91-c2a843a426ad-root\") pod \"node-exporter-d2wl9\" (UID: \"55c81f55-e3b2-4006-ad91-c2a843a426ad\") " pod="openshift-monitoring/node-exporter-d2wl9"
Nov 02 02:48:11.484498 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:11.484472    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/55c81f55-e3b2-4006-ad91-c2a843a426ad-node-exporter-wtmp\") pod \"node-exporter-d2wl9\" (UID: \"55c81f55-e3b2-4006-ad91-c2a843a426ad\") " pod="openshift-monitoring/node-exporter-d2wl9"
Nov 02 02:48:11.484601 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:11.484519    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9cf4b8c4-e1ad-4880-a501-53d6a5970ac8-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-sdtnc\" (UID: \"9cf4b8c4-e1ad-4880-a501-53d6a5970ac8\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sdtnc"
Nov 02 02:48:11.484601 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:11.484550    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/55c81f55-e3b2-4006-ad91-c2a843a426ad-sys\") pod \"node-exporter-d2wl9\" (UID: \"55c81f55-e3b2-4006-ad91-c2a843a426ad\") " pod="openshift-monitoring/node-exporter-d2wl9"
Nov 02 02:48:11.484682 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:11.484598    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/55c81f55-e3b2-4006-ad91-c2a843a426ad-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-d2wl9\" (UID: \"55c81f55-e3b2-4006-ad91-c2a843a426ad\") " pod="openshift-monitoring/node-exporter-d2wl9"
Nov 02 02:48:11.484756 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:11.484735    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgfjs\" (UniqueName: \"kubernetes.io/projected/55c81f55-e3b2-4006-ad91-c2a843a426ad-kube-api-access-mgfjs\") pod \"node-exporter-d2wl9\" (UID: \"55c81f55-e3b2-4006-ad91-c2a843a426ad\") " pod="openshift-monitoring/node-exporter-d2wl9"
Nov 02 02:48:11.484845 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:11.484795    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2d589c8b-5c19-4cee-a17e-c8bc46c6fa35-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-q2bzg\" (UID: \"2d589c8b-5c19-4cee-a17e-c8bc46c6fa35\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-q2bzg"
Nov 02 02:48:11.484845 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:11.484827    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/55c81f55-e3b2-4006-ad91-c2a843a426ad-node-exporter-tls\") pod \"node-exporter-d2wl9\" (UID: \"55c81f55-e3b2-4006-ad91-c2a843a426ad\") " pod="openshift-monitoring/node-exporter-d2wl9"
Nov 02 02:48:11.485204 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:11.485177    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9cf4b8c4-e1ad-4880-a501-53d6a5970ac8-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-sdtnc\" (UID: \"9cf4b8c4-e1ad-4880-a501-53d6a5970ac8\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sdtnc"
Nov 02 02:48:11.485621 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:11.485597    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2d589c8b-5c19-4cee-a17e-c8bc46c6fa35-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-q2bzg\" (UID: \"2d589c8b-5c19-4cee-a17e-c8bc46c6fa35\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-q2bzg"
Nov 02 02:48:11.486583 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:11.486562    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9cf4b8c4-e1ad-4880-a501-53d6a5970ac8-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-sdtnc\" (UID: \"9cf4b8c4-e1ad-4880-a501-53d6a5970ac8\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sdtnc"
Nov 02 02:48:11.487254 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:11.487238    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9cf4b8c4-e1ad-4880-a501-53d6a5970ac8-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-sdtnc\" (UID: \"9cf4b8c4-e1ad-4880-a501-53d6a5970ac8\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sdtnc"
Nov 02 02:48:11.487466 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:11.487450    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2d589c8b-5c19-4cee-a17e-c8bc46c6fa35-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-q2bzg\" (UID: \"2d589c8b-5c19-4cee-a17e-c8bc46c6fa35\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-q2bzg"
Nov 02 02:48:11.487531 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:11.487471    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2d589c8b-5c19-4cee-a17e-c8bc46c6fa35-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-q2bzg\" (UID: \"2d589c8b-5c19-4cee-a17e-c8bc46c6fa35\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-q2bzg"
Nov 02 02:48:11.493950 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:11.493895    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6ff565b847-nzkql"]
Nov 02 02:48:11.543251 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:11.543178    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jn4t\" (UniqueName: \"kubernetes.io/projected/2d589c8b-5c19-4cee-a17e-c8bc46c6fa35-kube-api-access-8jn4t\") pod \"openshift-state-metrics-7bd7489957-q2bzg\" (UID: \"2d589c8b-5c19-4cee-a17e-c8bc46c6fa35\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-q2bzg"
Nov 02 02:48:11.543439 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:11.543416    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llw5n\" (UniqueName: \"kubernetes.io/projected/9cf4b8c4-e1ad-4880-a501-53d6a5970ac8-kube-api-access-llw5n\") pod \"kube-state-metrics-5dc8d7d89b-sdtnc\" (UID: \"9cf4b8c4-e1ad-4880-a501-53d6a5970ac8\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sdtnc"
Nov 02 02:48:11.554190 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:11.554163    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 02 02:48:11.554322 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:11.554203    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 02 02:48:11.554322 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:11.554249    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-gtx4f\""
Nov 02 02:48:11.585547 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:11.585517    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/55c81f55-e3b2-4006-ad91-c2a843a426ad-node-exporter-textfile\") pod \"node-exporter-d2wl9\" (UID: \"55c81f55-e3b2-4006-ad91-c2a843a426ad\") " pod="openshift-monitoring/node-exporter-d2wl9"
Nov 02 02:48:11.585685 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:11.585668    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/55c81f55-e3b2-4006-ad91-c2a843a426ad-node-exporter-accelerators-collector-config\") pod \"node-exporter-d2wl9\" (UID: \"55c81f55-e3b2-4006-ad91-c2a843a426ad\") " pod="openshift-monitoring/node-exporter-d2wl9"
Nov 02 02:48:11.585731 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:11.585714    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29k72\" (UniqueName: \"kubernetes.io/projected/f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1-kube-api-access-29k72\") pod \"console-6ff565b847-nzkql\" (UID: \"f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1\") " pod="openshift-console/console-6ff565b847-nzkql"
Nov 02 02:48:11.585768 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:11.585749    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/55c81f55-e3b2-4006-ad91-c2a843a426ad-metrics-client-ca\") pod \"node-exporter-d2wl9\" (UID: \"55c81f55-e3b2-4006-ad91-c2a843a426ad\") " pod="openshift-monitoring/node-exporter-d2wl9"
Nov 02 02:48:11.585806 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:11.585779    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/55c81f55-e3b2-4006-ad91-c2a843a426ad-root\") pod \"node-exporter-d2wl9\" (UID: \"55c81f55-e3b2-4006-ad91-c2a843a426ad\") " pod="openshift-monitoring/node-exporter-d2wl9"
Nov 02 02:48:11.585845 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:11.585801    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/55c81f55-e3b2-4006-ad91-c2a843a426ad-node-exporter-wtmp\") pod \"node-exporter-d2wl9\" (UID: \"55c81f55-e3b2-4006-ad91-c2a843a426ad\") " pod="openshift-monitoring/node-exporter-d2wl9"
Nov 02 02:48:11.585890 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:11.585844    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1-console-serving-cert\") pod \"console-6ff565b847-nzkql\" (UID: \"f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1\") " pod="openshift-console/console-6ff565b847-nzkql"
Nov 02 02:48:11.585890 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:11.585867    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1-trusted-ca-bundle\") pod \"console-6ff565b847-nzkql\" (UID: \"f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1\") " pod="openshift-console/console-6ff565b847-nzkql"
Nov 02 02:48:11.585979 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:11.585903    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/55c81f55-e3b2-4006-ad91-c2a843a426ad-sys\") pod \"node-exporter-d2wl9\" (UID: \"55c81f55-e3b2-4006-ad91-c2a843a426ad\") " pod="openshift-monitoring/node-exporter-d2wl9"
Nov 02 02:48:11.585979 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:11.585926    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1-tmp\") pod \"console-6ff565b847-nzkql\" (UID: \"f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1\") " pod="openshift-console/console-6ff565b847-nzkql"
Nov 02 02:48:11.585979 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:11.585948    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1-oauth-serving-cert\") pod \"console-6ff565b847-nzkql\" (UID: \"f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1\") " pod="openshift-console/console-6ff565b847-nzkql"
Nov 02 02:48:11.586173 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:11.585975    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/55c81f55-e3b2-4006-ad91-c2a843a426ad-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-d2wl9\" (UID: \"55c81f55-e3b2-4006-ad91-c2a843a426ad\") " pod="openshift-monitoring/node-exporter-d2wl9"
Nov 02 02:48:11.586173 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:11.586029    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgfjs\" (UniqueName: \"kubernetes.io/projected/55c81f55-e3b2-4006-ad91-c2a843a426ad-kube-api-access-mgfjs\") pod \"node-exporter-d2wl9\" (UID: \"55c81f55-e3b2-4006-ad91-c2a843a426ad\") " pod="openshift-monitoring/node-exporter-d2wl9"
Nov 02 02:48:11.586173 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:11.586060    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1-service-ca\") pod \"console-6ff565b847-nzkql\" (UID: \"f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1\") " pod="openshift-console/console-6ff565b847-nzkql"
Nov 02 02:48:11.586173 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:11.586132    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/55c81f55-e3b2-4006-ad91-c2a843a426ad-sys\") pod \"node-exporter-d2wl9\" (UID: \"55c81f55-e3b2-4006-ad91-c2a843a426ad\") " pod="openshift-monitoring/node-exporter-d2wl9"
Nov 02 02:48:11.586173 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:11.586150    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1-console-config\") pod \"console-6ff565b847-nzkql\" (UID: \"f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1\") " pod="openshift-console/console-6ff565b847-nzkql"
Nov 02 02:48:11.586414 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:11.586293    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/55c81f55-e3b2-4006-ad91-c2a843a426ad-node-exporter-wtmp\") pod \"node-exporter-d2wl9\" (UID: \"55c81f55-e3b2-4006-ad91-c2a843a426ad\") " pod="openshift-monitoring/node-exporter-d2wl9"
Nov 02 02:48:11.586472 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:11.586458    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/55c81f55-e3b2-4006-ad91-c2a843a426ad-metrics-client-ca\") pod \"node-exporter-d2wl9\" (UID: \"55c81f55-e3b2-4006-ad91-c2a843a426ad\") " pod="openshift-monitoring/node-exporter-d2wl9"
Nov 02 02:48:11.586528 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:11.586517    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/55c81f55-e3b2-4006-ad91-c2a843a426ad-root\") pod \"node-exporter-d2wl9\" (UID: \"55c81f55-e3b2-4006-ad91-c2a843a426ad\") " pod="openshift-monitoring/node-exporter-d2wl9"
Nov 02 02:48:11.586867 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:11.586844    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/55c81f55-e3b2-4006-ad91-c2a843a426ad-node-exporter-accelerators-collector-config\") pod \"node-exporter-d2wl9\" (UID: \"55c81f55-e3b2-4006-ad91-c2a843a426ad\") " pod="openshift-monitoring/node-exporter-d2wl9"
Nov 02 02:48:11.586971 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:11.586907    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/55c81f55-e3b2-4006-ad91-c2a843a426ad-node-exporter-tls\") pod \"node-exporter-d2wl9\" (UID: \"55c81f55-e3b2-4006-ad91-c2a843a426ad\") " pod="openshift-monitoring/node-exporter-d2wl9"
Nov 02 02:48:11.587053 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:11.587002    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/55c81f55-e3b2-4006-ad91-c2a843a426ad-node-exporter-textfile\") pod \"node-exporter-d2wl9\" (UID: \"55c81f55-e3b2-4006-ad91-c2a843a426ad\") " pod="openshift-monitoring/node-exporter-d2wl9"
Nov 02 02:48:11.587160 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:11.587140    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1-console-oauth-config\") pod \"console-6ff565b847-nzkql\" (UID: \"f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1\") " pod="openshift-console/console-6ff565b847-nzkql"
Nov 02 02:48:11.590654 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:11.589571    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/55c81f55-e3b2-4006-ad91-c2a843a426ad-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-d2wl9\" (UID: \"55c81f55-e3b2-4006-ad91-c2a843a426ad\") " pod="openshift-monitoring/node-exporter-d2wl9"
Nov 02 02:48:11.591123 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:11.591101    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/55c81f55-e3b2-4006-ad91-c2a843a426ad-node-exporter-tls\") pod \"node-exporter-d2wl9\" (UID: \"55c81f55-e3b2-4006-ad91-c2a843a426ad\") " pod="openshift-monitoring/node-exporter-d2wl9"
Nov 02 02:48:11.615350 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:11.615329    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Nov 02 02:48:11.644292 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:11.644266    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgfjs\" (UniqueName: \"kubernetes.io/projected/55c81f55-e3b2-4006-ad91-c2a843a426ad-kube-api-access-mgfjs\") pod \"node-exporter-d2wl9\" (UID: \"55c81f55-e3b2-4006-ad91-c2a843a426ad\") " pod="openshift-monitoring/node-exporter-d2wl9"
Nov 02 02:48:11.644415 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:11.644383    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-q2bzg"
Nov 02 02:48:11.687930 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:11.687904    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1-console-serving-cert\") pod \"console-6ff565b847-nzkql\" (UID: \"f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1\") " pod="openshift-console/console-6ff565b847-nzkql"
Nov 02 02:48:11.688062 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:11.687942    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1-trusted-ca-bundle\") pod \"console-6ff565b847-nzkql\" (UID: \"f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1\") " pod="openshift-console/console-6ff565b847-nzkql"
Nov 02 02:48:11.688062 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:11.687984    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1-tmp\") pod \"console-6ff565b847-nzkql\" (UID: \"f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1\") " pod="openshift-console/console-6ff565b847-nzkql"
Nov 02 02:48:11.688062 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:11.688008    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1-oauth-serving-cert\") pod \"console-6ff565b847-nzkql\" (UID: \"f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1\") " pod="openshift-console/console-6ff565b847-nzkql"
Nov 02 02:48:11.688241 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:11.688067    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1-service-ca\") pod \"console-6ff565b847-nzkql\" (UID: \"f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1\") " pod="openshift-console/console-6ff565b847-nzkql"
Nov 02 02:48:11.688241 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:11.688097    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1-console-config\") pod \"console-6ff565b847-nzkql\" (UID: \"f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1\") " pod="openshift-console/console-6ff565b847-nzkql"
Nov 02 02:48:11.688241 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:11.688126    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1-console-oauth-config\") pod \"console-6ff565b847-nzkql\" (UID: \"f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1\") " pod="openshift-console/console-6ff565b847-nzkql"
Nov 02 02:48:11.688241 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:11.688173    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29k72\" (UniqueName: \"kubernetes.io/projected/f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1-kube-api-access-29k72\") pod \"console-6ff565b847-nzkql\" (UID: \"f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1\") " pod="openshift-console/console-6ff565b847-nzkql"
Nov 02 02:48:11.689119 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:11.689087    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1-tmp\") pod \"console-6ff565b847-nzkql\" (UID: \"f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1\") " pod="openshift-console/console-6ff565b847-nzkql"
Nov 02 02:48:11.689317 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:11.689289    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1-oauth-serving-cert\") pod \"console-6ff565b847-nzkql\" (UID: \"f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1\") " pod="openshift-console/console-6ff565b847-nzkql"
Nov 02 02:48:11.689955 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:11.689928    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1-console-config\") pod \"console-6ff565b847-nzkql\" (UID: \"f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1\") " pod="openshift-console/console-6ff565b847-nzkql"
Nov 02 02:48:11.690052 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:11.689955    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1-trusted-ca-bundle\") pod \"console-6ff565b847-nzkql\" (UID: \"f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1\") " pod="openshift-console/console-6ff565b847-nzkql"
Nov 02 02:48:11.690103 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:11.690071    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1-service-ca\") pod \"console-6ff565b847-nzkql\" (UID: \"f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1\") " pod="openshift-console/console-6ff565b847-nzkql"
Nov 02 02:48:11.691646 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:11.691615    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1-console-oauth-config\") pod \"console-6ff565b847-nzkql\" (UID: \"f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1\") " pod="openshift-console/console-6ff565b847-nzkql"
Nov 02 02:48:11.691725 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:11.691696    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1-console-serving-cert\") pod \"console-6ff565b847-nzkql\" (UID: \"f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1\") " pod="openshift-console/console-6ff565b847-nzkql"
Nov 02 02:48:11.700534 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:11.700516    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sdtnc"
Nov 02 02:48:11.750570 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:11.750482    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29k72\" (UniqueName: \"kubernetes.io/projected/f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1-kube-api-access-29k72\") pod \"console-6ff565b847-nzkql\" (UID: \"f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1\") " pod="openshift-console/console-6ff565b847-nzkql"
Nov 02 02:48:11.772155 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:11.772123    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-d2wl9"
Nov 02 02:48:11.791183 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:11.788982    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6ff565b847-nzkql"
Nov 02 02:48:11.813144 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:48:11.813116    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55c81f55_e3b2_4006_ad91_c2a843a426ad.slice/crio-3027d15917df30ac59cb7e13167f0722faaa8b3be1074555b1b56273ed91a5fc WatchSource:0}: Error finding container 3027d15917df30ac59cb7e13167f0722faaa8b3be1074555b1b56273ed91a5fc: Status 404 returned error can't find the container with id 3027d15917df30ac59cb7e13167f0722faaa8b3be1074555b1b56273ed91a5fc
Nov 02 02:48:12.103883 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:12.103809    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-q2bzg"]
Nov 02 02:48:12.106833 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:48:12.106799    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d589c8b_5c19_4cee_a17e_c8bc46c6fa35.slice/crio-4b62f2ac58f948d0f29d6e361275cb6b267103f1ae6db70c73f4f91f78f94b7d WatchSource:0}: Error finding container 4b62f2ac58f948d0f29d6e361275cb6b267103f1ae6db70c73f4f91f78f94b7d: Status 404 returned error can't find the container with id 4b62f2ac58f948d0f29d6e361275cb6b267103f1ae6db70c73f4f91f78f94b7d
Nov 02 02:48:12.155202 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:12.155175    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-sdtnc"]
Nov 02 02:48:12.157722 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:48:12.157695    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cf4b8c4_e1ad_4880_a501_53d6a5970ac8.slice/crio-59afa495a05e02b87e79634a065c798a24097afa47c078962258dc5698bf43b4 WatchSource:0}: Error finding container 59afa495a05e02b87e79634a065c798a24097afa47c078962258dc5698bf43b4: Status 404 returned error can't find the container with id 59afa495a05e02b87e79634a065c798a24097afa47c078962258dc5698bf43b4
Nov 02 02:48:12.202625 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:12.201402    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6ff565b847-nzkql"]
Nov 02 02:48:12.204181 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:48:12.204152    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf72c1a17_afce_45cb_ad70_b5a2f4ab0cd1.slice/crio-d73e5c6ad2798058bd4e36d4dfdbdd34980fbb9aee08723a2f67059640684e6e WatchSource:0}: Error finding container d73e5c6ad2798058bd4e36d4dfdbdd34980fbb9aee08723a2f67059640684e6e: Status 404 returned error can't find the container with id d73e5c6ad2798058bd4e36d4dfdbdd34980fbb9aee08723a2f67059640684e6e
Nov 02 02:48:12.742074 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:12.742032    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6ff565b847-nzkql" event={"ID":"f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1","Type":"ContainerStarted","Data":"e7af03309e92c1826dfe8d3ed60747b9c25f011f063e6214c15c25b15327520f"}
Nov 02 02:48:12.742265 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:12.742104    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6ff565b847-nzkql" event={"ID":"f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1","Type":"ContainerStarted","Data":"d73e5c6ad2798058bd4e36d4dfdbdd34980fbb9aee08723a2f67059640684e6e"}
Nov 02 02:48:12.744348 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:12.744308    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-q2bzg" event={"ID":"2d589c8b-5c19-4cee-a17e-c8bc46c6fa35","Type":"ContainerStarted","Data":"c64b305a77b5f01ddf000b0aaa815609d38c982e903ec4956da86448d2fe10fb"}
Nov 02 02:48:12.744487 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:12.744373    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-q2bzg" event={"ID":"2d589c8b-5c19-4cee-a17e-c8bc46c6fa35","Type":"ContainerStarted","Data":"50300a56d6f52a3809b2daf9b87997d3f8b6b00015e392fdf3d17e57b362edca"}
Nov 02 02:48:12.744487 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:12.744410    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-q2bzg" event={"ID":"2d589c8b-5c19-4cee-a17e-c8bc46c6fa35","Type":"ContainerStarted","Data":"4b62f2ac58f948d0f29d6e361275cb6b267103f1ae6db70c73f4f91f78f94b7d"}
Nov 02 02:48:12.745710 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:12.745685    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-d2wl9" event={"ID":"55c81f55-e3b2-4006-ad91-c2a843a426ad","Type":"ContainerStarted","Data":"3027d15917df30ac59cb7e13167f0722faaa8b3be1074555b1b56273ed91a5fc"}
Nov 02 02:48:12.747068 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:12.746995    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sdtnc" event={"ID":"9cf4b8c4-e1ad-4880-a501-53d6a5970ac8","Type":"ContainerStarted","Data":"59afa495a05e02b87e79634a065c798a24097afa47c078962258dc5698bf43b4"}
Nov 02 02:48:12.858284 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:12.858236    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-6ff565b847-nzkql" podStartSLOduration=1.85821892 podStartE2EDuration="1.85821892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:48:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:48:12.857689346 +0000 UTC m=+33.436296727" watchObservedRunningTime="2026-11-02 02:48:12.85821892 +0000 UTC m=+33.436826299"
Nov 02 02:48:13.732286 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:13.732258    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-47f7x"
Nov 02 02:48:13.905895 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:13.905860    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-fb86b55c4-sq549"]
Nov 02 02:48:13.919426 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:13.919402    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-fb86b55c4-sq549"
Nov 02 02:48:13.973957 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:13.973928    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Nov 02 02:48:13.974152 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:13.973986    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-fb86b55c4-sq549"]
Nov 02 02:48:14.009418 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:14.009339    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6ff7dc8e-86b1-4e4b-b16d-9fbac2dea1cd-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-fb86b55c4-sq549\" (UID: \"6ff7dc8e-86b1-4e4b-b16d-9fbac2dea1cd\") " pod="openshift-monitoring/thanos-querier-fb86b55c4-sq549"
Nov 02 02:48:14.009418 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:14.009399    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6ff7dc8e-86b1-4e4b-b16d-9fbac2dea1cd-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-fb86b55c4-sq549\" (UID: \"6ff7dc8e-86b1-4e4b-b16d-9fbac2dea1cd\") " pod="openshift-monitoring/thanos-querier-fb86b55c4-sq549"
Nov 02 02:48:14.009626 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:14.009511    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/6ff7dc8e-86b1-4e4b-b16d-9fbac2dea1cd-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-fb86b55c4-sq549\" (UID: \"6ff7dc8e-86b1-4e4b-b16d-9fbac2dea1cd\") " pod="openshift-monitoring/thanos-querier-fb86b55c4-sq549"
Nov 02 02:48:14.009626 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:14.009557    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6ff7dc8e-86b1-4e4b-b16d-9fbac2dea1cd-tmp\") pod \"thanos-querier-fb86b55c4-sq549\" (UID: \"6ff7dc8e-86b1-4e4b-b16d-9fbac2dea1cd\") " pod="openshift-monitoring/thanos-querier-fb86b55c4-sq549"
Nov 02 02:48:14.009729 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:14.009640    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6ff7dc8e-86b1-4e4b-b16d-9fbac2dea1cd-metrics-client-ca\") pod \"thanos-querier-fb86b55c4-sq549\" (UID: \"6ff7dc8e-86b1-4e4b-b16d-9fbac2dea1cd\") " pod="openshift-monitoring/thanos-querier-fb86b55c4-sq549"
Nov 02 02:48:14.009729 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:14.009669    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtlbh\" (UniqueName: \"kubernetes.io/projected/6ff7dc8e-86b1-4e4b-b16d-9fbac2dea1cd-kube-api-access-qtlbh\") pod \"thanos-querier-fb86b55c4-sq549\" (UID: \"6ff7dc8e-86b1-4e4b-b16d-9fbac2dea1cd\") " pod="openshift-monitoring/thanos-querier-fb86b55c4-sq549"
Nov 02 02:48:14.009729 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:14.009695    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6ff7dc8e-86b1-4e4b-b16d-9fbac2dea1cd-secret-grpc-tls\") pod \"thanos-querier-fb86b55c4-sq549\" (UID: \"6ff7dc8e-86b1-4e4b-b16d-9fbac2dea1cd\") " pod="openshift-monitoring/thanos-querier-fb86b55c4-sq549"
Nov 02 02:48:14.009851 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:14.009813    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/6ff7dc8e-86b1-4e4b-b16d-9fbac2dea1cd-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-fb86b55c4-sq549\" (UID: \"6ff7dc8e-86b1-4e4b-b16d-9fbac2dea1cd\") " pod="openshift-monitoring/thanos-querier-fb86b55c4-sq549"
Nov 02 02:48:14.009893 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:14.009857    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/6ff7dc8e-86b1-4e4b-b16d-9fbac2dea1cd-secret-thanos-querier-tls\") pod \"thanos-querier-fb86b55c4-sq549\" (UID: \"6ff7dc8e-86b1-4e4b-b16d-9fbac2dea1cd\") " pod="openshift-monitoring/thanos-querier-fb86b55c4-sq549"
Nov 02 02:48:14.023738 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:14.023620    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Nov 02 02:48:14.023738 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:14.023625    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Nov 02 02:48:14.023738 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:14.023702    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Nov 02 02:48:14.023738 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:14.023726    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-qkkhd\""
Nov 02 02:48:14.023991 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:14.023801    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-9cpnrn7ocbsgj\""
Nov 02 02:48:14.023991 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:14.023873    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Nov 02 02:48:14.110843 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:14.110808    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/6ff7dc8e-86b1-4e4b-b16d-9fbac2dea1cd-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-fb86b55c4-sq549\" (UID: \"6ff7dc8e-86b1-4e4b-b16d-9fbac2dea1cd\") " pod="openshift-monitoring/thanos-querier-fb86b55c4-sq549"
Nov 02 02:48:14.110843 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:14.110848    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/6ff7dc8e-86b1-4e4b-b16d-9fbac2dea1cd-secret-thanos-querier-tls\") pod \"thanos-querier-fb86b55c4-sq549\" (UID: \"6ff7dc8e-86b1-4e4b-b16d-9fbac2dea1cd\") " pod="openshift-monitoring/thanos-querier-fb86b55c4-sq549"
Nov 02 02:48:14.111053 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:14.110985    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6ff7dc8e-86b1-4e4b-b16d-9fbac2dea1cd-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-fb86b55c4-sq549\" (UID: \"6ff7dc8e-86b1-4e4b-b16d-9fbac2dea1cd\") " pod="openshift-monitoring/thanos-querier-fb86b55c4-sq549"
Nov 02 02:48:14.111053 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:14.111050    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6ff7dc8e-86b1-4e4b-b16d-9fbac2dea1cd-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-fb86b55c4-sq549\" (UID: \"6ff7dc8e-86b1-4e4b-b16d-9fbac2dea1cd\") " pod="openshift-monitoring/thanos-querier-fb86b55c4-sq549"
Nov 02 02:48:14.111149 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:14.111077    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/6ff7dc8e-86b1-4e4b-b16d-9fbac2dea1cd-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-fb86b55c4-sq549\" (UID: \"6ff7dc8e-86b1-4e4b-b16d-9fbac2dea1cd\") " pod="openshift-monitoring/thanos-querier-fb86b55c4-sq549"
Nov 02 02:48:14.111149 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:14.111107    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6ff7dc8e-86b1-4e4b-b16d-9fbac2dea1cd-tmp\") pod \"thanos-querier-fb86b55c4-sq549\" (UID: \"6ff7dc8e-86b1-4e4b-b16d-9fbac2dea1cd\") " pod="openshift-monitoring/thanos-querier-fb86b55c4-sq549"
Nov 02 02:48:14.111251 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:14.111157    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6ff7dc8e-86b1-4e4b-b16d-9fbac2dea1cd-metrics-client-ca\") pod \"thanos-querier-fb86b55c4-sq549\" (UID: \"6ff7dc8e-86b1-4e4b-b16d-9fbac2dea1cd\") " pod="openshift-monitoring/thanos-querier-fb86b55c4-sq549"
Nov 02 02:48:14.111251 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:14.111187    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtlbh\" (UniqueName: \"kubernetes.io/projected/6ff7dc8e-86b1-4e4b-b16d-9fbac2dea1cd-kube-api-access-qtlbh\") pod \"thanos-querier-fb86b55c4-sq549\" (UID: \"6ff7dc8e-86b1-4e4b-b16d-9fbac2dea1cd\") " pod="openshift-monitoring/thanos-querier-fb86b55c4-sq549"
Nov 02 02:48:14.111251 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:14.111216    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6ff7dc8e-86b1-4e4b-b16d-9fbac2dea1cd-secret-grpc-tls\") pod \"thanos-querier-fb86b55c4-sq549\" (UID: \"6ff7dc8e-86b1-4e4b-b16d-9fbac2dea1cd\") " pod="openshift-monitoring/thanos-querier-fb86b55c4-sq549"
Nov 02 02:48:14.111924 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:14.111580    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6ff7dc8e-86b1-4e4b-b16d-9fbac2dea1cd-tmp\") pod \"thanos-querier-fb86b55c4-sq549\" (UID: \"6ff7dc8e-86b1-4e4b-b16d-9fbac2dea1cd\") " pod="openshift-monitoring/thanos-querier-fb86b55c4-sq549"
Nov 02 02:48:14.112195 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:14.112172    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6ff7dc8e-86b1-4e4b-b16d-9fbac2dea1cd-metrics-client-ca\") pod \"thanos-querier-fb86b55c4-sq549\" (UID: \"6ff7dc8e-86b1-4e4b-b16d-9fbac2dea1cd\") " pod="openshift-monitoring/thanos-querier-fb86b55c4-sq549"
Nov 02 02:48:14.113858 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:14.113834    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6ff7dc8e-86b1-4e4b-b16d-9fbac2dea1cd-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-fb86b55c4-sq549\" (UID: \"6ff7dc8e-86b1-4e4b-b16d-9fbac2dea1cd\") " pod="openshift-monitoring/thanos-querier-fb86b55c4-sq549"
Nov 02 02:48:14.113979 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:14.113892    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/6ff7dc8e-86b1-4e4b-b16d-9fbac2dea1cd-secret-thanos-querier-tls\") pod \"thanos-querier-fb86b55c4-sq549\" (UID: \"6ff7dc8e-86b1-4e4b-b16d-9fbac2dea1cd\") " pod="openshift-monitoring/thanos-querier-fb86b55c4-sq549"
Nov 02 02:48:14.113979 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:14.113939    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6ff7dc8e-86b1-4e4b-b16d-9fbac2dea1cd-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-fb86b55c4-sq549\" (UID: \"6ff7dc8e-86b1-4e4b-b16d-9fbac2dea1cd\") " pod="openshift-monitoring/thanos-querier-fb86b55c4-sq549"
Nov 02 02:48:14.113979 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:14.113962    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/6ff7dc8e-86b1-4e4b-b16d-9fbac2dea1cd-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-fb86b55c4-sq549\" (UID: \"6ff7dc8e-86b1-4e4b-b16d-9fbac2dea1cd\") " pod="openshift-monitoring/thanos-querier-fb86b55c4-sq549"
Nov 02 02:48:14.114257 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:14.114241    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6ff7dc8e-86b1-4e4b-b16d-9fbac2dea1cd-secret-grpc-tls\") pod \"thanos-querier-fb86b55c4-sq549\" (UID: \"6ff7dc8e-86b1-4e4b-b16d-9fbac2dea1cd\") " pod="openshift-monitoring/thanos-querier-fb86b55c4-sq549"
Nov 02 02:48:14.114501 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:14.114481    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/6ff7dc8e-86b1-4e4b-b16d-9fbac2dea1cd-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-fb86b55c4-sq549\" (UID: \"6ff7dc8e-86b1-4e4b-b16d-9fbac2dea1cd\") " pod="openshift-monitoring/thanos-querier-fb86b55c4-sq549"
Nov 02 02:48:14.168949 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:14.168923    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtlbh\" (UniqueName: \"kubernetes.io/projected/6ff7dc8e-86b1-4e4b-b16d-9fbac2dea1cd-kube-api-access-qtlbh\") pod \"thanos-querier-fb86b55c4-sq549\" (UID: \"6ff7dc8e-86b1-4e4b-b16d-9fbac2dea1cd\") " pod="openshift-monitoring/thanos-querier-fb86b55c4-sq549"
Nov 02 02:48:14.327202 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:14.327147    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-fb86b55c4-sq549"
Nov 02 02:48:15.278370 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:15.278337    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-fb86b55c4-sq549"]
Nov 02 02:48:15.283273 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:48:15.283136    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ff7dc8e_86b1_4e4b_b16d_9fbac2dea1cd.slice/crio-8378e3a725dc6c87ab0577d222ce642aa7c4799cd9c5f7132bafac2fc5bc04af WatchSource:0}: Error finding container 8378e3a725dc6c87ab0577d222ce642aa7c4799cd9c5f7132bafac2fc5bc04af: Status 404 returned error can't find the container with id 8378e3a725dc6c87ab0577d222ce642aa7c4799cd9c5f7132bafac2fc5bc04af
Nov 02 02:48:15.764619 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:15.764566    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-q2bzg" event={"ID":"2d589c8b-5c19-4cee-a17e-c8bc46c6fa35","Type":"ContainerStarted","Data":"89f3d1c75d1566d380219a39493e7ff01e63fdfb0cb4be840b517a7eb85c68cf"}
Nov 02 02:48:15.767552 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:15.767523    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sdtnc" event={"ID":"9cf4b8c4-e1ad-4880-a501-53d6a5970ac8","Type":"ContainerStarted","Data":"c2746ac4497816b9b7d77cb4168868068a1fe4744545e10ac40a808e96ac3095"}
Nov 02 02:48:15.767759 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:15.767741    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sdtnc" event={"ID":"9cf4b8c4-e1ad-4880-a501-53d6a5970ac8","Type":"ContainerStarted","Data":"60922d03525efc9d2330bcf69c4c7ff7bc9a8d7af2f526d5b376ddc4d157885b"}
Nov 02 02:48:15.767916 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:15.767868    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sdtnc" event={"ID":"9cf4b8c4-e1ad-4880-a501-53d6a5970ac8","Type":"ContainerStarted","Data":"ab41d6888765ce624e840235907b83b064aa019d2875c1eca6524907fff60d08"}
Nov 02 02:48:15.769570 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:15.769533    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-zrssn" event={"ID":"87db08be-0446-48e1-8012-623b96e7c646","Type":"ContainerStarted","Data":"848ecec77ed020d378a50986b3031681aa0cbfb4df6e58fb4c9f9130e5b93eba"}
Nov 02 02:48:15.769767 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:15.769749    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-zrssn"
Nov 02 02:48:15.772054 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:15.771983    4496 generic.go:356] "Generic (PLEG): container finished" podID="55c81f55-e3b2-4006-ad91-c2a843a426ad" containerID="0222c82d5207b9ae9ef535a0acab65ccd5ff9ba7527ac381273a4a166af193c1" exitCode=0
Nov 02 02:48:15.772152 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:15.772065    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-d2wl9" event={"ID":"55c81f55-e3b2-4006-ad91-c2a843a426ad","Type":"ContainerDied","Data":"0222c82d5207b9ae9ef535a0acab65ccd5ff9ba7527ac381273a4a166af193c1"}
Nov 02 02:48:15.774241 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:15.774220    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-fb86b55c4-sq549" event={"ID":"6ff7dc8e-86b1-4e4b-b16d-9fbac2dea1cd","Type":"ContainerStarted","Data":"8378e3a725dc6c87ab0577d222ce642aa7c4799cd9c5f7132bafac2fc5bc04af"}
Nov 02 02:48:15.782497 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:15.782476    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-zrssn"
Nov 02 02:48:16.004068 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:16.003986    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-sdtnc" podStartSLOduration=2.359422946 podStartE2EDuration="5.003973522s" totalImagesPullingTime="2.644550576s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:48:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:48:16.003132602 +0000 UTC m=+36.581739982" watchObservedRunningTime="2026-11-02 02:48:16.003973522 +0000 UTC m=+36.582580901"
Nov 02 02:48:16.004440 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:16.004405    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-q2bzg" podStartSLOduration=2.52284472 podStartE2EDuration="5.00439475s" totalImagesPullingTime="2.48155003s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:48:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:48:15.883344451 +0000 UTC m=+36.461951832" watchObservedRunningTime="2026-11-02 02:48:16.00439475 +0000 UTC m=+36.583002129"
Nov 02 02:48:16.018131 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:16.018051    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-d7ljw"]
Nov 02 02:48:16.020924 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:16.020902    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-d7ljw"
Nov 02 02:48:16.073709 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:16.073662    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-qmwzx\""
Nov 02 02:48:16.074518 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:16.074493    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Nov 02 02:48:16.082942 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:16.079238    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-d7ljw"]
Nov 02 02:48:16.120595 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:16.120533    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-zrssn" podStartSLOduration=2.658383993 podStartE2EDuration="21.120516999s" totalImagesPullingTime="18.462133006s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:47:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:48:16.119838834 +0000 UTC m=+36.698446219" watchObservedRunningTime="2026-11-02 02:48:16.120516999 +0000 UTC m=+36.699124382"
Nov 02 02:48:16.132700 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:16.132667    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnrrh\" (UniqueName: \"kubernetes.io/projected/67a0e3bb-443b-4f8e-aa30-ddedc41efb83-kube-api-access-mnrrh\") pod \"monitoring-plugin-77f4f76594-d7ljw\" (UID: \"67a0e3bb-443b-4f8e-aa30-ddedc41efb83\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-d7ljw"
Nov 02 02:48:16.132879 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:16.132849    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/67a0e3bb-443b-4f8e-aa30-ddedc41efb83-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-d7ljw\" (UID: \"67a0e3bb-443b-4f8e-aa30-ddedc41efb83\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-d7ljw"
Nov 02 02:48:16.210421 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:16.210384    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-76dcdd8ff6-59rns"]
Nov 02 02:48:16.225299 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:16.225269    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-76dcdd8ff6-59rns"
Nov 02 02:48:16.233447 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:16.233381    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/67a0e3bb-443b-4f8e-aa30-ddedc41efb83-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-d7ljw\" (UID: \"67a0e3bb-443b-4f8e-aa30-ddedc41efb83\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-d7ljw"
Nov 02 02:48:16.233447 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:16.233433    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnrrh\" (UniqueName: \"kubernetes.io/projected/67a0e3bb-443b-4f8e-aa30-ddedc41efb83-kube-api-access-mnrrh\") pod \"monitoring-plugin-77f4f76594-d7ljw\" (UID: \"67a0e3bb-443b-4f8e-aa30-ddedc41efb83\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-d7ljw"
Nov 02 02:48:16.237461 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:16.237390    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/67a0e3bb-443b-4f8e-aa30-ddedc41efb83-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-d7ljw\" (UID: \"67a0e3bb-443b-4f8e-aa30-ddedc41efb83\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-d7ljw"
Nov 02 02:48:16.274607 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:16.274290    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-76dcdd8ff6-59rns"]
Nov 02 02:48:16.278812 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:16.278785    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Nov 02 02:48:16.279472 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:16.278786    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Nov 02 02:48:16.279604 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:16.278860    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Nov 02 02:48:16.279686 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:16.278983    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Nov 02 02:48:16.280003 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:16.279869    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-5stgm1run3109\""
Nov 02 02:48:16.280003 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:16.279961    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-jrtlg\""
Nov 02 02:48:16.292616 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:16.292590    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnrrh\" (UniqueName: \"kubernetes.io/projected/67a0e3bb-443b-4f8e-aa30-ddedc41efb83-kube-api-access-mnrrh\") pod \"monitoring-plugin-77f4f76594-d7ljw\" (UID: \"67a0e3bb-443b-4f8e-aa30-ddedc41efb83\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-d7ljw"
Nov 02 02:48:16.334224 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:16.334116    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/5d932f33-ed28-4494-930b-c87718b95d42-secret-metrics-server-client-certs\") pod \"metrics-server-76dcdd8ff6-59rns\" (UID: \"5d932f33-ed28-4494-930b-c87718b95d42\") " pod="openshift-monitoring/metrics-server-76dcdd8ff6-59rns"
Nov 02 02:48:16.334224 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:16.334159    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnh65\" (UniqueName: \"kubernetes.io/projected/5d932f33-ed28-4494-930b-c87718b95d42-kube-api-access-rnh65\") pod \"metrics-server-76dcdd8ff6-59rns\" (UID: \"5d932f33-ed28-4494-930b-c87718b95d42\") " pod="openshift-monitoring/metrics-server-76dcdd8ff6-59rns"
Nov 02 02:48:16.334224 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:16.334194    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/5d932f33-ed28-4494-930b-c87718b95d42-secret-metrics-server-tls\") pod \"metrics-server-76dcdd8ff6-59rns\" (UID: \"5d932f33-ed28-4494-930b-c87718b95d42\") " pod="openshift-monitoring/metrics-server-76dcdd8ff6-59rns"
Nov 02 02:48:16.334499 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:16.334307    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/5d932f33-ed28-4494-930b-c87718b95d42-audit-log\") pod \"metrics-server-76dcdd8ff6-59rns\" (UID: \"5d932f33-ed28-4494-930b-c87718b95d42\") " pod="openshift-monitoring/metrics-server-76dcdd8ff6-59rns"
Nov 02 02:48:16.334499 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:16.334350    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/5d932f33-ed28-4494-930b-c87718b95d42-client-ca-bundle\") pod \"metrics-server-76dcdd8ff6-59rns\" (UID: \"5d932f33-ed28-4494-930b-c87718b95d42\") " pod="openshift-monitoring/metrics-server-76dcdd8ff6-59rns"
Nov 02 02:48:16.334499 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:16.334408    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/5d932f33-ed28-4494-930b-c87718b95d42-metrics-server-audit-profiles\") pod \"metrics-server-76dcdd8ff6-59rns\" (UID: \"5d932f33-ed28-4494-930b-c87718b95d42\") " pod="openshift-monitoring/metrics-server-76dcdd8ff6-59rns"
Nov 02 02:48:16.334499 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:16.334461    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5d932f33-ed28-4494-930b-c87718b95d42-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-76dcdd8ff6-59rns\" (UID: \"5d932f33-ed28-4494-930b-c87718b95d42\") " pod="openshift-monitoring/metrics-server-76dcdd8ff6-59rns"
Nov 02 02:48:16.345963 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:16.345937    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-d7ljw"
Nov 02 02:48:16.437476 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:16.435788    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/5d932f33-ed28-4494-930b-c87718b95d42-secret-metrics-server-tls\") pod \"metrics-server-76dcdd8ff6-59rns\" (UID: \"5d932f33-ed28-4494-930b-c87718b95d42\") " pod="openshift-monitoring/metrics-server-76dcdd8ff6-59rns"
Nov 02 02:48:16.437476 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:16.435826    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/5d932f33-ed28-4494-930b-c87718b95d42-audit-log\") pod \"metrics-server-76dcdd8ff6-59rns\" (UID: \"5d932f33-ed28-4494-930b-c87718b95d42\") " pod="openshift-monitoring/metrics-server-76dcdd8ff6-59rns"
Nov 02 02:48:16.437476 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:16.435849    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/5d932f33-ed28-4494-930b-c87718b95d42-client-ca-bundle\") pod \"metrics-server-76dcdd8ff6-59rns\" (UID: \"5d932f33-ed28-4494-930b-c87718b95d42\") " pod="openshift-monitoring/metrics-server-76dcdd8ff6-59rns"
Nov 02 02:48:16.437476 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:16.435885    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/5d932f33-ed28-4494-930b-c87718b95d42-metrics-server-audit-profiles\") pod \"metrics-server-76dcdd8ff6-59rns\" (UID: \"5d932f33-ed28-4494-930b-c87718b95d42\") " pod="openshift-monitoring/metrics-server-76dcdd8ff6-59rns"
Nov 02 02:48:16.437476 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:16.435936    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5d932f33-ed28-4494-930b-c87718b95d42-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-76dcdd8ff6-59rns\" (UID: \"5d932f33-ed28-4494-930b-c87718b95d42\") " pod="openshift-monitoring/metrics-server-76dcdd8ff6-59rns"
Nov 02 02:48:16.437476 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:16.436005    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/5d932f33-ed28-4494-930b-c87718b95d42-secret-metrics-server-client-certs\") pod \"metrics-server-76dcdd8ff6-59rns\" (UID: \"5d932f33-ed28-4494-930b-c87718b95d42\") " pod="openshift-monitoring/metrics-server-76dcdd8ff6-59rns"
Nov 02 02:48:16.437476 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:16.436043    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnh65\" (UniqueName: \"kubernetes.io/projected/5d932f33-ed28-4494-930b-c87718b95d42-kube-api-access-rnh65\") pod \"metrics-server-76dcdd8ff6-59rns\" (UID: \"5d932f33-ed28-4494-930b-c87718b95d42\") " pod="openshift-monitoring/metrics-server-76dcdd8ff6-59rns"
Nov 02 02:48:16.437476 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:16.437262    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/5d932f33-ed28-4494-930b-c87718b95d42-metrics-server-audit-profiles\") pod \"metrics-server-76dcdd8ff6-59rns\" (UID: \"5d932f33-ed28-4494-930b-c87718b95d42\") " pod="openshift-monitoring/metrics-server-76dcdd8ff6-59rns"
Nov 02 02:48:16.438565 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:16.438403    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5d932f33-ed28-4494-930b-c87718b95d42-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-76dcdd8ff6-59rns\" (UID: \"5d932f33-ed28-4494-930b-c87718b95d42\") " pod="openshift-monitoring/metrics-server-76dcdd8ff6-59rns"
Nov 02 02:48:16.439785 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:16.439725    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/5d932f33-ed28-4494-930b-c87718b95d42-audit-log\") pod \"metrics-server-76dcdd8ff6-59rns\" (UID: \"5d932f33-ed28-4494-930b-c87718b95d42\") " pod="openshift-monitoring/metrics-server-76dcdd8ff6-59rns"
Nov 02 02:48:16.443298 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:16.443254    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/5d932f33-ed28-4494-930b-c87718b95d42-client-ca-bundle\") pod \"metrics-server-76dcdd8ff6-59rns\" (UID: \"5d932f33-ed28-4494-930b-c87718b95d42\") " pod="openshift-monitoring/metrics-server-76dcdd8ff6-59rns"
Nov 02 02:48:16.456121 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:16.456030    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/5d932f33-ed28-4494-930b-c87718b95d42-secret-metrics-server-tls\") pod \"metrics-server-76dcdd8ff6-59rns\" (UID: \"5d932f33-ed28-4494-930b-c87718b95d42\") " pod="openshift-monitoring/metrics-server-76dcdd8ff6-59rns"
Nov 02 02:48:16.460334 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:16.460300    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/5d932f33-ed28-4494-930b-c87718b95d42-secret-metrics-server-client-certs\") pod \"metrics-server-76dcdd8ff6-59rns\" (UID: \"5d932f33-ed28-4494-930b-c87718b95d42\") " pod="openshift-monitoring/metrics-server-76dcdd8ff6-59rns"
Nov 02 02:48:16.494470 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:16.494407    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnh65\" (UniqueName: \"kubernetes.io/projected/5d932f33-ed28-4494-930b-c87718b95d42-kube-api-access-rnh65\") pod \"metrics-server-76dcdd8ff6-59rns\" (UID: \"5d932f33-ed28-4494-930b-c87718b95d42\") " pod="openshift-monitoring/metrics-server-76dcdd8ff6-59rns"
Nov 02 02:48:16.554685 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:16.554386    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-76dcdd8ff6-59rns"
Nov 02 02:48:16.744646 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:16.744609    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-d7ljw"]
Nov 02 02:48:16.748845 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:48:16.748812    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67a0e3bb_443b_4f8e_aa30_ddedc41efb83.slice/crio-d9ccf67eca6342f49eb6000a532e3db6f858f1e652c2c742489d633b757242bb WatchSource:0}: Error finding container d9ccf67eca6342f49eb6000a532e3db6f858f1e652c2c742489d633b757242bb: Status 404 returned error can't find the container with id d9ccf67eca6342f49eb6000a532e3db6f858f1e652c2c742489d633b757242bb
Nov 02 02:48:16.782475 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:16.782405    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-d7ljw" event={"ID":"67a0e3bb-443b-4f8e-aa30-ddedc41efb83","Type":"ContainerStarted","Data":"d9ccf67eca6342f49eb6000a532e3db6f858f1e652c2c742489d633b757242bb"}
Nov 02 02:48:16.787995 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:16.787968    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-d2wl9" event={"ID":"55c81f55-e3b2-4006-ad91-c2a843a426ad","Type":"ContainerStarted","Data":"86d47998f9c98638759918909087a8bbd18a932f19455b9d3a35c4ea75947c75"}
Nov 02 02:48:16.788153 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:16.788034    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-d2wl9" event={"ID":"55c81f55-e3b2-4006-ad91-c2a843a426ad","Type":"ContainerStarted","Data":"9707ff8646e12a9d6b11bab4b74c2203bc38aab5fff46e395727039ea6a06afd"}
Nov 02 02:48:16.959332 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:16.959215    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-d2wl9" podStartSLOduration=2.901868167 podStartE2EDuration="5.959195517s" totalImagesPullingTime="2.985030544s" totalInitContainerRuntime="72.296806ms" isStatefulPod=true podCreationTimestamp="2026-11-02 02:48:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:48:16.908758569 +0000 UTC m=+37.487365956" watchObservedRunningTime="2026-11-02 02:48:16.959195517 +0000 UTC m=+37.537802898"
Nov 02 02:48:16.960282 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:16.960255    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-76dcdd8ff6-59rns"]
Nov 02 02:48:16.964182 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:48:16.964148    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d932f33_ed28_4494_930b_c87718b95d42.slice/crio-9b6b78d12fe2d1216dbd9d3a19854c40d34fcefdaaa4736e71b95497da8cb299 WatchSource:0}: Error finding container 9b6b78d12fe2d1216dbd9d3a19854c40d34fcefdaaa4736e71b95497da8cb299: Status 404 returned error can't find the container with id 9b6b78d12fe2d1216dbd9d3a19854c40d34fcefdaaa4736e71b95497da8cb299
Nov 02 02:48:17.793759 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:17.793715    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-76dcdd8ff6-59rns" event={"ID":"5d932f33-ed28-4494-930b-c87718b95d42","Type":"ContainerStarted","Data":"9b6b78d12fe2d1216dbd9d3a19854c40d34fcefdaaa4736e71b95497da8cb299"}
Nov 02 02:48:18.079449 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:18.079360    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6ff565b847-nzkql"]
Nov 02 02:48:18.103826 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:18.103789    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-dfc4976c6-pm8m7"]
Nov 02 02:48:18.117616 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:18.117586    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-dfc4976c6-pm8m7"
Nov 02 02:48:18.166906 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:18.165868    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-dfc4976c6-pm8m7"]
Nov 02 02:48:18.255343 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:18.255241    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/407c5630-8ce0-4678-aeda-d8527589990e-service-ca\") pod \"console-dfc4976c6-pm8m7\" (UID: \"407c5630-8ce0-4678-aeda-d8527589990e\") " pod="openshift-console/console-dfc4976c6-pm8m7"
Nov 02 02:48:18.255343 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:18.255306    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84449\" (UniqueName: \"kubernetes.io/projected/407c5630-8ce0-4678-aeda-d8527589990e-kube-api-access-84449\") pod \"console-dfc4976c6-pm8m7\" (UID: \"407c5630-8ce0-4678-aeda-d8527589990e\") " pod="openshift-console/console-dfc4976c6-pm8m7"
Nov 02 02:48:18.255591 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:18.255409    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/407c5630-8ce0-4678-aeda-d8527589990e-trusted-ca-bundle\") pod \"console-dfc4976c6-pm8m7\" (UID: \"407c5630-8ce0-4678-aeda-d8527589990e\") " pod="openshift-console/console-dfc4976c6-pm8m7"
Nov 02 02:48:18.255591 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:18.255453    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/407c5630-8ce0-4678-aeda-d8527589990e-console-oauth-config\") pod \"console-dfc4976c6-pm8m7\" (UID: \"407c5630-8ce0-4678-aeda-d8527589990e\") " pod="openshift-console/console-dfc4976c6-pm8m7"
Nov 02 02:48:18.255706 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:18.255597    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/407c5630-8ce0-4678-aeda-d8527589990e-console-config\") pod \"console-dfc4976c6-pm8m7\" (UID: \"407c5630-8ce0-4678-aeda-d8527589990e\") " pod="openshift-console/console-dfc4976c6-pm8m7"
Nov 02 02:48:18.255706 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:18.255649    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/407c5630-8ce0-4678-aeda-d8527589990e-tmp\") pod \"console-dfc4976c6-pm8m7\" (UID: \"407c5630-8ce0-4678-aeda-d8527589990e\") " pod="openshift-console/console-dfc4976c6-pm8m7"
Nov 02 02:48:18.255811 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:18.255718    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/407c5630-8ce0-4678-aeda-d8527589990e-console-serving-cert\") pod \"console-dfc4976c6-pm8m7\" (UID: \"407c5630-8ce0-4678-aeda-d8527589990e\") " pod="openshift-console/console-dfc4976c6-pm8m7"
Nov 02 02:48:18.255811 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:18.255767    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/407c5630-8ce0-4678-aeda-d8527589990e-oauth-serving-cert\") pod \"console-dfc4976c6-pm8m7\" (UID: \"407c5630-8ce0-4678-aeda-d8527589990e\") " pod="openshift-console/console-dfc4976c6-pm8m7"
Nov 02 02:48:18.358382 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:18.357269    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/407c5630-8ce0-4678-aeda-d8527589990e-console-serving-cert\") pod \"console-dfc4976c6-pm8m7\" (UID: \"407c5630-8ce0-4678-aeda-d8527589990e\") " pod="openshift-console/console-dfc4976c6-pm8m7"
Nov 02 02:48:18.358382 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:18.357329    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/407c5630-8ce0-4678-aeda-d8527589990e-oauth-serving-cert\") pod \"console-dfc4976c6-pm8m7\" (UID: \"407c5630-8ce0-4678-aeda-d8527589990e\") " pod="openshift-console/console-dfc4976c6-pm8m7"
Nov 02 02:48:18.358382 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:18.357360    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/407c5630-8ce0-4678-aeda-d8527589990e-service-ca\") pod \"console-dfc4976c6-pm8m7\" (UID: \"407c5630-8ce0-4678-aeda-d8527589990e\") " pod="openshift-console/console-dfc4976c6-pm8m7"
Nov 02 02:48:18.358382 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:18.357393    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84449\" (UniqueName: \"kubernetes.io/projected/407c5630-8ce0-4678-aeda-d8527589990e-kube-api-access-84449\") pod \"console-dfc4976c6-pm8m7\" (UID: \"407c5630-8ce0-4678-aeda-d8527589990e\") " pod="openshift-console/console-dfc4976c6-pm8m7"
Nov 02 02:48:18.358382 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:18.357415    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/407c5630-8ce0-4678-aeda-d8527589990e-trusted-ca-bundle\") pod \"console-dfc4976c6-pm8m7\" (UID: \"407c5630-8ce0-4678-aeda-d8527589990e\") " pod="openshift-console/console-dfc4976c6-pm8m7"
Nov 02 02:48:18.358382 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:18.357434    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/407c5630-8ce0-4678-aeda-d8527589990e-console-oauth-config\") pod \"console-dfc4976c6-pm8m7\" (UID: \"407c5630-8ce0-4678-aeda-d8527589990e\") " pod="openshift-console/console-dfc4976c6-pm8m7"
Nov 02 02:48:18.358382 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:18.357487    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/407c5630-8ce0-4678-aeda-d8527589990e-console-config\") pod \"console-dfc4976c6-pm8m7\" (UID: \"407c5630-8ce0-4678-aeda-d8527589990e\") " pod="openshift-console/console-dfc4976c6-pm8m7"
Nov 02 02:48:18.358382 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:18.357534    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/407c5630-8ce0-4678-aeda-d8527589990e-tmp\") pod \"console-dfc4976c6-pm8m7\" (UID: \"407c5630-8ce0-4678-aeda-d8527589990e\") " pod="openshift-console/console-dfc4976c6-pm8m7"
Nov 02 02:48:18.358382 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:18.357906    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/407c5630-8ce0-4678-aeda-d8527589990e-tmp\") pod \"console-dfc4976c6-pm8m7\" (UID: \"407c5630-8ce0-4678-aeda-d8527589990e\") " pod="openshift-console/console-dfc4976c6-pm8m7"
Nov 02 02:48:18.358889 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:18.358830    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/407c5630-8ce0-4678-aeda-d8527589990e-service-ca\") pod \"console-dfc4976c6-pm8m7\" (UID: \"407c5630-8ce0-4678-aeda-d8527589990e\") " pod="openshift-console/console-dfc4976c6-pm8m7"
Nov 02 02:48:18.358889 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:18.358830    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/407c5630-8ce0-4678-aeda-d8527589990e-oauth-serving-cert\") pod \"console-dfc4976c6-pm8m7\" (UID: \"407c5630-8ce0-4678-aeda-d8527589990e\") " pod="openshift-console/console-dfc4976c6-pm8m7"
Nov 02 02:48:18.359326 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:18.359304    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/407c5630-8ce0-4678-aeda-d8527589990e-console-config\") pod \"console-dfc4976c6-pm8m7\" (UID: \"407c5630-8ce0-4678-aeda-d8527589990e\") " pod="openshift-console/console-dfc4976c6-pm8m7"
Nov 02 02:48:18.359892 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:18.359859    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/407c5630-8ce0-4678-aeda-d8527589990e-trusted-ca-bundle\") pod \"console-dfc4976c6-pm8m7\" (UID: \"407c5630-8ce0-4678-aeda-d8527589990e\") " pod="openshift-console/console-dfc4976c6-pm8m7"
Nov 02 02:48:18.362347 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:18.362302    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/407c5630-8ce0-4678-aeda-d8527589990e-console-serving-cert\") pod \"console-dfc4976c6-pm8m7\" (UID: \"407c5630-8ce0-4678-aeda-d8527589990e\") " pod="openshift-console/console-dfc4976c6-pm8m7"
Nov 02 02:48:18.374161 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:18.374108    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/407c5630-8ce0-4678-aeda-d8527589990e-console-oauth-config\") pod \"console-dfc4976c6-pm8m7\" (UID: \"407c5630-8ce0-4678-aeda-d8527589990e\") " pod="openshift-console/console-dfc4976c6-pm8m7"
Nov 02 02:48:18.418313 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:18.418223    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84449\" (UniqueName: \"kubernetes.io/projected/407c5630-8ce0-4678-aeda-d8527589990e-kube-api-access-84449\") pod \"console-dfc4976c6-pm8m7\" (UID: \"407c5630-8ce0-4678-aeda-d8527589990e\") " pod="openshift-console/console-dfc4976c6-pm8m7"
Nov 02 02:48:18.460452 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:18.460259    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-dfc4976c6-pm8m7"
Nov 02 02:48:18.900955 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:18.900866    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-dfc4976c6-pm8m7"]
Nov 02 02:48:18.908182 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:48:18.908146    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod407c5630_8ce0_4678_aeda_d8527589990e.slice/crio-6d58e14c2f34c26b542ebcc9d1f7f8e218749efb04515e279ebf19fc475bd091 WatchSource:0}: Error finding container 6d58e14c2f34c26b542ebcc9d1f7f8e218749efb04515e279ebf19fc475bd091: Status 404 returned error can't find the container with id 6d58e14c2f34c26b542ebcc9d1f7f8e218749efb04515e279ebf19fc475bd091
Nov 02 02:48:19.804811 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:19.804769    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-dfc4976c6-pm8m7" event={"ID":"407c5630-8ce0-4678-aeda-d8527589990e","Type":"ContainerStarted","Data":"c21e5032850ef598c235437a7e42be752b27a8de9451b03406cf90c431657bd8"}
Nov 02 02:48:19.805000 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:19.804833    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-dfc4976c6-pm8m7" event={"ID":"407c5630-8ce0-4678-aeda-d8527589990e","Type":"ContainerStarted","Data":"6d58e14c2f34c26b542ebcc9d1f7f8e218749efb04515e279ebf19fc475bd091"}
Nov 02 02:48:19.807295 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:19.807258    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-fb86b55c4-sq549" event={"ID":"6ff7dc8e-86b1-4e4b-b16d-9fbac2dea1cd","Type":"ContainerStarted","Data":"1e4d437576b109d78332c46c9e5552881186ad90df2afcce3316a9b4e43416ae"}
Nov 02 02:48:19.807429 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:19.807331    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-fb86b55c4-sq549" event={"ID":"6ff7dc8e-86b1-4e4b-b16d-9fbac2dea1cd","Type":"ContainerStarted","Data":"f3d149854d1b1364194ee0a92784beccbdb618af0c0e524d9d935997381adbfd"}
Nov 02 02:48:19.807429 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:19.807373    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-fb86b55c4-sq549" event={"ID":"6ff7dc8e-86b1-4e4b-b16d-9fbac2dea1cd","Type":"ContainerStarted","Data":"547501aa645475c2817b258a10669a47f6aa4fa28459192c52da6a33f83c4f9f"}
Nov 02 02:48:20.812865 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:20.812818    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-76dcdd8ff6-59rns" event={"ID":"5d932f33-ed28-4494-930b-c87718b95d42","Type":"ContainerStarted","Data":"0924dd22a7a9d4423ae04251c146ba384fa0c609ad500835751189feaa661de5"}
Nov 02 02:48:20.815113 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:20.815082    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-d7ljw" event={"ID":"67a0e3bb-443b-4f8e-aa30-ddedc41efb83","Type":"ContainerStarted","Data":"8cc8f678f1881f7831a765beefacb4e9e7f49ab6f7986c8badfb5f17a2f9b6ba"}
Nov 02 02:48:20.815326 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:20.815304    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-d7ljw"
Nov 02 02:48:20.821024 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:20.820998    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-d7ljw"
Nov 02 02:48:20.946242 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:20.946178    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-dfc4976c6-pm8m7" podStartSLOduration=2.946160817 podStartE2EDuration="2.946160817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:48:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:48:19.92380625 +0000 UTC m=+40.502413633" watchObservedRunningTime="2026-11-02 02:48:20.946160817 +0000 UTC m=+41.524768195"
Nov 02 02:48:20.946693 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:20.946655    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-76dcdd8ff6-59rns" podStartSLOduration=1.579981396 podStartE2EDuration="4.946645043s" totalImagesPullingTime="3.366663647s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:48:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:48:20.945726693 +0000 UTC m=+41.524334086" watchObservedRunningTime="2026-11-02 02:48:20.946645043 +0000 UTC m=+41.525252423"
Nov 02 02:48:21.789849 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:21.789811    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6ff565b847-nzkql"
Nov 02 02:48:21.823211 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:21.823168    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-fb86b55c4-sq549" event={"ID":"6ff7dc8e-86b1-4e4b-b16d-9fbac2dea1cd","Type":"ContainerStarted","Data":"dc31345823163a15bc2b93f0562fd6495feee6e9e84527eb230b3c16f8f4d4bb"}
Nov 02 02:48:21.823703 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:21.823259    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-fb86b55c4-sq549" event={"ID":"6ff7dc8e-86b1-4e4b-b16d-9fbac2dea1cd","Type":"ContainerStarted","Data":"90ad13b7776aa48fb53409af028ff080e43e79b63d186cddf75a4bc3c0d517ce"}
Nov 02 02:48:21.823703 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:21.823288    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-fb86b55c4-sq549" event={"ID":"6ff7dc8e-86b1-4e4b-b16d-9fbac2dea1cd","Type":"ContainerStarted","Data":"b421ff906773fe05b1d8fe821e8506e07e06809b6f2ec99cc5ff3fdaf9060895"}
Nov 02 02:48:21.943378 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:21.943328    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-fb86b55c4-sq549" podStartSLOduration=3.592503457 podStartE2EDuration="8.943315275s" totalImagesPullingTime="5.350811818s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:48:13 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:48:21.943082927 +0000 UTC m=+42.521690307" watchObservedRunningTime="2026-11-02 02:48:21.943315275 +0000 UTC m=+42.521922656"
Nov 02 02:48:21.943718 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:21.943688    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-d7ljw" podStartSLOduration=3.372600416 podStartE2EDuration="6.943677537s" totalImagesPullingTime="3.571077121s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:48:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:48:21.063463859 +0000 UTC m=+41.642071241" watchObservedRunningTime="2026-11-02 02:48:21.943677537 +0000 UTC m=+42.522284920"
Nov 02 02:48:22.828193 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:22.828110    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-fb86b55c4-sq549"
Nov 02 02:48:25.883460 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:25.883425    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-dfc4976c6-pm8m7"]
Nov 02 02:48:25.934157 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:25.934128    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-56947894cb-st97d"]
Nov 02 02:48:25.951903 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:25.951878    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56947894cb-st97d"
Nov 02 02:48:25.994202 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:25.994176    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-56947894cb-st97d"]
Nov 02 02:48:26.041451 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:26.041423    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmzrr\" (UniqueName: \"kubernetes.io/projected/f7f8de24-eea6-41db-8d5d-a742aac0fb47-kube-api-access-rmzrr\") pod \"console-56947894cb-st97d\" (UID: \"f7f8de24-eea6-41db-8d5d-a742aac0fb47\") " pod="openshift-console/console-56947894cb-st97d"
Nov 02 02:48:26.041574 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:26.041457    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f7f8de24-eea6-41db-8d5d-a742aac0fb47-console-serving-cert\") pod \"console-56947894cb-st97d\" (UID: \"f7f8de24-eea6-41db-8d5d-a742aac0fb47\") " pod="openshift-console/console-56947894cb-st97d"
Nov 02 02:48:26.041574 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:26.041489    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f7f8de24-eea6-41db-8d5d-a742aac0fb47-console-config\") pod \"console-56947894cb-st97d\" (UID: \"f7f8de24-eea6-41db-8d5d-a742aac0fb47\") " pod="openshift-console/console-56947894cb-st97d"
Nov 02 02:48:26.041574 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:26.041555    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f7f8de24-eea6-41db-8d5d-a742aac0fb47-service-ca\") pod \"console-56947894cb-st97d\" (UID: \"f7f8de24-eea6-41db-8d5d-a742aac0fb47\") " pod="openshift-console/console-56947894cb-st97d"
Nov 02 02:48:26.041695 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:26.041675    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f7f8de24-eea6-41db-8d5d-a742aac0fb47-tmp\") pod \"console-56947894cb-st97d\" (UID: \"f7f8de24-eea6-41db-8d5d-a742aac0fb47\") " pod="openshift-console/console-56947894cb-st97d"
Nov 02 02:48:26.041730 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:26.041722    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f7f8de24-eea6-41db-8d5d-a742aac0fb47-oauth-serving-cert\") pod \"console-56947894cb-st97d\" (UID: \"f7f8de24-eea6-41db-8d5d-a742aac0fb47\") " pod="openshift-console/console-56947894cb-st97d"
Nov 02 02:48:26.041762 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:26.041741    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f7f8de24-eea6-41db-8d5d-a742aac0fb47-console-oauth-config\") pod \"console-56947894cb-st97d\" (UID: \"f7f8de24-eea6-41db-8d5d-a742aac0fb47\") " pod="openshift-console/console-56947894cb-st97d"
Nov 02 02:48:26.041796 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:26.041779    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7f8de24-eea6-41db-8d5d-a742aac0fb47-trusted-ca-bundle\") pod \"console-56947894cb-st97d\" (UID: \"f7f8de24-eea6-41db-8d5d-a742aac0fb47\") " pod="openshift-console/console-56947894cb-st97d"
Nov 02 02:48:26.143096 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:26.143030    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7f8de24-eea6-41db-8d5d-a742aac0fb47-trusted-ca-bundle\") pod \"console-56947894cb-st97d\" (UID: \"f7f8de24-eea6-41db-8d5d-a742aac0fb47\") " pod="openshift-console/console-56947894cb-st97d"
Nov 02 02:48:26.143096 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:26.143074    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmzrr\" (UniqueName: \"kubernetes.io/projected/f7f8de24-eea6-41db-8d5d-a742aac0fb47-kube-api-access-rmzrr\") pod \"console-56947894cb-st97d\" (UID: \"f7f8de24-eea6-41db-8d5d-a742aac0fb47\") " pod="openshift-console/console-56947894cb-st97d"
Nov 02 02:48:26.143268 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:26.143245    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f7f8de24-eea6-41db-8d5d-a742aac0fb47-console-serving-cert\") pod \"console-56947894cb-st97d\" (UID: \"f7f8de24-eea6-41db-8d5d-a742aac0fb47\") " pod="openshift-console/console-56947894cb-st97d"
Nov 02 02:48:26.143321 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:26.143309    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f7f8de24-eea6-41db-8d5d-a742aac0fb47-console-config\") pod \"console-56947894cb-st97d\" (UID: \"f7f8de24-eea6-41db-8d5d-a742aac0fb47\") " pod="openshift-console/console-56947894cb-st97d"
Nov 02 02:48:26.143376 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:26.143334    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f7f8de24-eea6-41db-8d5d-a742aac0fb47-service-ca\") pod \"console-56947894cb-st97d\" (UID: \"f7f8de24-eea6-41db-8d5d-a742aac0fb47\") " pod="openshift-console/console-56947894cb-st97d"
Nov 02 02:48:26.143376 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:26.143364    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f7f8de24-eea6-41db-8d5d-a742aac0fb47-tmp\") pod \"console-56947894cb-st97d\" (UID: \"f7f8de24-eea6-41db-8d5d-a742aac0fb47\") " pod="openshift-console/console-56947894cb-st97d"
Nov 02 02:48:26.143469 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:26.143395    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f7f8de24-eea6-41db-8d5d-a742aac0fb47-oauth-serving-cert\") pod \"console-56947894cb-st97d\" (UID: \"f7f8de24-eea6-41db-8d5d-a742aac0fb47\") " pod="openshift-console/console-56947894cb-st97d"
Nov 02 02:48:26.143516 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:26.143494    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f7f8de24-eea6-41db-8d5d-a742aac0fb47-console-oauth-config\") pod \"console-56947894cb-st97d\" (UID: \"f7f8de24-eea6-41db-8d5d-a742aac0fb47\") " pod="openshift-console/console-56947894cb-st97d"
Nov 02 02:48:26.143814 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:26.143776    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f7f8de24-eea6-41db-8d5d-a742aac0fb47-tmp\") pod \"console-56947894cb-st97d\" (UID: \"f7f8de24-eea6-41db-8d5d-a742aac0fb47\") " pod="openshift-console/console-56947894cb-st97d"
Nov 02 02:48:26.144067 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:26.144041    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f7f8de24-eea6-41db-8d5d-a742aac0fb47-service-ca\") pod \"console-56947894cb-st97d\" (UID: \"f7f8de24-eea6-41db-8d5d-a742aac0fb47\") " pod="openshift-console/console-56947894cb-st97d"
Nov 02 02:48:26.144193 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:26.144081    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f7f8de24-eea6-41db-8d5d-a742aac0fb47-console-config\") pod \"console-56947894cb-st97d\" (UID: \"f7f8de24-eea6-41db-8d5d-a742aac0fb47\") " pod="openshift-console/console-56947894cb-st97d"
Nov 02 02:48:26.144193 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:26.144088    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7f8de24-eea6-41db-8d5d-a742aac0fb47-trusted-ca-bundle\") pod \"console-56947894cb-st97d\" (UID: \"f7f8de24-eea6-41db-8d5d-a742aac0fb47\") " pod="openshift-console/console-56947894cb-st97d"
Nov 02 02:48:26.144193 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:26.144169    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f7f8de24-eea6-41db-8d5d-a742aac0fb47-oauth-serving-cert\") pod \"console-56947894cb-st97d\" (UID: \"f7f8de24-eea6-41db-8d5d-a742aac0fb47\") " pod="openshift-console/console-56947894cb-st97d"
Nov 02 02:48:26.146098 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:26.146075    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f7f8de24-eea6-41db-8d5d-a742aac0fb47-console-oauth-config\") pod \"console-56947894cb-st97d\" (UID: \"f7f8de24-eea6-41db-8d5d-a742aac0fb47\") " pod="openshift-console/console-56947894cb-st97d"
Nov 02 02:48:26.146154 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:26.146129    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f7f8de24-eea6-41db-8d5d-a742aac0fb47-console-serving-cert\") pod \"console-56947894cb-st97d\" (UID: \"f7f8de24-eea6-41db-8d5d-a742aac0fb47\") " pod="openshift-console/console-56947894cb-st97d"
Nov 02 02:48:26.199962 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:26.199937    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmzrr\" (UniqueName: \"kubernetes.io/projected/f7f8de24-eea6-41db-8d5d-a742aac0fb47-kube-api-access-rmzrr\") pod \"console-56947894cb-st97d\" (UID: \"f7f8de24-eea6-41db-8d5d-a742aac0fb47\") " pod="openshift-console/console-56947894cb-st97d"
Nov 02 02:48:26.278092 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:26.278064    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56947894cb-st97d"
Nov 02 02:48:26.671726 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:26.671690    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-56947894cb-st97d"]
Nov 02 02:48:26.673805 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:48:26.673780    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7f8de24_eea6_41db_8d5d_a742aac0fb47.slice/crio-8bcc83b3d07aa17ba034b0522c12398cfd231904d4b73d519a93861a8d98b9a3 WatchSource:0}: Error finding container 8bcc83b3d07aa17ba034b0522c12398cfd231904d4b73d519a93861a8d98b9a3: Status 404 returned error can't find the container with id 8bcc83b3d07aa17ba034b0522c12398cfd231904d4b73d519a93861a8d98b9a3
Nov 02 02:48:26.843267 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:26.843229    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56947894cb-st97d" event={"ID":"f7f8de24-eea6-41db-8d5d-a742aac0fb47","Type":"ContainerStarted","Data":"d5d61630f8e4d797c8251241494cf1fb7f4804f4562c0785cd09967d1a8f1be5"}
Nov 02 02:48:26.843415 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:26.843288    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56947894cb-st97d" event={"ID":"f7f8de24-eea6-41db-8d5d-a742aac0fb47","Type":"ContainerStarted","Data":"8bcc83b3d07aa17ba034b0522c12398cfd231904d4b73d519a93861a8d98b9a3"}
Nov 02 02:48:26.960909 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:26.960822    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-56947894cb-st97d" podStartSLOduration=1.960806922 podStartE2EDuration="1.960806922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:48:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:48:26.960443663 +0000 UTC m=+47.539051044" watchObservedRunningTime="2026-11-02 02:48:26.960806922 +0000 UTC m=+47.539414301"
Nov 02 02:48:28.460810 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:28.460774    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-dfc4976c6-pm8m7"
Nov 02 02:48:28.837774 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:28.837748    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-fb86b55c4-sq549"
Nov 02 02:48:33.525033 ip-10-0-133-163 kubenswrapper[4496]: E1102 02:48:33.524980    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 02:48:36.279180 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:36.279148    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-56947894cb-st97d"
Nov 02 02:48:36.279673 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:36.279205    4496 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-56947894cb-st97d"
Nov 02 02:48:36.283565 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:36.283545    4496 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-56947894cb-st97d"
Nov 02 02:48:36.555636 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:36.555554    4496 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-76dcdd8ff6-59rns"
Nov 02 02:48:36.555771 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:36.555654    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-76dcdd8ff6-59rns"
Nov 02 02:48:36.890096 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:36.890026    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-56947894cb-st97d"
Nov 02 02:48:37.060690 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:37.060659    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-88c54d969-g7zvl"]
Nov 02 02:48:43.106873 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:43.106801    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-6ff565b847-nzkql" podUID="f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1" containerName="console" containerID="cri-o://e7af03309e92c1826dfe8d3ed60747b9c25f011f063e6214c15c25b15327520f" gracePeriod=15
Nov 02 02:48:43.343792 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:43.343768    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6ff565b847-nzkql"
Nov 02 02:48:43.382170 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:43.382121    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1-trusted-ca-bundle\") pod \"f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1\" (UID: \"f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1\") "
Nov 02 02:48:43.382262 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:43.382170    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1-tmp\" (UniqueName: \"kubernetes.io/empty-dir/f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1-tmp\") pod \"f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1\" (UID: \"f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1\") "
Nov 02 02:48:43.382262 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:43.382218    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1-service-ca\" (UniqueName: \"kubernetes.io/configmap/f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1-service-ca\") pod \"f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1\" (UID: \"f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1\") "
Nov 02 02:48:43.382343 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:43.382258    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1-console-serving-cert\") pod \"f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1\" (UID: \"f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1\") "
Nov 02 02:48:43.382343 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:43.382290    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1-kube-api-access-29k72\" (UniqueName: \"kubernetes.io/projected/f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1-kube-api-access-29k72\") pod \"f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1\" (UID: \"f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1\") "
Nov 02 02:48:43.382453 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:43.382427    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1-console-oauth-config\") pod \"f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1\" (UID: \"f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1\") "
Nov 02 02:48:43.382508 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:43.382474    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1-tmp" pod "f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1" (UID: "f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:48:43.382565 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:43.382529    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1-oauth-serving-cert\") pod \"f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1\" (UID: \"f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1\") "
Nov 02 02:48:43.382618 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:43.382602    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1-console-config\" (UniqueName: \"kubernetes.io/configmap/f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1-console-config\") pod \"f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1\" (UID: \"f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1\") "
Nov 02 02:48:43.382618 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:43.382529    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1-trusted-ca-bundle" pod "f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1" (UID: "f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:48:43.382920 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:43.382841    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1-oauth-serving-cert" pod "f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1" (UID: "f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:48:43.382920 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:43.382886    4496 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1-trusted-ca-bundle\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:48:43.382920 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:43.382906    4496 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1-tmp\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:48:43.383123 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:43.382936    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1-console-config" pod "f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1" (UID: "f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:48:43.383123 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:43.382947    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1-service-ca" pod "f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1" (UID: "f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:48:43.384595 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:43.384564    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1-console-serving-cert" pod "f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1" (UID: "f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:48:43.384679 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:43.384597    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1-console-oauth-config" pod "f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1" (UID: "f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:48:43.384679 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:43.384636    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1-kube-api-access-29k72" pod "f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1" (UID: "f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1"). InnerVolumeSpecName "kube-api-access-29k72". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:48:43.484253 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:43.484223    4496 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1-console-config\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:48:43.484253 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:43.484247    4496 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1-service-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:48:43.484253 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:43.484257    4496 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1-console-serving-cert\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:48:43.484448 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:43.484267    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29k72\" (UniqueName: \"kubernetes.io/projected/f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1-kube-api-access-29k72\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:48:43.484448 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:43.484277    4496 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1-console-oauth-config\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:48:43.484448 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:43.484286    4496 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1-oauth-serving-cert\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:48:43.915851 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:43.915817    4496 generic.go:356] "Generic (PLEG): container finished" podID="f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1" containerID="e7af03309e92c1826dfe8d3ed60747b9c25f011f063e6214c15c25b15327520f" exitCode=0
Nov 02 02:48:43.916042 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:43.915863    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6ff565b847-nzkql"
Nov 02 02:48:43.916042 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:43.915863    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6ff565b847-nzkql" event={"ID":"f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1","Type":"ContainerDied","Data":"e7af03309e92c1826dfe8d3ed60747b9c25f011f063e6214c15c25b15327520f"}
Nov 02 02:48:43.916042 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:43.915986    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6ff565b847-nzkql" event={"ID":"f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1","Type":"ContainerDied","Data":"d73e5c6ad2798058bd4e36d4dfdbdd34980fbb9aee08723a2f67059640684e6e"}
Nov 02 02:48:43.916220 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:43.916047    4496 scope.go:122] "RemoveContainer" containerID="e7af03309e92c1826dfe8d3ed60747b9c25f011f063e6214c15c25b15327520f"
Nov 02 02:48:43.925538 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:43.925519    4496 scope.go:122] "RemoveContainer" containerID="e7af03309e92c1826dfe8d3ed60747b9c25f011f063e6214c15c25b15327520f"
Nov 02 02:48:43.925844 ip-10-0-133-163 kubenswrapper[4496]: E1102 02:48:43.925815    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7af03309e92c1826dfe8d3ed60747b9c25f011f063e6214c15c25b15327520f\": container with ID starting with e7af03309e92c1826dfe8d3ed60747b9c25f011f063e6214c15c25b15327520f not found: ID does not exist" containerID="e7af03309e92c1826dfe8d3ed60747b9c25f011f063e6214c15c25b15327520f"
Nov 02 02:48:43.925936 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:43.925855    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7af03309e92c1826dfe8d3ed60747b9c25f011f063e6214c15c25b15327520f"} err="failed to get container status \"e7af03309e92c1826dfe8d3ed60747b9c25f011f063e6214c15c25b15327520f\": rpc error: code = NotFound desc = could not find container \"e7af03309e92c1826dfe8d3ed60747b9c25f011f063e6214c15c25b15327520f\": container with ID starting with e7af03309e92c1826dfe8d3ed60747b9c25f011f063e6214c15c25b15327520f not found: ID does not exist"
Nov 02 02:48:44.082007 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:44.081973    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6ff565b847-nzkql"]
Nov 02 02:48:44.133058 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:44.133010    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6ff565b847-nzkql"]
Nov 02 02:48:45.571338 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:45.571309    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1" path="/var/lib/kubelet/pods/f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1/volumes"
Nov 02 02:48:50.902370 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:50.902327    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-dfc4976c6-pm8m7" podUID="407c5630-8ce0-4678-aeda-d8527589990e" containerName="console" containerID="cri-o://c21e5032850ef598c235437a7e42be752b27a8de9451b03406cf90c431657bd8" gracePeriod=15
Nov 02 02:48:51.202696 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:51.202669    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-dfc4976c6-pm8m7"
Nov 02 02:48:51.248198 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:51.248170    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/407c5630-8ce0-4678-aeda-d8527589990e-console-config\" (UniqueName: \"kubernetes.io/configmap/407c5630-8ce0-4678-aeda-d8527589990e-console-config\") pod \"407c5630-8ce0-4678-aeda-d8527589990e\" (UID: \"407c5630-8ce0-4678-aeda-d8527589990e\") "
Nov 02 02:48:51.248331 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:51.248220    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/407c5630-8ce0-4678-aeda-d8527589990e-kube-api-access-84449\" (UniqueName: \"kubernetes.io/projected/407c5630-8ce0-4678-aeda-d8527589990e-kube-api-access-84449\") pod \"407c5630-8ce0-4678-aeda-d8527589990e\" (UID: \"407c5630-8ce0-4678-aeda-d8527589990e\") "
Nov 02 02:48:51.248331 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:51.248251    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/407c5630-8ce0-4678-aeda-d8527589990e-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/407c5630-8ce0-4678-aeda-d8527589990e-console-oauth-config\") pod \"407c5630-8ce0-4678-aeda-d8527589990e\" (UID: \"407c5630-8ce0-4678-aeda-d8527589990e\") "
Nov 02 02:48:51.248331 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:51.248300    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/407c5630-8ce0-4678-aeda-d8527589990e-tmp\" (UniqueName: \"kubernetes.io/empty-dir/407c5630-8ce0-4678-aeda-d8527589990e-tmp\") pod \"407c5630-8ce0-4678-aeda-d8527589990e\" (UID: \"407c5630-8ce0-4678-aeda-d8527589990e\") "
Nov 02 02:48:51.248496 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:51.248332    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/407c5630-8ce0-4678-aeda-d8527589990e-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/407c5630-8ce0-4678-aeda-d8527589990e-console-serving-cert\") pod \"407c5630-8ce0-4678-aeda-d8527589990e\" (UID: \"407c5630-8ce0-4678-aeda-d8527589990e\") "
Nov 02 02:48:51.248496 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:51.248461    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/407c5630-8ce0-4678-aeda-d8527589990e-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/407c5630-8ce0-4678-aeda-d8527589990e-trusted-ca-bundle\") pod \"407c5630-8ce0-4678-aeda-d8527589990e\" (UID: \"407c5630-8ce0-4678-aeda-d8527589990e\") "
Nov 02 02:48:51.248595 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:51.248510    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/407c5630-8ce0-4678-aeda-d8527589990e-service-ca\" (UniqueName: \"kubernetes.io/configmap/407c5630-8ce0-4678-aeda-d8527589990e-service-ca\") pod \"407c5630-8ce0-4678-aeda-d8527589990e\" (UID: \"407c5630-8ce0-4678-aeda-d8527589990e\") "
Nov 02 02:48:51.248657 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:51.248585    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/407c5630-8ce0-4678-aeda-d8527589990e-tmp" pod "407c5630-8ce0-4678-aeda-d8527589990e" (UID: "407c5630-8ce0-4678-aeda-d8527589990e"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:48:51.248657 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:51.248604    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/407c5630-8ce0-4678-aeda-d8527589990e-console-config" pod "407c5630-8ce0-4678-aeda-d8527589990e" (UID: "407c5630-8ce0-4678-aeda-d8527589990e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:48:51.248657 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:51.248610    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/407c5630-8ce0-4678-aeda-d8527589990e-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/407c5630-8ce0-4678-aeda-d8527589990e-oauth-serving-cert\") pod \"407c5630-8ce0-4678-aeda-d8527589990e\" (UID: \"407c5630-8ce0-4678-aeda-d8527589990e\") "
Nov 02 02:48:51.248973 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:51.248927    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/407c5630-8ce0-4678-aeda-d8527589990e-service-ca" pod "407c5630-8ce0-4678-aeda-d8527589990e" (UID: "407c5630-8ce0-4678-aeda-d8527589990e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:48:51.248973 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:51.248950    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/407c5630-8ce0-4678-aeda-d8527589990e-trusted-ca-bundle" pod "407c5630-8ce0-4678-aeda-d8527589990e" (UID: "407c5630-8ce0-4678-aeda-d8527589990e"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:48:51.248973 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:51.248961    4496 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/407c5630-8ce0-4678-aeda-d8527589990e-console-config\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:48:51.249162 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:51.248970    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/407c5630-8ce0-4678-aeda-d8527589990e-oauth-serving-cert" pod "407c5630-8ce0-4678-aeda-d8527589990e" (UID: "407c5630-8ce0-4678-aeda-d8527589990e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:48:51.249162 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:51.248981    4496 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/407c5630-8ce0-4678-aeda-d8527589990e-tmp\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:48:51.250430 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:51.250408    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/407c5630-8ce0-4678-aeda-d8527589990e-console-serving-cert" pod "407c5630-8ce0-4678-aeda-d8527589990e" (UID: "407c5630-8ce0-4678-aeda-d8527589990e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:48:51.250523 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:51.250464    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/407c5630-8ce0-4678-aeda-d8527589990e-console-oauth-config" pod "407c5630-8ce0-4678-aeda-d8527589990e" (UID: "407c5630-8ce0-4678-aeda-d8527589990e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:48:51.250523 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:51.250473    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/407c5630-8ce0-4678-aeda-d8527589990e-kube-api-access-84449" pod "407c5630-8ce0-4678-aeda-d8527589990e" (UID: "407c5630-8ce0-4678-aeda-d8527589990e"). InnerVolumeSpecName "kube-api-access-84449". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:48:51.349999 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:51.349967    4496 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/407c5630-8ce0-4678-aeda-d8527589990e-oauth-serving-cert\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:48:51.349999 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:51.349993    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84449\" (UniqueName: \"kubernetes.io/projected/407c5630-8ce0-4678-aeda-d8527589990e-kube-api-access-84449\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:48:51.349999 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:51.350005    4496 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/407c5630-8ce0-4678-aeda-d8527589990e-console-oauth-config\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:48:51.350217 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:51.350037    4496 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/407c5630-8ce0-4678-aeda-d8527589990e-console-serving-cert\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:48:51.350217 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:51.350054    4496 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/407c5630-8ce0-4678-aeda-d8527589990e-trusted-ca-bundle\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:48:51.350217 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:51.350068    4496 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/407c5630-8ce0-4678-aeda-d8527589990e-service-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:48:51.947915 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:51.947839    4496 generic.go:356] "Generic (PLEG): container finished" podID="407c5630-8ce0-4678-aeda-d8527589990e" containerID="c21e5032850ef598c235437a7e42be752b27a8de9451b03406cf90c431657bd8" exitCode=0
Nov 02 02:48:51.947915 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:51.947899    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-dfc4976c6-pm8m7"
Nov 02 02:48:51.948314 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:51.947926    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-dfc4976c6-pm8m7" event={"ID":"407c5630-8ce0-4678-aeda-d8527589990e","Type":"ContainerDied","Data":"c21e5032850ef598c235437a7e42be752b27a8de9451b03406cf90c431657bd8"}
Nov 02 02:48:51.948314 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:51.947982    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-dfc4976c6-pm8m7" event={"ID":"407c5630-8ce0-4678-aeda-d8527589990e","Type":"ContainerDied","Data":"6d58e14c2f34c26b542ebcc9d1f7f8e218749efb04515e279ebf19fc475bd091"}
Nov 02 02:48:51.948314 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:51.948008    4496 scope.go:122] "RemoveContainer" containerID="c21e5032850ef598c235437a7e42be752b27a8de9451b03406cf90c431657bd8"
Nov 02 02:48:51.973249 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:51.973228    4496 scope.go:122] "RemoveContainer" containerID="c21e5032850ef598c235437a7e42be752b27a8de9451b03406cf90c431657bd8"
Nov 02 02:48:51.973566 ip-10-0-133-163 kubenswrapper[4496]: E1102 02:48:51.973542    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c21e5032850ef598c235437a7e42be752b27a8de9451b03406cf90c431657bd8\": container with ID starting with c21e5032850ef598c235437a7e42be752b27a8de9451b03406cf90c431657bd8 not found: ID does not exist" containerID="c21e5032850ef598c235437a7e42be752b27a8de9451b03406cf90c431657bd8"
Nov 02 02:48:51.973682 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:51.973572    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c21e5032850ef598c235437a7e42be752b27a8de9451b03406cf90c431657bd8"} err="failed to get container status \"c21e5032850ef598c235437a7e42be752b27a8de9451b03406cf90c431657bd8\": rpc error: code = NotFound desc = could not find container \"c21e5032850ef598c235437a7e42be752b27a8de9451b03406cf90c431657bd8\": container with ID starting with c21e5032850ef598c235437a7e42be752b27a8de9451b03406cf90c431657bd8 not found: ID does not exist"
Nov 02 02:48:52.112260 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:52.112231    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-dfc4976c6-pm8m7"]
Nov 02 02:48:52.162991 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:52.162966    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-dfc4976c6-pm8m7"]
Nov 02 02:48:53.570450 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:53.570415    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="407c5630-8ce0-4678-aeda-d8527589990e" path="/var/lib/kubelet/pods/407c5630-8ce0-4678-aeda-d8527589990e/volumes"
Nov 02 02:48:55.696257 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:55.696223    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-t9g7w-hqs6p"]
Nov 02 02:48:55.696634 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:55.696386    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-t9g7w-hqs6p" podUID="95effc61-7dc3-42c1-9609-be607c3bc65f" containerName="insights-gathering" containerID="cri-o://684b7419e6f30d5e3c07e887c1abdf27a11c764d68a8a6944a8901e7659ff795" gracePeriod=30
Nov 02 02:48:55.967735 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:55.967666    4496 generic.go:356] "Generic (PLEG): container finished" podID="95effc61-7dc3-42c1-9609-be607c3bc65f" containerID="684b7419e6f30d5e3c07e887c1abdf27a11c764d68a8a6944a8901e7659ff795" exitCode=2
Nov 02 02:48:55.967735 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:55.967724    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-t9g7w-hqs6p" event={"ID":"95effc61-7dc3-42c1-9609-be607c3bc65f","Type":"ContainerDied","Data":"684b7419e6f30d5e3c07e887c1abdf27a11c764d68a8a6944a8901e7659ff795"}
Nov 02 02:48:56.560759 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:56.560731    4496 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-76dcdd8ff6-59rns"
Nov 02 02:48:56.564493 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:48:56.564469    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-76dcdd8ff6-59rns"
Nov 02 02:49:02.080068 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:49:02.079988    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-88c54d969-g7zvl" podUID="66d05c06-04c1-4ecf-8003-4bd1b75c6c5f" containerName="console" containerID="cri-o://41eb0fac565a14deb2282f290abaffcd420cd3704ea0e62bb330c2a2332fa7e5" gracePeriod=15
Nov 02 02:49:02.302438 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:49:02.302407    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-645d98fbf4-px2w4"]
Nov 02 02:49:02.302789 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:49:02.302771    4496 cpu_manager.go:414] "containerMap: removing container" podUID="407c5630-8ce0-4678-aeda-d8527589990e" containerName="console"
Nov 02 02:49:02.302866 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:49:02.302791    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="407c5630-8ce0-4678-aeda-d8527589990e" containerName="console"
Nov 02 02:49:02.302866 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:49:02.302822    4496 cpu_manager.go:414] "containerMap: removing container" podUID="f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1" containerName="console"
Nov 02 02:49:02.302866 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:49:02.302830    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1" containerName="console"
Nov 02 02:49:02.303044 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:49:02.302918    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="f72c1a17-afce-45cb-ad70-b5a2f4ab0cd1" containerName="console"
Nov 02 02:49:02.303044 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:49:02.302936    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="407c5630-8ce0-4678-aeda-d8527589990e" containerName="console"
Nov 02 02:49:02.316168 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:49:02.316149    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-645d98fbf4-px2w4"
Nov 02 02:49:02.344449 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:49:02.344430    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-88c54d969-g7zvl"
Nov 02 02:49:02.405821 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:49:02.405783    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-645d98fbf4-px2w4"]
Nov 02 02:49:02.443797 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:49:02.443766    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66d05c06-04c1-4ecf-8003-4bd1b75c6c5f-console-config\" (UniqueName: \"kubernetes.io/configmap/66d05c06-04c1-4ecf-8003-4bd1b75c6c5f-console-config\") pod \"66d05c06-04c1-4ecf-8003-4bd1b75c6c5f\" (UID: \"66d05c06-04c1-4ecf-8003-4bd1b75c6c5f\") "
Nov 02 02:49:02.443971 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:49:02.443831    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66d05c06-04c1-4ecf-8003-4bd1b75c6c5f-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/66d05c06-04c1-4ecf-8003-4bd1b75c6c5f-console-oauth-config\") pod \"66d05c06-04c1-4ecf-8003-4bd1b75c6c5f\" (UID: \"66d05c06-04c1-4ecf-8003-4bd1b75c6c5f\") "
Nov 02 02:49:02.443971 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:49:02.443861    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66d05c06-04c1-4ecf-8003-4bd1b75c6c5f-service-ca\" (UniqueName: \"kubernetes.io/configmap/66d05c06-04c1-4ecf-8003-4bd1b75c6c5f-service-ca\") pod \"66d05c06-04c1-4ecf-8003-4bd1b75c6c5f\" (UID: \"66d05c06-04c1-4ecf-8003-4bd1b75c6c5f\") "
Nov 02 02:49:02.443971 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:49:02.443912    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66d05c06-04c1-4ecf-8003-4bd1b75c6c5f-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/66d05c06-04c1-4ecf-8003-4bd1b75c6c5f-console-serving-cert\") pod \"66d05c06-04c1-4ecf-8003-4bd1b75c6c5f\" (UID: \"66d05c06-04c1-4ecf-8003-4bd1b75c6c5f\") "
Nov 02 02:49:02.443971 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:49:02.443961    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66d05c06-04c1-4ecf-8003-4bd1b75c6c5f-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/66d05c06-04c1-4ecf-8003-4bd1b75c6c5f-oauth-serving-cert\") pod \"66d05c06-04c1-4ecf-8003-4bd1b75c6c5f\" (UID: \"66d05c06-04c1-4ecf-8003-4bd1b75c6c5f\") "
Nov 02 02:49:02.444221 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:49:02.443992    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66d05c06-04c1-4ecf-8003-4bd1b75c6c5f-kube-api-access-6qx2s\" (UniqueName: \"kubernetes.io/projected/66d05c06-04c1-4ecf-8003-4bd1b75c6c5f-kube-api-access-6qx2s\") pod \"66d05c06-04c1-4ecf-8003-4bd1b75c6c5f\" (UID: \"66d05c06-04c1-4ecf-8003-4bd1b75c6c5f\") "
Nov 02 02:49:02.444221 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:49:02.444074    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66d05c06-04c1-4ecf-8003-4bd1b75c6c5f-tmp\" (UniqueName: \"kubernetes.io/empty-dir/66d05c06-04c1-4ecf-8003-4bd1b75c6c5f-tmp\") pod \"66d05c06-04c1-4ecf-8003-4bd1b75c6c5f\" (UID: \"66d05c06-04c1-4ecf-8003-4bd1b75c6c5f\") "
Nov 02 02:49:02.444221 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:49:02.444197    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6b8c2ed4-e39f-420e-b613-dc1b6cde651f-oauth-serving-cert\") pod \"console-645d98fbf4-px2w4\" (UID: \"6b8c2ed4-e39f-420e-b613-dc1b6cde651f\") " pod="openshift-console/console-645d98fbf4-px2w4"
Nov 02 02:49:02.444376 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:49:02.444244    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6b8c2ed4-e39f-420e-b613-dc1b6cde651f-service-ca\") pod \"console-645d98fbf4-px2w4\" (UID: \"6b8c2ed4-e39f-420e-b613-dc1b6cde651f\") " pod="openshift-console/console-645d98fbf4-px2w4"
Nov 02 02:49:02.444376 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:49:02.444311    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6b8c2ed4-e39f-420e-b613-dc1b6cde651f-tmp\") pod \"console-645d98fbf4-px2w4\" (UID: \"6b8c2ed4-e39f-420e-b613-dc1b6cde651f\") " pod="openshift-console/console-645d98fbf4-px2w4"
Nov 02 02:49:02.444479 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:49:02.444375    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66d05c06-04c1-4ecf-8003-4bd1b75c6c5f-console-config" pod "66d05c06-04c1-4ecf-8003-4bd1b75c6c5f" (UID: "66d05c06-04c1-4ecf-8003-4bd1b75c6c5f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:49:02.444479 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:49:02.444452    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6b8c2ed4-e39f-420e-b613-dc1b6cde651f-console-serving-cert\") pod \"console-645d98fbf4-px2w4\" (UID: \"6b8c2ed4-e39f-420e-b613-dc1b6cde651f\") " pod="openshift-console/console-645d98fbf4-px2w4"
Nov 02 02:49:02.444577 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:49:02.444487    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6b8c2ed4-e39f-420e-b613-dc1b6cde651f-console-config\") pod \"console-645d98fbf4-px2w4\" (UID: \"6b8c2ed4-e39f-420e-b613-dc1b6cde651f\") " pod="openshift-console/console-645d98fbf4-px2w4"
Nov 02 02:49:02.444577 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:49:02.444512    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2vd5\" (UniqueName: \"kubernetes.io/projected/6b8c2ed4-e39f-420e-b613-dc1b6cde651f-kube-api-access-v2vd5\") pod \"console-645d98fbf4-px2w4\" (UID: \"6b8c2ed4-e39f-420e-b613-dc1b6cde651f\") " pod="openshift-console/console-645d98fbf4-px2w4"
Nov 02 02:49:02.444577 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:49:02.444526    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66d05c06-04c1-4ecf-8003-4bd1b75c6c5f-oauth-serving-cert" pod "66d05c06-04c1-4ecf-8003-4bd1b75c6c5f" (UID: "66d05c06-04c1-4ecf-8003-4bd1b75c6c5f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:49:02.444577 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:49:02.444558    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66d05c06-04c1-4ecf-8003-4bd1b75c6c5f-service-ca" pod "66d05c06-04c1-4ecf-8003-4bd1b75c6c5f" (UID: "66d05c06-04c1-4ecf-8003-4bd1b75c6c5f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:49:02.444838 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:49:02.444657    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6b8c2ed4-e39f-420e-b613-dc1b6cde651f-console-oauth-config\") pod \"console-645d98fbf4-px2w4\" (UID: \"6b8c2ed4-e39f-420e-b613-dc1b6cde651f\") " pod="openshift-console/console-645d98fbf4-px2w4"
Nov 02 02:49:02.444838 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:49:02.444716    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b8c2ed4-e39f-420e-b613-dc1b6cde651f-trusted-ca-bundle\") pod \"console-645d98fbf4-px2w4\" (UID: \"6b8c2ed4-e39f-420e-b613-dc1b6cde651f\") " pod="openshift-console/console-645d98fbf4-px2w4"
Nov 02 02:49:02.444838 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:49:02.444816    4496 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/66d05c06-04c1-4ecf-8003-4bd1b75c6c5f-service-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:49:02.444838 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:49:02.444835    4496 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/66d05c06-04c1-4ecf-8003-4bd1b75c6c5f-oauth-serving-cert\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:49:02.445114 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:49:02.444853    4496 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/66d05c06-04c1-4ecf-8003-4bd1b75c6c5f-console-config\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:49:02.445114 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:49:02.444505    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66d05c06-04c1-4ecf-8003-4bd1b75c6c5f-tmp" pod "66d05c06-04c1-4ecf-8003-4bd1b75c6c5f" (UID: "66d05c06-04c1-4ecf-8003-4bd1b75c6c5f"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:49:02.446176 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:49:02.446142    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66d05c06-04c1-4ecf-8003-4bd1b75c6c5f-kube-api-access-6qx2s" pod "66d05c06-04c1-4ecf-8003-4bd1b75c6c5f" (UID: "66d05c06-04c1-4ecf-8003-4bd1b75c6c5f"). InnerVolumeSpecName "kube-api-access-6qx2s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:49:02.446281 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:49:02.446258    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66d05c06-04c1-4ecf-8003-4bd1b75c6c5f-console-oauth-config" pod "66d05c06-04c1-4ecf-8003-4bd1b75c6c5f" (UID: "66d05c06-04c1-4ecf-8003-4bd1b75c6c5f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:49:02.446281 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:49:02.446273    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66d05c06-04c1-4ecf-8003-4bd1b75c6c5f-console-serving-cert" pod "66d05c06-04c1-4ecf-8003-4bd1b75c6c5f" (UID: "66d05c06-04c1-4ecf-8003-4bd1b75c6c5f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:49:02.545413 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:49:02.545362    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6b8c2ed4-e39f-420e-b613-dc1b6cde651f-oauth-serving-cert\") pod \"console-645d98fbf4-px2w4\" (UID: \"6b8c2ed4-e39f-420e-b613-dc1b6cde651f\") " pod="openshift-console/console-645d98fbf4-px2w4"
Nov 02 02:49:02.545611 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:49:02.545436    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6b8c2ed4-e39f-420e-b613-dc1b6cde651f-service-ca\") pod \"console-645d98fbf4-px2w4\" (UID: \"6b8c2ed4-e39f-420e-b613-dc1b6cde651f\") " pod="openshift-console/console-645d98fbf4-px2w4"
Nov 02 02:49:02.545611 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:49:02.545496    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6b8c2ed4-e39f-420e-b613-dc1b6cde651f-tmp\") pod \"console-645d98fbf4-px2w4\" (UID: \"6b8c2ed4-e39f-420e-b613-dc1b6cde651f\") " pod="openshift-console/console-645d98fbf4-px2w4"
Nov 02 02:49:02.545611 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:49:02.545527    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6b8c2ed4-e39f-420e-b613-dc1b6cde651f-console-serving-cert\") pod \"console-645d98fbf4-px2w4\" (UID: \"6b8c2ed4-e39f-420e-b613-dc1b6cde651f\") " pod="openshift-console/console-645d98fbf4-px2w4"
Nov 02 02:49:02.545834 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:49:02.545694    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6b8c2ed4-e39f-420e-b613-dc1b6cde651f-console-config\") pod \"console-645d98fbf4-px2w4\" (UID: \"6b8c2ed4-e39f-420e-b613-dc1b6cde651f\") " pod="openshift-console/console-645d98fbf4-px2w4"
Nov 02 02:49:02.545834 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:49:02.545747    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2vd5\" (UniqueName: \"kubernetes.io/projected/6b8c2ed4-e39f-420e-b613-dc1b6cde651f-kube-api-access-v2vd5\") pod \"console-645d98fbf4-px2w4\" (UID: \"6b8c2ed4-e39f-420e-b613-dc1b6cde651f\") " pod="openshift-console/console-645d98fbf4-px2w4"
Nov 02 02:49:02.545834 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:49:02.545795    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6b8c2ed4-e39f-420e-b613-dc1b6cde651f-console-oauth-config\") pod \"console-645d98fbf4-px2w4\" (UID: \"6b8c2ed4-e39f-420e-b613-dc1b6cde651f\") " pod="openshift-console/console-645d98fbf4-px2w4"
Nov 02 02:49:02.545989 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:49:02.545850    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b8c2ed4-e39f-420e-b613-dc1b6cde651f-trusted-ca-bundle\") pod \"console-645d98fbf4-px2w4\" (UID: \"6b8c2ed4-e39f-420e-b613-dc1b6cde651f\") " pod="openshift-console/console-645d98fbf4-px2w4"
Nov 02 02:49:02.545989 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:49:02.545901    4496 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/66d05c06-04c1-4ecf-8003-4bd1b75c6c5f-console-oauth-config\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:49:02.545989 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:49:02.545915    4496 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/66d05c06-04c1-4ecf-8003-4bd1b75c6c5f-console-serving-cert\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:49:02.545989 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:49:02.545938    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qx2s\" (UniqueName: \"kubernetes.io/projected/66d05c06-04c1-4ecf-8003-4bd1b75c6c5f-kube-api-access-6qx2s\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:49:02.545989 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:49:02.545952    4496 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/66d05c06-04c1-4ecf-8003-4bd1b75c6c5f-tmp\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:49:02.546478 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:49:02.546377    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6b8c2ed4-e39f-420e-b613-dc1b6cde651f-tmp\") pod \"console-645d98fbf4-px2w4\" (UID: \"6b8c2ed4-e39f-420e-b613-dc1b6cde651f\") " pod="openshift-console/console-645d98fbf4-px2w4"
Nov 02 02:49:02.547047 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:49:02.546603    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6b8c2ed4-e39f-420e-b613-dc1b6cde651f-console-config\") pod \"console-645d98fbf4-px2w4\" (UID: \"6b8c2ed4-e39f-420e-b613-dc1b6cde651f\") " pod="openshift-console/console-645d98fbf4-px2w4"
Nov 02 02:49:02.547047 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:49:02.546772    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6b8c2ed4-e39f-420e-b613-dc1b6cde651f-oauth-serving-cert\") pod \"console-645d98fbf4-px2w4\" (UID: \"6b8c2ed4-e39f-420e-b613-dc1b6cde651f\") " pod="openshift-console/console-645d98fbf4-px2w4"
Nov 02 02:49:02.547203 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:49:02.547043    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6b8c2ed4-e39f-420e-b613-dc1b6cde651f-service-ca\") pod \"console-645d98fbf4-px2w4\" (UID: \"6b8c2ed4-e39f-420e-b613-dc1b6cde651f\") " pod="openshift-console/console-645d98fbf4-px2w4"
Nov 02 02:49:02.547625 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:49:02.547428    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b8c2ed4-e39f-420e-b613-dc1b6cde651f-trusted-ca-bundle\") pod \"console-645d98fbf4-px2w4\" (UID: \"6b8c2ed4-e39f-420e-b613-dc1b6cde651f\") " pod="openshift-console/console-645d98fbf4-px2w4"
Nov 02 02:49:02.548696 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:49:02.548677    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6b8c2ed4-e39f-420e-b613-dc1b6cde651f-console-serving-cert\") pod \"console-645d98fbf4-px2w4\" (UID: \"6b8c2ed4-e39f-420e-b613-dc1b6cde651f\") " pod="openshift-console/console-645d98fbf4-px2w4"
Nov 02 02:49:02.549346 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:49:02.549328    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6b8c2ed4-e39f-420e-b613-dc1b6cde651f-console-oauth-config\") pod \"console-645d98fbf4-px2w4\" (UID: \"6b8c2ed4-e39f-420e-b613-dc1b6cde651f\") " pod="openshift-console/console-645d98fbf4-px2w4"
Nov 02 02:49:02.603806 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:49:02.603738    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2vd5\" (UniqueName: \"kubernetes.io/projected/6b8c2ed4-e39f-420e-b613-dc1b6cde651f-kube-api-access-v2vd5\") pod \"console-645d98fbf4-px2w4\" (UID: \"6b8c2ed4-e39f-420e-b613-dc1b6cde651f\") " pod="openshift-console/console-645d98fbf4-px2w4"
Nov 02 02:49:02.649695 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:49:02.649666    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-645d98fbf4-px2w4"
Nov 02 02:49:03.001645 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:49:03.001603    4496 generic.go:356] "Generic (PLEG): container finished" podID="66d05c06-04c1-4ecf-8003-4bd1b75c6c5f" containerID="41eb0fac565a14deb2282f290abaffcd420cd3704ea0e62bb330c2a2332fa7e5" exitCode=0
Nov 02 02:49:03.001848 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:49:03.001652    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-88c54d969-g7zvl" event={"ID":"66d05c06-04c1-4ecf-8003-4bd1b75c6c5f","Type":"ContainerDied","Data":"41eb0fac565a14deb2282f290abaffcd420cd3704ea0e62bb330c2a2332fa7e5"}
Nov 02 02:49:03.001848 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:49:03.001697    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-88c54d969-g7zvl" event={"ID":"66d05c06-04c1-4ecf-8003-4bd1b75c6c5f","Type":"ContainerDied","Data":"b1bd8c8d6d3e355458f62c9aa8694abb7271563a6bdf1629fe4c4bc55e99aca7"}
Nov 02 02:49:03.001848 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:49:03.001721    4496 scope.go:122] "RemoveContainer" containerID="41eb0fac565a14deb2282f290abaffcd420cd3704ea0e62bb330c2a2332fa7e5"
Nov 02 02:49:03.001848 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:49:03.001654    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-88c54d969-g7zvl"
Nov 02 02:49:03.010524 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:49:03.010508    4496 scope.go:122] "RemoveContainer" containerID="41eb0fac565a14deb2282f290abaffcd420cd3704ea0e62bb330c2a2332fa7e5"
Nov 02 02:49:03.010777 ip-10-0-133-163 kubenswrapper[4496]: E1102 02:49:03.010761    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41eb0fac565a14deb2282f290abaffcd420cd3704ea0e62bb330c2a2332fa7e5\": container with ID starting with 41eb0fac565a14deb2282f290abaffcd420cd3704ea0e62bb330c2a2332fa7e5 not found: ID does not exist" containerID="41eb0fac565a14deb2282f290abaffcd420cd3704ea0e62bb330c2a2332fa7e5"
Nov 02 02:49:03.010825 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:49:03.010788    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41eb0fac565a14deb2282f290abaffcd420cd3704ea0e62bb330c2a2332fa7e5"} err="failed to get container status \"41eb0fac565a14deb2282f290abaffcd420cd3704ea0e62bb330c2a2332fa7e5\": rpc error: code = NotFound desc = could not find container \"41eb0fac565a14deb2282f290abaffcd420cd3704ea0e62bb330c2a2332fa7e5\": container with ID starting with 41eb0fac565a14deb2282f290abaffcd420cd3704ea0e62bb330c2a2332fa7e5 not found: ID does not exist"
Nov 02 02:49:03.036539 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:49:03.036478    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-645d98fbf4-px2w4"]
Nov 02 02:49:03.038838 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:49:03.038811    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b8c2ed4_e39f_420e_b613_dc1b6cde651f.slice/crio-776d396e3cc011bea73e324f3b8a5c1152cdcdaffa66fd31bb20186d5919ae53 WatchSource:0}: Error finding container 776d396e3cc011bea73e324f3b8a5c1152cdcdaffa66fd31bb20186d5919ae53: Status 404 returned error can't find the container with id 776d396e3cc011bea73e324f3b8a5c1152cdcdaffa66fd31bb20186d5919ae53
Nov 02 02:49:03.168319 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:49:03.168284    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-88c54d969-g7zvl"]
Nov 02 02:49:03.169577 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:49:03.169552    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-88c54d969-g7zvl"]
Nov 02 02:49:03.525876 ip-10-0-133-163 kubenswrapper[4496]: E1102 02:49:03.525834    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 02:49:03.570996 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:49:03.570971    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66d05c06-04c1-4ecf-8003-4bd1b75c6c5f" path="/var/lib/kubelet/pods/66d05c06-04c1-4ecf-8003-4bd1b75c6c5f/volumes"
Nov 02 02:49:04.007566 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:49:04.007532    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-645d98fbf4-px2w4" event={"ID":"6b8c2ed4-e39f-420e-b613-dc1b6cde651f","Type":"ContainerStarted","Data":"e7bd96b2c0884b1f243f6aec0604fff4fa14f2300b3c23ce93d41851f06b2ace"}
Nov 02 02:49:04.007731 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:49:04.007582    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-645d98fbf4-px2w4" event={"ID":"6b8c2ed4-e39f-420e-b613-dc1b6cde651f","Type":"ContainerStarted","Data":"776d396e3cc011bea73e324f3b8a5c1152cdcdaffa66fd31bb20186d5919ae53"}
Nov 02 02:49:04.124958 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:49:04.124914    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-645d98fbf4-px2w4" podStartSLOduration=2.124900477 podStartE2EDuration="2.124900477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:49:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:49:04.124862202 +0000 UTC m=+84.703469584" watchObservedRunningTime="2026-11-02 02:49:04.124900477 +0000 UTC m=+84.703507854"
Nov 02 02:49:12.650807 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:49:12.650774    4496 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-645d98fbf4-px2w4"
Nov 02 02:49:12.651299 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:49:12.650847    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-645d98fbf4-px2w4"
Nov 02 02:49:12.656836 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:49:12.656812    4496 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-645d98fbf4-px2w4"
Nov 02 02:49:13.047886 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:49:13.047856    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-645d98fbf4-px2w4"
Nov 02 02:49:13.216947 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:49:13.216910    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-56947894cb-st97d"]
Nov 02 02:49:33.523955 ip-10-0-133-163 kubenswrapper[4496]: E1102 02:49:33.523923    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 02:49:38.236341 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:49:38.236295    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-56947894cb-st97d" podUID="f7f8de24-eea6-41db-8d5d-a742aac0fb47" containerName="console" containerID="cri-o://d5d61630f8e4d797c8251241494cf1fb7f4804f4562c0785cd09967d1a8f1be5" gracePeriod=15
Nov 02 02:49:38.500917 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:49:38.500893    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56947894cb-st97d"
Nov 02 02:49:38.515732 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:49:38.515711    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7f8de24-eea6-41db-8d5d-a742aac0fb47-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7f8de24-eea6-41db-8d5d-a742aac0fb47-trusted-ca-bundle\") pod \"f7f8de24-eea6-41db-8d5d-a742aac0fb47\" (UID: \"f7f8de24-eea6-41db-8d5d-a742aac0fb47\") "
Nov 02 02:49:38.515819 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:49:38.515757    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7f8de24-eea6-41db-8d5d-a742aac0fb47-kube-api-access-rmzrr\" (UniqueName: \"kubernetes.io/projected/f7f8de24-eea6-41db-8d5d-a742aac0fb47-kube-api-access-rmzrr\") pod \"f7f8de24-eea6-41db-8d5d-a742aac0fb47\" (UID: \"f7f8de24-eea6-41db-8d5d-a742aac0fb47\") "
Nov 02 02:49:38.515819 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:49:38.515787    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7f8de24-eea6-41db-8d5d-a742aac0fb47-console-config\" (UniqueName: \"kubernetes.io/configmap/f7f8de24-eea6-41db-8d5d-a742aac0fb47-console-config\") pod \"f7f8de24-eea6-41db-8d5d-a742aac0fb47\" (UID: \"f7f8de24-eea6-41db-8d5d-a742aac0fb47\") "
Nov 02 02:49:38.515903 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:49:38.515817    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7f8de24-eea6-41db-8d5d-a742aac0fb47-tmp\" (UniqueName: \"kubernetes.io/empty-dir/f7f8de24-eea6-41db-8d5d-a742aac0fb47-tmp\") pod \"f7f8de24-eea6-41db-8d5d-a742aac0fb47\" (UID: \"f7f8de24-eea6-41db-8d5d-a742aac0fb47\") "
Nov 02 02:49:38.515903 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:49:38.515838    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7f8de24-eea6-41db-8d5d-a742aac0fb47-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f7f8de24-eea6-41db-8d5d-a742aac0fb47-console-oauth-config\") pod \"f7f8de24-eea6-41db-8d5d-a742aac0fb47\" (UID: \"f7f8de24-eea6-41db-8d5d-a742aac0fb47\") "
Nov 02 02:49:38.515903 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:49:38.515888    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7f8de24-eea6-41db-8d5d-a742aac0fb47-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f7f8de24-eea6-41db-8d5d-a742aac0fb47-console-serving-cert\") pod \"f7f8de24-eea6-41db-8d5d-a742aac0fb47\" (UID: \"f7f8de24-eea6-41db-8d5d-a742aac0fb47\") "
Nov 02 02:49:38.516061 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:49:38.515917    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7f8de24-eea6-41db-8d5d-a742aac0fb47-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f7f8de24-eea6-41db-8d5d-a742aac0fb47-oauth-serving-cert\") pod \"f7f8de24-eea6-41db-8d5d-a742aac0fb47\" (UID: \"f7f8de24-eea6-41db-8d5d-a742aac0fb47\") "
Nov 02 02:49:38.516061 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:49:38.515979    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7f8de24-eea6-41db-8d5d-a742aac0fb47-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7f8de24-eea6-41db-8d5d-a742aac0fb47-service-ca\") pod \"f7f8de24-eea6-41db-8d5d-a742aac0fb47\" (UID: \"f7f8de24-eea6-41db-8d5d-a742aac0fb47\") "
Nov 02 02:49:38.516149 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:49:38.516080    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7f8de24-eea6-41db-8d5d-a742aac0fb47-tmp" pod "f7f8de24-eea6-41db-8d5d-a742aac0fb47" (UID: "f7f8de24-eea6-41db-8d5d-a742aac0fb47"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:49:38.516149 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:49:38.516102    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7f8de24-eea6-41db-8d5d-a742aac0fb47-trusted-ca-bundle" pod "f7f8de24-eea6-41db-8d5d-a742aac0fb47" (UID: "f7f8de24-eea6-41db-8d5d-a742aac0fb47"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:49:38.516264 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:49:38.516247    4496 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f7f8de24-eea6-41db-8d5d-a742aac0fb47-tmp\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:49:38.516323 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:49:38.516272    4496 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7f8de24-eea6-41db-8d5d-a742aac0fb47-trusted-ca-bundle\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:49:38.516428 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:49:38.516404    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7f8de24-eea6-41db-8d5d-a742aac0fb47-console-config" pod "f7f8de24-eea6-41db-8d5d-a742aac0fb47" (UID: "f7f8de24-eea6-41db-8d5d-a742aac0fb47"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:49:38.516482 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:49:38.516411    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7f8de24-eea6-41db-8d5d-a742aac0fb47-service-ca" pod "f7f8de24-eea6-41db-8d5d-a742aac0fb47" (UID: "f7f8de24-eea6-41db-8d5d-a742aac0fb47"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:49:38.516482 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:49:38.516445    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7f8de24-eea6-41db-8d5d-a742aac0fb47-oauth-serving-cert" pod "f7f8de24-eea6-41db-8d5d-a742aac0fb47" (UID: "f7f8de24-eea6-41db-8d5d-a742aac0fb47"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:49:38.518234 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:49:38.518204    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7f8de24-eea6-41db-8d5d-a742aac0fb47-console-serving-cert" pod "f7f8de24-eea6-41db-8d5d-a742aac0fb47" (UID: "f7f8de24-eea6-41db-8d5d-a742aac0fb47"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:49:38.518333 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:49:38.518249    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7f8de24-eea6-41db-8d5d-a742aac0fb47-console-oauth-config" pod "f7f8de24-eea6-41db-8d5d-a742aac0fb47" (UID: "f7f8de24-eea6-41db-8d5d-a742aac0fb47"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:49:38.518333 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:49:38.518269    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7f8de24-eea6-41db-8d5d-a742aac0fb47-kube-api-access-rmzrr" pod "f7f8de24-eea6-41db-8d5d-a742aac0fb47" (UID: "f7f8de24-eea6-41db-8d5d-a742aac0fb47"). InnerVolumeSpecName "kube-api-access-rmzrr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:49:38.616666 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:49:38.616629    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmzrr\" (UniqueName: \"kubernetes.io/projected/f7f8de24-eea6-41db-8d5d-a742aac0fb47-kube-api-access-rmzrr\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:49:38.616666 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:49:38.616663    4496 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f7f8de24-eea6-41db-8d5d-a742aac0fb47-console-config\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:49:38.616666 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:49:38.616673    4496 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f7f8de24-eea6-41db-8d5d-a742aac0fb47-console-oauth-config\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:49:38.617106 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:49:38.616682    4496 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f7f8de24-eea6-41db-8d5d-a742aac0fb47-console-serving-cert\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:49:38.617106 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:49:38.616692    4496 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f7f8de24-eea6-41db-8d5d-a742aac0fb47-oauth-serving-cert\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:49:38.617106 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:49:38.616701    4496 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f7f8de24-eea6-41db-8d5d-a742aac0fb47-service-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:49:39.141901 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:49:39.141866    4496 generic.go:356] "Generic (PLEG): container finished" podID="f7f8de24-eea6-41db-8d5d-a742aac0fb47" containerID="d5d61630f8e4d797c8251241494cf1fb7f4804f4562c0785cd09967d1a8f1be5" exitCode=0
Nov 02 02:49:39.142097 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:49:39.141929    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56947894cb-st97d"
Nov 02 02:49:39.142097 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:49:39.141930    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56947894cb-st97d" event={"ID":"f7f8de24-eea6-41db-8d5d-a742aac0fb47","Type":"ContainerDied","Data":"d5d61630f8e4d797c8251241494cf1fb7f4804f4562c0785cd09967d1a8f1be5"}
Nov 02 02:49:39.142097 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:49:39.142050    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56947894cb-st97d" event={"ID":"f7f8de24-eea6-41db-8d5d-a742aac0fb47","Type":"ContainerDied","Data":"8bcc83b3d07aa17ba034b0522c12398cfd231904d4b73d519a93861a8d98b9a3"}
Nov 02 02:49:39.142097 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:49:39.142076    4496 scope.go:122] "RemoveContainer" containerID="d5d61630f8e4d797c8251241494cf1fb7f4804f4562c0785cd09967d1a8f1be5"
Nov 02 02:49:39.151802 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:49:39.151776    4496 scope.go:122] "RemoveContainer" containerID="d5d61630f8e4d797c8251241494cf1fb7f4804f4562c0785cd09967d1a8f1be5"
Nov 02 02:49:39.152229 ip-10-0-133-163 kubenswrapper[4496]: E1102 02:49:39.152122    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5d61630f8e4d797c8251241494cf1fb7f4804f4562c0785cd09967d1a8f1be5\": container with ID starting with d5d61630f8e4d797c8251241494cf1fb7f4804f4562c0785cd09967d1a8f1be5 not found: ID does not exist" containerID="d5d61630f8e4d797c8251241494cf1fb7f4804f4562c0785cd09967d1a8f1be5"
Nov 02 02:49:39.152229 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:49:39.152161    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5d61630f8e4d797c8251241494cf1fb7f4804f4562c0785cd09967d1a8f1be5"} err="failed to get container status \"d5d61630f8e4d797c8251241494cf1fb7f4804f4562c0785cd09967d1a8f1be5\": rpc error: code = NotFound desc = could not find container \"d5d61630f8e4d797c8251241494cf1fb7f4804f4562c0785cd09967d1a8f1be5\": container with ID starting with d5d61630f8e4d797c8251241494cf1fb7f4804f4562c0785cd09967d1a8f1be5 not found: ID does not exist"
Nov 02 02:49:39.307423 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:49:39.307390    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-56947894cb-st97d"]
Nov 02 02:49:39.358031 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:49:39.357989    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-56947894cb-st97d"]
Nov 02 02:49:39.574063 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:49:39.574033    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7f8de24-eea6-41db-8d5d-a742aac0fb47" path="/var/lib/kubelet/pods/f7f8de24-eea6-41db-8d5d-a742aac0fb47/volumes"
Nov 02 02:50:03.524465 ip-10-0-133-163 kubenswrapper[4496]: E1102 02:50:03.524420    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 02:50:33.524430 ip-10-0-133-163 kubenswrapper[4496]: E1102 02:50:33.524397    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 02:50:55.839646 ip-10-0-133-163 kubenswrapper[4496]: E1102 02:50:55.839596    4496 log.go:32] "StopPodSandbox from runtime service failed" err="rpc error: code = DeadlineExceeded desc = context deadline exceeded" podSandboxID="d960a2b9fc534555ae1a4989812606630fdb546f04750fe5c7a6129dd61abe4d"
Nov 02 02:50:55.840083 ip-10-0-133-163 kubenswrapper[4496]: E1102 02:50:55.839655    4496 kuberuntime_manager.go:1939] "Failed to stop sandbox" podSandboxID={"Type":"cri-o","ID":"d960a2b9fc534555ae1a4989812606630fdb546f04750fe5c7a6129dd61abe4d"}
Nov 02 02:50:55.840083 ip-10-0-133-163 kubenswrapper[4496]: E1102 02:50:55.839736    4496 kubelet.go:2243] "Unhandled Error" err="failed to \"KillPodSandbox\" for \"95effc61-7dc3-42c1-9609-be607c3bc65f\" with KillPodSandboxError: \"rpc error: code = DeadlineExceeded desc = context deadline exceeded\"" logger="UnhandledError"
Nov 02 02:50:55.840083 ip-10-0-133-163 kubenswrapper[4496]: E1102 02:50:55.839752    4496 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"KillPodSandbox\" for \"95effc61-7dc3-42c1-9609-be607c3bc65f\" with KillPodSandboxError: \"rpc error: code = DeadlineExceeded desc = context deadline exceeded\"" pod="openshift-insights/periodic-gathering-t9g7w-hqs6p" podUID="95effc61-7dc3-42c1-9609-be607c3bc65f"
Nov 02 02:50:57.743307 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:50:57.743278    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-t9g7w-hqs6p"
Nov 02 02:50:57.809114 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:50:57.809083    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95effc61-7dc3-42c1-9609-be607c3bc65f-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95effc61-7dc3-42c1-9609-be607c3bc65f-trusted-ca-bundle\") pod \"95effc61-7dc3-42c1-9609-be607c3bc65f\" (UID: \"95effc61-7dc3-42c1-9609-be607c3bc65f\") "
Nov 02 02:50:57.809244 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:50:57.809152    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95effc61-7dc3-42c1-9609-be607c3bc65f-kube-api-access-2dl7x\" (UniqueName: \"kubernetes.io/projected/95effc61-7dc3-42c1-9609-be607c3bc65f-kube-api-access-2dl7x\") pod \"95effc61-7dc3-42c1-9609-be607c3bc65f\" (UID: \"95effc61-7dc3-42c1-9609-be607c3bc65f\") "
Nov 02 02:50:57.809244 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:50:57.809189    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95effc61-7dc3-42c1-9609-be607c3bc65f-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/95effc61-7dc3-42c1-9609-be607c3bc65f-archives-path\") pod \"95effc61-7dc3-42c1-9609-be607c3bc65f\" (UID: \"95effc61-7dc3-42c1-9609-be607c3bc65f\") "
Nov 02 02:50:57.809244 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:50:57.809215    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95effc61-7dc3-42c1-9609-be607c3bc65f-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95effc61-7dc3-42c1-9609-be607c3bc65f-service-ca-bundle\") pod \"95effc61-7dc3-42c1-9609-be607c3bc65f\" (UID: \"95effc61-7dc3-42c1-9609-be607c3bc65f\") "
Nov 02 02:50:57.809525 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:50:57.809494    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95effc61-7dc3-42c1-9609-be607c3bc65f-archives-path" pod "95effc61-7dc3-42c1-9609-be607c3bc65f" (UID: "95effc61-7dc3-42c1-9609-be607c3bc65f"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:50:57.809613 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:50:57.809536    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95effc61-7dc3-42c1-9609-be607c3bc65f-trusted-ca-bundle" pod "95effc61-7dc3-42c1-9609-be607c3bc65f" (UID: "95effc61-7dc3-42c1-9609-be607c3bc65f"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:50:57.809613 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:50:57.809570    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95effc61-7dc3-42c1-9609-be607c3bc65f-service-ca-bundle" pod "95effc61-7dc3-42c1-9609-be607c3bc65f" (UID: "95effc61-7dc3-42c1-9609-be607c3bc65f"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:50:57.811363 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:50:57.811345    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95effc61-7dc3-42c1-9609-be607c3bc65f-kube-api-access-2dl7x" pod "95effc61-7dc3-42c1-9609-be607c3bc65f" (UID: "95effc61-7dc3-42c1-9609-be607c3bc65f"). InnerVolumeSpecName "kube-api-access-2dl7x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:50:57.910721 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:50:57.910690    4496 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95effc61-7dc3-42c1-9609-be607c3bc65f-trusted-ca-bundle\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:50:57.910721 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:50:57.910716    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dl7x\" (UniqueName: \"kubernetes.io/projected/95effc61-7dc3-42c1-9609-be607c3bc65f-kube-api-access-2dl7x\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:50:57.910721 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:50:57.910727    4496 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/95effc61-7dc3-42c1-9609-be607c3bc65f-archives-path\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:50:57.910912 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:50:57.910736    4496 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95effc61-7dc3-42c1-9609-be607c3bc65f-service-ca-bundle\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:50:58.426208 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:50:58.426179    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-t9g7w-hqs6p"
Nov 02 02:50:58.426208 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:50:58.426187    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-t9g7w-hqs6p" event={"ID":"95effc61-7dc3-42c1-9609-be607c3bc65f","Type":"ContainerDied","Data":"d960a2b9fc534555ae1a4989812606630fdb546f04750fe5c7a6129dd61abe4d"}
Nov 02 02:50:58.426413 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:50:58.426239    4496 scope.go:122] "RemoveContainer" containerID="684b7419e6f30d5e3c07e887c1abdf27a11c764d68a8a6944a8901e7659ff795"
Nov 02 02:50:58.591903 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:50:58.591880    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-t9g7w-hqs6p"]
Nov 02 02:50:58.642594 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:50:58.642570    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-t9g7w-hqs6p"]
Nov 02 02:50:59.570035 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:50:59.569998    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95effc61-7dc3-42c1-9609-be607c3bc65f" path="/var/lib/kubelet/pods/95effc61-7dc3-42c1-9609-be607c3bc65f/volumes"
Nov 02 02:51:03.524453 ip-10-0-133-163 kubenswrapper[4496]: E1102 02:51:03.524420    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 02:51:20.727053 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:20.727001    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nxjjl"]
Nov 02 02:51:20.727421 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:20.727306    4496 cpu_manager.go:414] "containerMap: removing container" podUID="f7f8de24-eea6-41db-8d5d-a742aac0fb47" containerName="console"
Nov 02 02:51:20.727421 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:20.727317    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7f8de24-eea6-41db-8d5d-a742aac0fb47" containerName="console"
Nov 02 02:51:20.727421 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:20.727326    4496 cpu_manager.go:414] "containerMap: removing container" podUID="66d05c06-04c1-4ecf-8003-4bd1b75c6c5f" containerName="console"
Nov 02 02:51:20.727421 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:20.727332    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66d05c06-04c1-4ecf-8003-4bd1b75c6c5f" containerName="console"
Nov 02 02:51:20.727421 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:20.727340    4496 cpu_manager.go:414] "containerMap: removing container" podUID="95effc61-7dc3-42c1-9609-be607c3bc65f" containerName="insights-gathering"
Nov 02 02:51:20.727421 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:20.727346    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95effc61-7dc3-42c1-9609-be607c3bc65f" containerName="insights-gathering"
Nov 02 02:51:20.727421 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:20.727399    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7f8de24-eea6-41db-8d5d-a742aac0fb47" containerName="console"
Nov 02 02:51:20.727421 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:20.727408    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="66d05c06-04c1-4ecf-8003-4bd1b75c6c5f" containerName="console"
Nov 02 02:51:20.727421 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:20.727414    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="95effc61-7dc3-42c1-9609-be607c3bc65f" containerName="insights-gathering"
Nov 02 02:51:20.730196 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:20.730179    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nxjjl"
Nov 02 02:51:20.784710 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:20.784681    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7944df8d-53d3-4c2b-b243-20bd43467d36-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nxjjl\" (UID: \"7944df8d-53d3-4c2b-b243-20bd43467d36\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nxjjl"
Nov 02 02:51:20.784843 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:20.784734    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5s7k\" (UniqueName: \"kubernetes.io/projected/7944df8d-53d3-4c2b-b243-20bd43467d36-kube-api-access-s5s7k\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nxjjl\" (UID: \"7944df8d-53d3-4c2b-b243-20bd43467d36\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nxjjl"
Nov 02 02:51:20.784843 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:20.784762    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7944df8d-53d3-4c2b-b243-20bd43467d36-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nxjjl\" (UID: \"7944df8d-53d3-4c2b-b243-20bd43467d36\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nxjjl"
Nov 02 02:51:20.787542 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:20.787519    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nxjjl"]
Nov 02 02:51:20.837167 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:20.837145    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-jnzcg\""
Nov 02 02:51:20.837167 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:20.837145    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Nov 02 02:51:20.837312 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:20.837212    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Nov 02 02:51:20.885555 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:20.885527    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5s7k\" (UniqueName: \"kubernetes.io/projected/7944df8d-53d3-4c2b-b243-20bd43467d36-kube-api-access-s5s7k\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nxjjl\" (UID: \"7944df8d-53d3-4c2b-b243-20bd43467d36\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nxjjl"
Nov 02 02:51:20.885674 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:20.885560    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7944df8d-53d3-4c2b-b243-20bd43467d36-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nxjjl\" (UID: \"7944df8d-53d3-4c2b-b243-20bd43467d36\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nxjjl"
Nov 02 02:51:20.885674 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:20.885598    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7944df8d-53d3-4c2b-b243-20bd43467d36-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nxjjl\" (UID: \"7944df8d-53d3-4c2b-b243-20bd43467d36\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nxjjl"
Nov 02 02:51:20.885900 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:20.885883    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7944df8d-53d3-4c2b-b243-20bd43467d36-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nxjjl\" (UID: \"7944df8d-53d3-4c2b-b243-20bd43467d36\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nxjjl"
Nov 02 02:51:20.885982 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:20.885959    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7944df8d-53d3-4c2b-b243-20bd43467d36-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nxjjl\" (UID: \"7944df8d-53d3-4c2b-b243-20bd43467d36\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nxjjl"
Nov 02 02:51:20.943306 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:20.943281    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5s7k\" (UniqueName: \"kubernetes.io/projected/7944df8d-53d3-4c2b-b243-20bd43467d36-kube-api-access-s5s7k\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nxjjl\" (UID: \"7944df8d-53d3-4c2b-b243-20bd43467d36\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nxjjl"
Nov 02 02:51:21.042342 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:21.042282    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nxjjl"
Nov 02 02:51:21.411404 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:21.411378    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nxjjl"]
Nov 02 02:51:21.413375 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:51:21.413346    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7944df8d_53d3_4c2b_b243_20bd43467d36.slice/crio-f9e7fb8ede950ec59f7024045f3ea24c1dffab9a854ceda8c4bedb847af53a0e WatchSource:0}: Error finding container f9e7fb8ede950ec59f7024045f3ea24c1dffab9a854ceda8c4bedb847af53a0e: Status 404 returned error can't find the container with id f9e7fb8ede950ec59f7024045f3ea24c1dffab9a854ceda8c4bedb847af53a0e
Nov 02 02:51:21.507075 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:21.507046    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nxjjl" event={"ID":"7944df8d-53d3-4c2b-b243-20bd43467d36","Type":"ContainerStarted","Data":"f9e7fb8ede950ec59f7024045f3ea24c1dffab9a854ceda8c4bedb847af53a0e"}
Nov 02 02:51:21.975733 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:21.975694    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mrd6c"]
Nov 02 02:51:21.978527 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:21.978501    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mrd6c"
Nov 02 02:51:22.085580 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:22.085546    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-jpcvg\""
Nov 02 02:51:22.086590 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:22.086571    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mrd6c"]
Nov 02 02:51:22.087434 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:22.087413    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mq29t"]
Nov 02 02:51:22.089658 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:22.089632    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mq29t"
Nov 02 02:51:22.095779 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:22.095757    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/629a2e2b-9caa-4544-9653-a612a79d065f-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mrd6c\" (UID: \"629a2e2b-9caa-4544-9653-a612a79d065f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mrd6c"
Nov 02 02:51:22.095900 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:22.095787    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78x49\" (UniqueName: \"kubernetes.io/projected/629a2e2b-9caa-4544-9653-a612a79d065f-kube-api-access-78x49\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mrd6c\" (UID: \"629a2e2b-9caa-4544-9653-a612a79d065f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mrd6c"
Nov 02 02:51:22.095900 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:22.095821    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/629a2e2b-9caa-4544-9653-a612a79d065f-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mrd6c\" (UID: \"629a2e2b-9caa-4544-9653-a612a79d065f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mrd6c"
Nov 02 02:51:22.138377 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:22.138349    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mq29t"]
Nov 02 02:51:22.142860 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:22.142841    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-k6q9d\""
Nov 02 02:51:22.179101 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:22.179070    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2jh6l"]
Nov 02 02:51:22.181489 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:22.181472    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2jh6l"
Nov 02 02:51:22.196704 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:22.196684    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/629a2e2b-9caa-4544-9653-a612a79d065f-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mrd6c\" (UID: \"629a2e2b-9caa-4544-9653-a612a79d065f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mrd6c"
Nov 02 02:51:22.196783 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:22.196714    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78x49\" (UniqueName: \"kubernetes.io/projected/629a2e2b-9caa-4544-9653-a612a79d065f-kube-api-access-78x49\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mrd6c\" (UID: \"629a2e2b-9caa-4544-9653-a612a79d065f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mrd6c"
Nov 02 02:51:22.196783 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:22.196735    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k99qq\" (UniqueName: \"kubernetes.io/projected/72665a20-d25f-4519-963b-94f562ad524d-kube-api-access-k99qq\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mq29t\" (UID: \"72665a20-d25f-4519-963b-94f562ad524d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mq29t"
Nov 02 02:51:22.196783 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:22.196771    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/72665a20-d25f-4519-963b-94f562ad524d-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mq29t\" (UID: \"72665a20-d25f-4519-963b-94f562ad524d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mq29t"
Nov 02 02:51:22.196884 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:22.196826    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/629a2e2b-9caa-4544-9653-a612a79d065f-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mrd6c\" (UID: \"629a2e2b-9caa-4544-9653-a612a79d065f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mrd6c"
Nov 02 02:51:22.196884 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:22.196862    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/72665a20-d25f-4519-963b-94f562ad524d-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mq29t\" (UID: \"72665a20-d25f-4519-963b-94f562ad524d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mq29t"
Nov 02 02:51:22.197192 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:22.197173    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/629a2e2b-9caa-4544-9653-a612a79d065f-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mrd6c\" (UID: \"629a2e2b-9caa-4544-9653-a612a79d065f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mrd6c"
Nov 02 02:51:22.197271 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:22.197212    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/629a2e2b-9caa-4544-9653-a612a79d065f-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mrd6c\" (UID: \"629a2e2b-9caa-4544-9653-a612a79d065f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mrd6c"
Nov 02 02:51:22.238066 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:22.237993    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2jh6l"]
Nov 02 02:51:22.243117 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:22.243097    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-jdtmx\""
Nov 02 02:51:22.256446 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:22.256419    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78x49\" (UniqueName: \"kubernetes.io/projected/629a2e2b-9caa-4544-9653-a612a79d065f-kube-api-access-78x49\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mrd6c\" (UID: \"629a2e2b-9caa-4544-9653-a612a79d065f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mrd6c"
Nov 02 02:51:22.279689 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:22.279667    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwz5jl"]
Nov 02 02:51:22.282726 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:22.282705    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwz5jl"
Nov 02 02:51:22.290876 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:22.290861    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mrd6c"
Nov 02 02:51:22.297512 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:22.297491    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/72665a20-d25f-4519-963b-94f562ad524d-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mq29t\" (UID: \"72665a20-d25f-4519-963b-94f562ad524d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mq29t"
Nov 02 02:51:22.297622 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:22.297558    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k99qq\" (UniqueName: \"kubernetes.io/projected/72665a20-d25f-4519-963b-94f562ad524d-kube-api-access-k99qq\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mq29t\" (UID: \"72665a20-d25f-4519-963b-94f562ad524d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mq29t"
Nov 02 02:51:22.297622 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:22.297592    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/62c5db55-67e4-4be7-bf51-d6ce0d45303d-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2jh6l\" (UID: \"62c5db55-67e4-4be7-bf51-d6ce0d45303d\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2jh6l"
Nov 02 02:51:22.297735 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:22.297629    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kz2l\" (UniqueName: \"kubernetes.io/projected/62c5db55-67e4-4be7-bf51-d6ce0d45303d-kube-api-access-8kz2l\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2jh6l\" (UID: \"62c5db55-67e4-4be7-bf51-d6ce0d45303d\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2jh6l"
Nov 02 02:51:22.297735 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:22.297680    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/72665a20-d25f-4519-963b-94f562ad524d-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mq29t\" (UID: \"72665a20-d25f-4519-963b-94f562ad524d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mq29t"
Nov 02 02:51:22.297836 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:22.297736    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/62c5db55-67e4-4be7-bf51-d6ce0d45303d-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2jh6l\" (UID: \"62c5db55-67e4-4be7-bf51-d6ce0d45303d\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2jh6l"
Nov 02 02:51:22.298139 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:22.298120    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/72665a20-d25f-4519-963b-94f562ad524d-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mq29t\" (UID: \"72665a20-d25f-4519-963b-94f562ad524d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mq29t"
Nov 02 02:51:22.298241 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:22.298223    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/72665a20-d25f-4519-963b-94f562ad524d-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mq29t\" (UID: \"72665a20-d25f-4519-963b-94f562ad524d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mq29t"
Nov 02 02:51:22.339691 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:22.339657    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwz5jl"]
Nov 02 02:51:22.346420 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:22.346334    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-lnh9m\""
Nov 02 02:51:22.359555 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:22.357869    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k99qq\" (UniqueName: \"kubernetes.io/projected/72665a20-d25f-4519-963b-94f562ad524d-kube-api-access-k99qq\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mq29t\" (UID: \"72665a20-d25f-4519-963b-94f562ad524d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mq29t"
Nov 02 02:51:22.399233 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:22.399201    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/62c5db55-67e4-4be7-bf51-d6ce0d45303d-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2jh6l\" (UID: \"62c5db55-67e4-4be7-bf51-d6ce0d45303d\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2jh6l"
Nov 02 02:51:22.399369 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:22.399254    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zb4s\" (UniqueName: \"kubernetes.io/projected/e87f4778-4d9a-4e2b-87f4-5bea8bc17156-kube-api-access-5zb4s\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwz5jl\" (UID: \"e87f4778-4d9a-4e2b-87f4-5bea8bc17156\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwz5jl"
Nov 02 02:51:22.399369 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:22.399289    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kz2l\" (UniqueName: \"kubernetes.io/projected/62c5db55-67e4-4be7-bf51-d6ce0d45303d-kube-api-access-8kz2l\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2jh6l\" (UID: \"62c5db55-67e4-4be7-bf51-d6ce0d45303d\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2jh6l"
Nov 02 02:51:22.399500 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:22.399380    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e87f4778-4d9a-4e2b-87f4-5bea8bc17156-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwz5jl\" (UID: \"e87f4778-4d9a-4e2b-87f4-5bea8bc17156\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwz5jl"
Nov 02 02:51:22.399597 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:22.399566    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/62c5db55-67e4-4be7-bf51-d6ce0d45303d-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2jh6l\" (UID: \"62c5db55-67e4-4be7-bf51-d6ce0d45303d\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2jh6l"
Nov 02 02:51:22.399652 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:22.399624    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/62c5db55-67e4-4be7-bf51-d6ce0d45303d-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2jh6l\" (UID: \"62c5db55-67e4-4be7-bf51-d6ce0d45303d\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2jh6l"
Nov 02 02:51:22.399652 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:22.399629    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e87f4778-4d9a-4e2b-87f4-5bea8bc17156-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwz5jl\" (UID: \"e87f4778-4d9a-4e2b-87f4-5bea8bc17156\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwz5jl"
Nov 02 02:51:22.399863 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:22.399847    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/62c5db55-67e4-4be7-bf51-d6ce0d45303d-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2jh6l\" (UID: \"62c5db55-67e4-4be7-bf51-d6ce0d45303d\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2jh6l"
Nov 02 02:51:22.402076 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:22.402058    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mq29t"
Nov 02 02:51:22.459046 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:22.459002    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kz2l\" (UniqueName: \"kubernetes.io/projected/62c5db55-67e4-4be7-bf51-d6ce0d45303d-kube-api-access-8kz2l\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2jh6l\" (UID: \"62c5db55-67e4-4be7-bf51-d6ce0d45303d\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2jh6l"
Nov 02 02:51:22.493981 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:22.493725    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2jh6l"
Nov 02 02:51:22.500532 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:22.500505    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zb4s\" (UniqueName: \"kubernetes.io/projected/e87f4778-4d9a-4e2b-87f4-5bea8bc17156-kube-api-access-5zb4s\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwz5jl\" (UID: \"e87f4778-4d9a-4e2b-87f4-5bea8bc17156\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwz5jl"
Nov 02 02:51:22.500638 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:22.500560    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e87f4778-4d9a-4e2b-87f4-5bea8bc17156-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwz5jl\" (UID: \"e87f4778-4d9a-4e2b-87f4-5bea8bc17156\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwz5jl"
Nov 02 02:51:22.500638 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:22.500631    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e87f4778-4d9a-4e2b-87f4-5bea8bc17156-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwz5jl\" (UID: \"e87f4778-4d9a-4e2b-87f4-5bea8bc17156\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwz5jl"
Nov 02 02:51:22.500991 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:22.500964    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e87f4778-4d9a-4e2b-87f4-5bea8bc17156-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwz5jl\" (UID: \"e87f4778-4d9a-4e2b-87f4-5bea8bc17156\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwz5jl"
Nov 02 02:51:22.501136 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:22.501034    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e87f4778-4d9a-4e2b-87f4-5bea8bc17156-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwz5jl\" (UID: \"e87f4778-4d9a-4e2b-87f4-5bea8bc17156\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwz5jl"
Nov 02 02:51:22.559254 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:22.559219    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zb4s\" (UniqueName: \"kubernetes.io/projected/e87f4778-4d9a-4e2b-87f4-5bea8bc17156-kube-api-access-5zb4s\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwz5jl\" (UID: \"e87f4778-4d9a-4e2b-87f4-5bea8bc17156\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwz5jl"
Nov 02 02:51:22.598150 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:22.598120    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwz5jl"
Nov 02 02:51:22.669936 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:22.669909    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mrd6c"]
Nov 02 02:51:22.673796 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:51:22.673770    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod629a2e2b_9caa_4544_9653_a612a79d065f.slice/crio-78989107c868a9c85b489b9b57273b0d9969518d7afad918b4b72974938e5c95 WatchSource:0}: Error finding container 78989107c868a9c85b489b9b57273b0d9969518d7afad918b4b72974938e5c95: Status 404 returned error can't find the container with id 78989107c868a9c85b489b9b57273b0d9969518d7afad918b4b72974938e5c95
Nov 02 02:51:22.778383 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:22.778358    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mq29t"]
Nov 02 02:51:22.781088 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:51:22.781059    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72665a20_d25f_4519_963b_94f562ad524d.slice/crio-8b2667a69bab011c90bbbe415a99c67152fc7c79ba699f6b0904f7a380898fdd WatchSource:0}: Error finding container 8b2667a69bab011c90bbbe415a99c67152fc7c79ba699f6b0904f7a380898fdd: Status 404 returned error can't find the container with id 8b2667a69bab011c90bbbe415a99c67152fc7c79ba699f6b0904f7a380898fdd
Nov 02 02:51:22.871782 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:22.871757    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2jh6l"]
Nov 02 02:51:22.875458 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:51:22.875425    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62c5db55_67e4_4be7_bf51_d6ce0d45303d.slice/crio-5b1bb1b07a30b477996fb2752f1d1fad5e79815d477de1384d3d36792d612182 WatchSource:0}: Error finding container 5b1bb1b07a30b477996fb2752f1d1fad5e79815d477de1384d3d36792d612182: Status 404 returned error can't find the container with id 5b1bb1b07a30b477996fb2752f1d1fad5e79815d477de1384d3d36792d612182
Nov 02 02:51:22.973818 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:22.973797    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwz5jl"]
Nov 02 02:51:22.976090 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:51:22.976061    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode87f4778_4d9a_4e2b_87f4_5bea8bc17156.slice/crio-e4f7f318a67e448d9db435353ebb738a9e7150b5b4a94f94c8ad866bac694154 WatchSource:0}: Error finding container e4f7f318a67e448d9db435353ebb738a9e7150b5b4a94f94c8ad866bac694154: Status 404 returned error can't find the container with id e4f7f318a67e448d9db435353ebb738a9e7150b5b4a94f94c8ad866bac694154
Nov 02 02:51:23.516776 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:23.516738    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mq29t" event={"ID":"72665a20-d25f-4519-963b-94f562ad524d","Type":"ContainerStarted","Data":"8b2667a69bab011c90bbbe415a99c67152fc7c79ba699f6b0904f7a380898fdd"}
Nov 02 02:51:23.518166 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:23.518138    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2jh6l" event={"ID":"62c5db55-67e4-4be7-bf51-d6ce0d45303d","Type":"ContainerStarted","Data":"5b1bb1b07a30b477996fb2752f1d1fad5e79815d477de1384d3d36792d612182"}
Nov 02 02:51:23.519481 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:23.519452    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwz5jl" event={"ID":"e87f4778-4d9a-4e2b-87f4-5bea8bc17156","Type":"ContainerStarted","Data":"e4f7f318a67e448d9db435353ebb738a9e7150b5b4a94f94c8ad866bac694154"}
Nov 02 02:51:23.520705 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:23.520673    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mrd6c" event={"ID":"629a2e2b-9caa-4544-9653-a612a79d065f","Type":"ContainerStarted","Data":"78989107c868a9c85b489b9b57273b0d9969518d7afad918b4b72974938e5c95"}
Nov 02 02:51:24.429339 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:24.427455    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pbvzz"]
Nov 02 02:51:24.431054 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:24.430445    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pbvzz"
Nov 02 02:51:24.486858 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:24.486826    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pbvzz"]
Nov 02 02:51:24.520944 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:24.520906    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/45374d9a-e4d9-49aa-9fde-f0d9e8ff3bd3-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pbvzz\" (UID: \"45374d9a-e4d9-49aa-9fde-f0d9e8ff3bd3\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pbvzz"
Nov 02 02:51:24.521232 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:24.521069    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdkmd\" (UniqueName: \"kubernetes.io/projected/45374d9a-e4d9-49aa-9fde-f0d9e8ff3bd3-kube-api-access-xdkmd\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pbvzz\" (UID: \"45374d9a-e4d9-49aa-9fde-f0d9e8ff3bd3\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pbvzz"
Nov 02 02:51:24.521232 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:24.521165    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/45374d9a-e4d9-49aa-9fde-f0d9e8ff3bd3-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pbvzz\" (UID: \"45374d9a-e4d9-49aa-9fde-f0d9e8ff3bd3\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pbvzz"
Nov 02 02:51:24.537540 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:24.537511    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-hds2w\""
Nov 02 02:51:24.622199 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:24.622161    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/45374d9a-e4d9-49aa-9fde-f0d9e8ff3bd3-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pbvzz\" (UID: \"45374d9a-e4d9-49aa-9fde-f0d9e8ff3bd3\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pbvzz"
Nov 02 02:51:24.622410 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:24.622233    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/45374d9a-e4d9-49aa-9fde-f0d9e8ff3bd3-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pbvzz\" (UID: \"45374d9a-e4d9-49aa-9fde-f0d9e8ff3bd3\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pbvzz"
Nov 02 02:51:24.622410 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:24.622309    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdkmd\" (UniqueName: \"kubernetes.io/projected/45374d9a-e4d9-49aa-9fde-f0d9e8ff3bd3-kube-api-access-xdkmd\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pbvzz\" (UID: \"45374d9a-e4d9-49aa-9fde-f0d9e8ff3bd3\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pbvzz"
Nov 02 02:51:24.623074 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:24.623044    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/45374d9a-e4d9-49aa-9fde-f0d9e8ff3bd3-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pbvzz\" (UID: \"45374d9a-e4d9-49aa-9fde-f0d9e8ff3bd3\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pbvzz"
Nov 02 02:51:24.623356 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:24.623335    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/45374d9a-e4d9-49aa-9fde-f0d9e8ff3bd3-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pbvzz\" (UID: \"45374d9a-e4d9-49aa-9fde-f0d9e8ff3bd3\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pbvzz"
Nov 02 02:51:24.657613 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:24.657579    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbbbct"]
Nov 02 02:51:24.660792 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:24.660763    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbbbct"
Nov 02 02:51:24.689934 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:24.689832    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdkmd\" (UniqueName: \"kubernetes.io/projected/45374d9a-e4d9-49aa-9fde-f0d9e8ff3bd3-kube-api-access-xdkmd\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pbvzz\" (UID: \"45374d9a-e4d9-49aa-9fde-f0d9e8ff3bd3\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pbvzz"
Nov 02 02:51:24.716087 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:24.716038    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-wvj9w\""
Nov 02 02:51:24.717751 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:24.717691    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbbbct"]
Nov 02 02:51:24.723808 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:24.723564    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snrwm\" (UniqueName: \"kubernetes.io/projected/ae3f087c-5607-44c2-aa1b-f616cc5def3d-kube-api-access-snrwm\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbbbct\" (UID: \"ae3f087c-5607-44c2-aa1b-f616cc5def3d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbbbct"
Nov 02 02:51:24.723808 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:24.723655    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ae3f087c-5607-44c2-aa1b-f616cc5def3d-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbbbct\" (UID: \"ae3f087c-5607-44c2-aa1b-f616cc5def3d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbbbct"
Nov 02 02:51:24.723808 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:24.723700    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ae3f087c-5607-44c2-aa1b-f616cc5def3d-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbbbct\" (UID: \"ae3f087c-5607-44c2-aa1b-f616cc5def3d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbbbct"
Nov 02 02:51:24.748590 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:24.748324    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pbvzz"
Nov 02 02:51:24.826263 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:24.824898    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ae3f087c-5607-44c2-aa1b-f616cc5def3d-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbbbct\" (UID: \"ae3f087c-5607-44c2-aa1b-f616cc5def3d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbbbct"
Nov 02 02:51:24.826263 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:24.824950    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ae3f087c-5607-44c2-aa1b-f616cc5def3d-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbbbct\" (UID: \"ae3f087c-5607-44c2-aa1b-f616cc5def3d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbbbct"
Nov 02 02:51:24.826263 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:24.825008    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ae3f087c-5607-44c2-aa1b-f616cc5def3d-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbbbct\" (UID: \"ae3f087c-5607-44c2-aa1b-f616cc5def3d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbbbct"
Nov 02 02:51:24.826263 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:24.825106    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snrwm\" (UniqueName: \"kubernetes.io/projected/ae3f087c-5607-44c2-aa1b-f616cc5def3d-kube-api-access-snrwm\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbbbct\" (UID: \"ae3f087c-5607-44c2-aa1b-f616cc5def3d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbbbct"
Nov 02 02:51:24.826263 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:24.825787    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ae3f087c-5607-44c2-aa1b-f616cc5def3d-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbbbct\" (UID: \"ae3f087c-5607-44c2-aa1b-f616cc5def3d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbbbct"
Nov 02 02:51:24.890467 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:24.890407    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snrwm\" (UniqueName: \"kubernetes.io/projected/ae3f087c-5607-44c2-aa1b-f616cc5def3d-kube-api-access-snrwm\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbbbct\" (UID: \"ae3f087c-5607-44c2-aa1b-f616cc5def3d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbbbct"
Nov 02 02:51:24.979731 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:24.979125    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbbbct"
Nov 02 02:51:25.147854 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:25.147763    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pbvzz"]
Nov 02 02:51:25.151536 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:51:25.151155    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45374d9a_e4d9_49aa_9fde_f0d9e8ff3bd3.slice/crio-9d6f8330ce798b0f14492da5d1abcd1c62adaa6446def43af9fa46fe6b382d40 WatchSource:0}: Error finding container 9d6f8330ce798b0f14492da5d1abcd1c62adaa6446def43af9fa46fe6b382d40: Status 404 returned error can't find the container with id 9d6f8330ce798b0f14492da5d1abcd1c62adaa6446def43af9fa46fe6b382d40
Nov 02 02:51:25.371979 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:25.371954    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbbbct"]
Nov 02 02:51:25.375197 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:51:25.375154    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae3f087c_5607_44c2_aa1b_f616cc5def3d.slice/crio-cc2acaa5592e74d4493c2a3d74ae64e1fd1b10d244367245a4ed4498556a3b4f WatchSource:0}: Error finding container cc2acaa5592e74d4493c2a3d74ae64e1fd1b10d244367245a4ed4498556a3b4f: Status 404 returned error can't find the container with id cc2acaa5592e74d4493c2a3d74ae64e1fd1b10d244367245a4ed4498556a3b4f
Nov 02 02:51:25.534886 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:25.534802    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbbbct" event={"ID":"ae3f087c-5607-44c2-aa1b-f616cc5def3d","Type":"ContainerStarted","Data":"cc2acaa5592e74d4493c2a3d74ae64e1fd1b10d244367245a4ed4498556a3b4f"}
Nov 02 02:51:25.536537 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:25.536508    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pbvzz" event={"ID":"45374d9a-e4d9-49aa-9fde-f0d9e8ff3bd3","Type":"ContainerStarted","Data":"9d6f8330ce798b0f14492da5d1abcd1c62adaa6446def43af9fa46fe6b382d40"}
Nov 02 02:51:25.554270 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:25.554236    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cdwxx"]
Nov 02 02:51:25.560475 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:25.560449    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cdwxx"
Nov 02 02:51:25.617457 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:25.617423    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cdwxx"]
Nov 02 02:51:25.640087 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:25.639474    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 02 02:51:25.640087 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:25.639507    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 02 02:51:25.640087 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:25.639552    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 02 02:51:25.640087 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:25.639703    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41697786-2b0f-4cb2-8975-1942d727c987-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cdwxx\" (UID: \"41697786-2b0f-4cb2-8975-1942d727c987\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cdwxx"
Nov 02 02:51:25.640087 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:25.639742    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41697786-2b0f-4cb2-8975-1942d727c987-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cdwxx\" (UID: \"41697786-2b0f-4cb2-8975-1942d727c987\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cdwxx"
Nov 02 02:51:25.640087 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:25.639887    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41697786-2b0f-4cb2-8975-1942d727c987-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cdwxx\" (UID: \"41697786-2b0f-4cb2-8975-1942d727c987\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cdwxx"
Nov 02 02:51:25.640087 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:25.639921    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41697786-2b0f-4cb2-8975-1942d727c987-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cdwxx\" (UID: \"41697786-2b0f-4cb2-8975-1942d727c987\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cdwxx"
Nov 02 02:51:25.640087 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:25.639991    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41697786-2b0f-4cb2-8975-1942d727c987-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cdwxx\" (UID: \"41697786-2b0f-4cb2-8975-1942d727c987\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cdwxx"
Nov 02 02:51:25.640087 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:25.640046    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvmwf\" (UniqueName: \"kubernetes.io/projected/41697786-2b0f-4cb2-8975-1942d727c987-kube-api-access-jvmwf\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cdwxx\" (UID: \"41697786-2b0f-4cb2-8975-1942d727c987\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cdwxx"
Nov 02 02:51:25.640631 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:25.640123    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41697786-2b0f-4cb2-8975-1942d727c987-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cdwxx\" (UID: \"41697786-2b0f-4cb2-8975-1942d727c987\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cdwxx"
Nov 02 02:51:25.640631 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:25.640154    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41697786-2b0f-4cb2-8975-1942d727c987-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cdwxx\" (UID: \"41697786-2b0f-4cb2-8975-1942d727c987\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cdwxx"
Nov 02 02:51:25.640631 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:25.640191    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41697786-2b0f-4cb2-8975-1942d727c987-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cdwxx\" (UID: \"41697786-2b0f-4cb2-8975-1942d727c987\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cdwxx"
Nov 02 02:51:25.666493 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:25.666467    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-tk8sg\""
Nov 02 02:51:25.741278 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:25.741247    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41697786-2b0f-4cb2-8975-1942d727c987-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cdwxx\" (UID: \"41697786-2b0f-4cb2-8975-1942d727c987\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cdwxx"
Nov 02 02:51:25.741432 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:25.741294    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41697786-2b0f-4cb2-8975-1942d727c987-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cdwxx\" (UID: \"41697786-2b0f-4cb2-8975-1942d727c987\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cdwxx"
Nov 02 02:51:25.741432 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:25.741321    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41697786-2b0f-4cb2-8975-1942d727c987-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cdwxx\" (UID: \"41697786-2b0f-4cb2-8975-1942d727c987\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cdwxx"
Nov 02 02:51:25.741432 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:25.741365    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41697786-2b0f-4cb2-8975-1942d727c987-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cdwxx\" (UID: \"41697786-2b0f-4cb2-8975-1942d727c987\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cdwxx"
Nov 02 02:51:25.741432 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:25.741389    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41697786-2b0f-4cb2-8975-1942d727c987-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cdwxx\" (UID: \"41697786-2b0f-4cb2-8975-1942d727c987\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cdwxx"
Nov 02 02:51:25.741432 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:25.741420    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41697786-2b0f-4cb2-8975-1942d727c987-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cdwxx\" (UID: \"41697786-2b0f-4cb2-8975-1942d727c987\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cdwxx"
Nov 02 02:51:25.741624 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:25.741444    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvmwf\" (UniqueName: \"kubernetes.io/projected/41697786-2b0f-4cb2-8975-1942d727c987-kube-api-access-jvmwf\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cdwxx\" (UID: \"41697786-2b0f-4cb2-8975-1942d727c987\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cdwxx"
Nov 02 02:51:25.741624 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:25.741463    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41697786-2b0f-4cb2-8975-1942d727c987-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cdwxx\" (UID: \"41697786-2b0f-4cb2-8975-1942d727c987\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cdwxx"
Nov 02 02:51:25.741624 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:25.741477    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41697786-2b0f-4cb2-8975-1942d727c987-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cdwxx\" (UID: \"41697786-2b0f-4cb2-8975-1942d727c987\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cdwxx"
Nov 02 02:51:25.742090 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:25.742064    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41697786-2b0f-4cb2-8975-1942d727c987-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cdwxx\" (UID: \"41697786-2b0f-4cb2-8975-1942d727c987\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cdwxx"
Nov 02 02:51:25.742749 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:25.742709    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41697786-2b0f-4cb2-8975-1942d727c987-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cdwxx\" (UID: \"41697786-2b0f-4cb2-8975-1942d727c987\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cdwxx"
Nov 02 02:51:25.743217 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:25.743147    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41697786-2b0f-4cb2-8975-1942d727c987-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cdwxx\" (UID: \"41697786-2b0f-4cb2-8975-1942d727c987\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cdwxx"
Nov 02 02:51:25.743377 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:25.742720    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41697786-2b0f-4cb2-8975-1942d727c987-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cdwxx\" (UID: \"41697786-2b0f-4cb2-8975-1942d727c987\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cdwxx"
Nov 02 02:51:25.743377 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:25.743302    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41697786-2b0f-4cb2-8975-1942d727c987-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cdwxx\" (UID: \"41697786-2b0f-4cb2-8975-1942d727c987\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cdwxx"
Nov 02 02:51:25.755336 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:25.755302    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41697786-2b0f-4cb2-8975-1942d727c987-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cdwxx\" (UID: \"41697786-2b0f-4cb2-8975-1942d727c987\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cdwxx"
Nov 02 02:51:25.772262 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:25.772235    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41697786-2b0f-4cb2-8975-1942d727c987-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cdwxx\" (UID: \"41697786-2b0f-4cb2-8975-1942d727c987\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cdwxx"
Nov 02 02:51:25.800846 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:25.800817    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvmwf\" (UniqueName: \"kubernetes.io/projected/41697786-2b0f-4cb2-8975-1942d727c987-kube-api-access-jvmwf\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cdwxx\" (UID: \"41697786-2b0f-4cb2-8975-1942d727c987\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cdwxx"
Nov 02 02:51:25.817965 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:25.817939    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41697786-2b0f-4cb2-8975-1942d727c987-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cdwxx\" (UID: \"41697786-2b0f-4cb2-8975-1942d727c987\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cdwxx"
Nov 02 02:51:25.852274 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:25.852243    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-8jm8s"]
Nov 02 02:51:25.855474 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:25.855446    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-8jm8s"
Nov 02 02:51:25.894865 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:25.892164    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cdwxx"
Nov 02 02:51:25.917674 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:25.915339    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-dqwxf\""
Nov 02 02:51:25.922456 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:25.922358    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-8jm8s"]
Nov 02 02:51:25.943638 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:25.943565    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slpll\" (UniqueName: \"kubernetes.io/projected/2f73e718-b58c-4073-8b3a-73cba88870e3-kube-api-access-slpll\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-8jm8s\" (UID: \"2f73e718-b58c-4073-8b3a-73cba88870e3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-8jm8s"
Nov 02 02:51:25.943638 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:25.943611    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f73e718-b58c-4073-8b3a-73cba88870e3-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-8jm8s\" (UID: \"2f73e718-b58c-4073-8b3a-73cba88870e3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-8jm8s"
Nov 02 02:51:25.943893 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:25.943698    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f73e718-b58c-4073-8b3a-73cba88870e3-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-8jm8s\" (UID: \"2f73e718-b58c-4073-8b3a-73cba88870e3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-8jm8s"
Nov 02 02:51:25.943893 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:25.943747    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f73e718-b58c-4073-8b3a-73cba88870e3-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-8jm8s\" (UID: \"2f73e718-b58c-4073-8b3a-73cba88870e3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-8jm8s"
Nov 02 02:51:25.943893 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:25.943780    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f73e718-b58c-4073-8b3a-73cba88870e3-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-8jm8s\" (UID: \"2f73e718-b58c-4073-8b3a-73cba88870e3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-8jm8s"
Nov 02 02:51:25.943893 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:25.943814    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f73e718-b58c-4073-8b3a-73cba88870e3-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-8jm8s\" (UID: \"2f73e718-b58c-4073-8b3a-73cba88870e3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-8jm8s"
Nov 02 02:51:25.943893 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:25.943859    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f73e718-b58c-4073-8b3a-73cba88870e3-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-8jm8s\" (UID: \"2f73e718-b58c-4073-8b3a-73cba88870e3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-8jm8s"
Nov 02 02:51:25.943893 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:25.943889    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f73e718-b58c-4073-8b3a-73cba88870e3-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-8jm8s\" (UID: \"2f73e718-b58c-4073-8b3a-73cba88870e3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-8jm8s"
Nov 02 02:51:25.944228 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:25.943935    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f73e718-b58c-4073-8b3a-73cba88870e3-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-8jm8s\" (UID: \"2f73e718-b58c-4073-8b3a-73cba88870e3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-8jm8s"
Nov 02 02:51:26.044395 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:26.044362    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f73e718-b58c-4073-8b3a-73cba88870e3-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-8jm8s\" (UID: \"2f73e718-b58c-4073-8b3a-73cba88870e3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-8jm8s"
Nov 02 02:51:26.044560 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:26.044412    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f73e718-b58c-4073-8b3a-73cba88870e3-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-8jm8s\" (UID: \"2f73e718-b58c-4073-8b3a-73cba88870e3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-8jm8s"
Nov 02 02:51:26.044560 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:26.044432    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f73e718-b58c-4073-8b3a-73cba88870e3-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-8jm8s\" (UID: \"2f73e718-b58c-4073-8b3a-73cba88870e3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-8jm8s"
Nov 02 02:51:26.044560 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:26.044465    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f73e718-b58c-4073-8b3a-73cba88870e3-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-8jm8s\" (UID: \"2f73e718-b58c-4073-8b3a-73cba88870e3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-8jm8s"
Nov 02 02:51:26.044560 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:26.044496    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slpll\" (UniqueName: \"kubernetes.io/projected/2f73e718-b58c-4073-8b3a-73cba88870e3-kube-api-access-slpll\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-8jm8s\" (UID: \"2f73e718-b58c-4073-8b3a-73cba88870e3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-8jm8s"
Nov 02 02:51:26.044560 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:26.044521    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f73e718-b58c-4073-8b3a-73cba88870e3-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-8jm8s\" (UID: \"2f73e718-b58c-4073-8b3a-73cba88870e3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-8jm8s"
Nov 02 02:51:26.044560 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:26.044544    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f73e718-b58c-4073-8b3a-73cba88870e3-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-8jm8s\" (UID: \"2f73e718-b58c-4073-8b3a-73cba88870e3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-8jm8s"
Nov 02 02:51:26.044960 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:26.044573    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f73e718-b58c-4073-8b3a-73cba88870e3-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-8jm8s\" (UID: \"2f73e718-b58c-4073-8b3a-73cba88870e3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-8jm8s"
Nov 02 02:51:26.044960 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:26.044593    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f73e718-b58c-4073-8b3a-73cba88870e3-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-8jm8s\" (UID: \"2f73e718-b58c-4073-8b3a-73cba88870e3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-8jm8s"
Nov 02 02:51:26.044960 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:26.044884    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f73e718-b58c-4073-8b3a-73cba88870e3-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-8jm8s\" (UID: \"2f73e718-b58c-4073-8b3a-73cba88870e3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-8jm8s"
Nov 02 02:51:26.045149 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:26.045107    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f73e718-b58c-4073-8b3a-73cba88870e3-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-8jm8s\" (UID: \"2f73e718-b58c-4073-8b3a-73cba88870e3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-8jm8s"
Nov 02 02:51:26.045254 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:26.045210    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f73e718-b58c-4073-8b3a-73cba88870e3-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-8jm8s\" (UID: \"2f73e718-b58c-4073-8b3a-73cba88870e3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-8jm8s"
Nov 02 02:51:26.046591 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:26.045500    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f73e718-b58c-4073-8b3a-73cba88870e3-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-8jm8s\" (UID: \"2f73e718-b58c-4073-8b3a-73cba88870e3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-8jm8s"
Nov 02 02:51:26.046591 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:26.045610    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f73e718-b58c-4073-8b3a-73cba88870e3-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-8jm8s\" (UID: \"2f73e718-b58c-4073-8b3a-73cba88870e3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-8jm8s"
Nov 02 02:51:26.048794 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:26.048619    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f73e718-b58c-4073-8b3a-73cba88870e3-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-8jm8s\" (UID: \"2f73e718-b58c-4073-8b3a-73cba88870e3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-8jm8s"
Nov 02 02:51:26.048794 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:26.048753    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f73e718-b58c-4073-8b3a-73cba88870e3-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-8jm8s\" (UID: \"2f73e718-b58c-4073-8b3a-73cba88870e3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-8jm8s"
Nov 02 02:51:26.107351 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:26.107256    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f73e718-b58c-4073-8b3a-73cba88870e3-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-8jm8s\" (UID: \"2f73e718-b58c-4073-8b3a-73cba88870e3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-8jm8s"
Nov 02 02:51:26.112849 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:26.112819    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slpll\" (UniqueName: \"kubernetes.io/projected/2f73e718-b58c-4073-8b3a-73cba88870e3-kube-api-access-slpll\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-8jm8s\" (UID: \"2f73e718-b58c-4073-8b3a-73cba88870e3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-8jm8s"
Nov 02 02:51:26.171772 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:26.171689    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-8jm8s"
Nov 02 02:51:26.304291 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:26.304259    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cdwxx"]
Nov 02 02:51:26.322246 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:51:26.322214    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41697786_2b0f_4cb2_8975_1942d727c987.slice/crio-7736aa78ae9e6073eaa844613d93a720975a8667bff0f081c8cafabf87e0ac5b WatchSource:0}: Error finding container 7736aa78ae9e6073eaa844613d93a720975a8667bff0f081c8cafabf87e0ac5b: Status 404 returned error can't find the container with id 7736aa78ae9e6073eaa844613d93a720975a8667bff0f081c8cafabf87e0ac5b
Nov 02 02:51:26.555381 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:26.555302    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cdwxx" event={"ID":"41697786-2b0f-4cb2-8975-1942d727c987","Type":"ContainerStarted","Data":"7736aa78ae9e6073eaa844613d93a720975a8667bff0f081c8cafabf87e0ac5b"}
Nov 02 02:51:26.622931 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:26.622844    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-8jm8s"]
Nov 02 02:51:27.578723 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:27.578654    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-8jm8s" event={"ID":"2f73e718-b58c-4073-8b3a-73cba88870e3","Type":"ContainerStarted","Data":"53d299b6de92642a8c51136c250b281cffa9af56c0c5e6f6abae7d032c00da41"}
Nov 02 02:51:33.575700 ip-10-0-133-163 kubenswrapper[4496]: E1102 02:51:33.575659    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 02:51:35.976403 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:35.976053    4496 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 02:51:35.976403 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:35.976144    4496 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 02:51:35.976403 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:35.976191    4496 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 02:51:36.323371 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:36.322700    4496 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 02:51:36.323371 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:36.322826    4496 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 02:51:36.323371 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:36.322878    4496 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 02:51:36.654443 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:36.654398    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cdwxx" event={"ID":"41697786-2b0f-4cb2-8975-1942d727c987","Type":"ContainerStarted","Data":"a9dec28ff6474e708ad4695c5434851a7e15a0187897222fc106592a444533b5"}
Nov 02 02:51:36.658674 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:36.657882    4496 generic.go:356] "Generic (PLEG): container finished" podID="ae3f087c-5607-44c2-aa1b-f616cc5def3d" containerID="1a501eb37e7b46407b37b99d5f24d003508da6a586d7949aca4bd7721fa0a034" exitCode=0
Nov 02 02:51:36.658674 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:36.657976    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbbbct" event={"ID":"ae3f087c-5607-44c2-aa1b-f616cc5def3d","Type":"ContainerDied","Data":"1a501eb37e7b46407b37b99d5f24d003508da6a586d7949aca4bd7721fa0a034"}
Nov 02 02:51:36.664809 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:36.664782    4496 generic.go:356] "Generic (PLEG): container finished" podID="7944df8d-53d3-4c2b-b243-20bd43467d36" containerID="8c272afe40825d79fe39585c91bf1211834105da0267a65da65572a4c05c5cb3" exitCode=0
Nov 02 02:51:36.664913 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:36.664866    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nxjjl" event={"ID":"7944df8d-53d3-4c2b-b243-20bd43467d36","Type":"ContainerDied","Data":"8c272afe40825d79fe39585c91bf1211834105da0267a65da65572a4c05c5cb3"}
Nov 02 02:51:36.668255 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:36.668224    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-8jm8s" event={"ID":"2f73e718-b58c-4073-8b3a-73cba88870e3","Type":"ContainerStarted","Data":"489b8455758ced291a48fd34c7172b392ed4d91f9008b92380e604ee1d41c445"}
Nov 02 02:51:36.817558 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:36.817496    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cdwxx" podStartSLOduration=2.169434745 podStartE2EDuration="11.817477302s" totalImagesPullingTime="9.648042557s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:51:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:51:36.816419699 +0000 UTC m=+237.395027081" watchObservedRunningTime="2026-11-02 02:51:36.817477302 +0000 UTC m=+237.396084680"
Nov 02 02:51:36.893067 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:36.893031    4496 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cdwxx"
Nov 02 02:51:37.056269 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:37.056176    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-8jm8s" podStartSLOduration=2.372118632 podStartE2EDuration="12.056161819s" totalImagesPullingTime="9.684043187s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:51:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:51:37.054942714 +0000 UTC m=+237.633550094" watchObservedRunningTime="2026-11-02 02:51:37.056161819 +0000 UTC m=+237.634769196"
Nov 02 02:51:37.173182 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:37.173102    4496 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-8jm8s"
Nov 02 02:51:37.178359 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:37.178135    4496 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-8jm8s"
Nov 02 02:51:37.674622 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:37.674582    4496 generic.go:356] "Generic (PLEG): container finished" podID="45374d9a-e4d9-49aa-9fde-f0d9e8ff3bd3" containerID="6a3244c84d00b590c0e0f8ae5767d3b35495d1405284c9da17a4144bd2f89a55" exitCode=0
Nov 02 02:51:37.674803 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:37.674689    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pbvzz" event={"ID":"45374d9a-e4d9-49aa-9fde-f0d9e8ff3bd3","Type":"ContainerDied","Data":"6a3244c84d00b590c0e0f8ae5767d3b35495d1405284c9da17a4144bd2f89a55"}
Nov 02 02:51:37.675149 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:37.675129    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-8jm8s"
Nov 02 02:51:37.675993 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:37.675900    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-8jm8s"
Nov 02 02:51:37.897795 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:37.897741    4496 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cdwxx"
Nov 02 02:51:38.683037 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:38.682980    4496 generic.go:356] "Generic (PLEG): container finished" podID="7944df8d-53d3-4c2b-b243-20bd43467d36" containerID="1135e1ef809fdec312628b73f471784eba99fe99e35609501cc53df3dac7d6dd" exitCode=0
Nov 02 02:51:38.683419 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:38.683068    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nxjjl" event={"ID":"7944df8d-53d3-4c2b-b243-20bd43467d36","Type":"ContainerDied","Data":"1135e1ef809fdec312628b73f471784eba99fe99e35609501cc53df3dac7d6dd"}
Nov 02 02:51:38.684904 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:38.684871    4496 generic.go:356] "Generic (PLEG): container finished" podID="72665a20-d25f-4519-963b-94f562ad524d" containerID="c302a644f6ca4629c33f7eb6696c192dba4fb5337f9025c1a41d8898c59f8d78" exitCode=0
Nov 02 02:51:38.685053 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:38.685010    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mq29t" event={"ID":"72665a20-d25f-4519-963b-94f562ad524d","Type":"ContainerDied","Data":"c302a644f6ca4629c33f7eb6696c192dba4fb5337f9025c1a41d8898c59f8d78"}
Nov 02 02:51:38.686902 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:38.686876    4496 generic.go:356] "Generic (PLEG): container finished" podID="62c5db55-67e4-4be7-bf51-d6ce0d45303d" containerID="84f9ff4f6b544c5146ef0727530b8585dd342bf870737d5a15f9a677911ba6c4" exitCode=0
Nov 02 02:51:38.686992 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:38.686970    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2jh6l" event={"ID":"62c5db55-67e4-4be7-bf51-d6ce0d45303d","Type":"ContainerDied","Data":"84f9ff4f6b544c5146ef0727530b8585dd342bf870737d5a15f9a677911ba6c4"}
Nov 02 02:51:38.689308 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:38.688772    4496 generic.go:356] "Generic (PLEG): container finished" podID="e87f4778-4d9a-4e2b-87f4-5bea8bc17156" containerID="8d247ef205d0cd9cc3eded26ddc55c06385988dbe2698086772ba9718a08373c" exitCode=0
Nov 02 02:51:38.689308 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:38.688843    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwz5jl" event={"ID":"e87f4778-4d9a-4e2b-87f4-5bea8bc17156","Type":"ContainerDied","Data":"8d247ef205d0cd9cc3eded26ddc55c06385988dbe2698086772ba9718a08373c"}
Nov 02 02:51:38.691165 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:38.691143    4496 generic.go:356] "Generic (PLEG): container finished" podID="629a2e2b-9caa-4544-9653-a612a79d065f" containerID="7ab4e2081e7ac377e3a58cbfa7bd448a57f0d94f6fa323c17e150d154ffef8c4" exitCode=0
Nov 02 02:51:38.691337 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:38.691281    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mrd6c" event={"ID":"629a2e2b-9caa-4544-9653-a612a79d065f","Type":"ContainerDied","Data":"7ab4e2081e7ac377e3a58cbfa7bd448a57f0d94f6fa323c17e150d154ffef8c4"}
Nov 02 02:51:38.691890 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:38.691871    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cdwxx"
Nov 02 02:51:38.692554 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:38.692534    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cdwxx"
Nov 02 02:51:40.703545 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:40.703513    4496 generic.go:356] "Generic (PLEG): container finished" podID="629a2e2b-9caa-4544-9653-a612a79d065f" containerID="523237c01ee83f9b63e5be2504b23d1d73780e2111a095f5af9a7b9e3a204450" exitCode=0
Nov 02 02:51:40.703921 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:40.703598    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mrd6c" event={"ID":"629a2e2b-9caa-4544-9653-a612a79d065f","Type":"ContainerDied","Data":"523237c01ee83f9b63e5be2504b23d1d73780e2111a095f5af9a7b9e3a204450"}
Nov 02 02:51:40.708331 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:40.708306    4496 generic.go:356] "Generic (PLEG): container finished" podID="ae3f087c-5607-44c2-aa1b-f616cc5def3d" containerID="27a3b4c4559247af5d203839e9bd4b11edba9f5eeea9e086ac97b66082133fe8" exitCode=0
Nov 02 02:51:40.708426 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:40.708411    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbbbct" event={"ID":"ae3f087c-5607-44c2-aa1b-f616cc5def3d","Type":"ContainerDied","Data":"27a3b4c4559247af5d203839e9bd4b11edba9f5eeea9e086ac97b66082133fe8"}
Nov 02 02:51:40.717591 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:40.717390    4496 generic.go:356] "Generic (PLEG): container finished" podID="72665a20-d25f-4519-963b-94f562ad524d" containerID="98421b9088ba16fee5117e98adfcf19a0c1d09838d927cabb69848881e280de1" exitCode=0
Nov 02 02:51:40.717591 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:40.717540    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mq29t" event={"ID":"72665a20-d25f-4519-963b-94f562ad524d","Type":"ContainerDied","Data":"98421b9088ba16fee5117e98adfcf19a0c1d09838d927cabb69848881e280de1"}
Nov 02 02:51:40.721234 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:40.720595    4496 generic.go:356] "Generic (PLEG): container finished" podID="62c5db55-67e4-4be7-bf51-d6ce0d45303d" containerID="380dd43644e38e4827081cab5145c227e72e727f57d64d991e12c9908b00933f" exitCode=0
Nov 02 02:51:40.721234 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:40.720673    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2jh6l" event={"ID":"62c5db55-67e4-4be7-bf51-d6ce0d45303d","Type":"ContainerDied","Data":"380dd43644e38e4827081cab5145c227e72e727f57d64d991e12c9908b00933f"}
Nov 02 02:51:41.734186 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:41.734154    4496 generic.go:356] "Generic (PLEG): container finished" podID="e87f4778-4d9a-4e2b-87f4-5bea8bc17156" containerID="db3a028b06e1fc7a5f438ebaa604b0ed5c84b3b9cee7489c620e9991a7397b20" exitCode=0
Nov 02 02:51:41.734567 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:41.734246    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwz5jl" event={"ID":"e87f4778-4d9a-4e2b-87f4-5bea8bc17156","Type":"ContainerDied","Data":"db3a028b06e1fc7a5f438ebaa604b0ed5c84b3b9cee7489c620e9991a7397b20"}
Nov 02 02:51:42.750563 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:42.749570    4496 generic.go:356] "Generic (PLEG): container finished" podID="45374d9a-e4d9-49aa-9fde-f0d9e8ff3bd3" containerID="1088fbd7fab74fe789f95df26dfe796a87d5adbfba4802ab9f774fa2027891c2" exitCode=0
Nov 02 02:51:42.750563 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:42.749685    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pbvzz" event={"ID":"45374d9a-e4d9-49aa-9fde-f0d9e8ff3bd3","Type":"ContainerDied","Data":"1088fbd7fab74fe789f95df26dfe796a87d5adbfba4802ab9f774fa2027891c2"}
Nov 02 02:51:52.830963 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:52.830880    4496 generic.go:356] "Generic (PLEG): container finished" podID="45374d9a-e4d9-49aa-9fde-f0d9e8ff3bd3" containerID="3266b7cbb7b28aaef9db441dc025391bcedf75ada1e67391349613ea96485ed4" exitCode=0
Nov 02 02:51:52.831444 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:52.831006    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pbvzz" event={"ID":"45374d9a-e4d9-49aa-9fde-f0d9e8ff3bd3","Type":"ContainerDied","Data":"3266b7cbb7b28aaef9db441dc025391bcedf75ada1e67391349613ea96485ed4"}
Nov 02 02:51:52.834256 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:52.834224    4496 generic.go:356] "Generic (PLEG): container finished" podID="7944df8d-53d3-4c2b-b243-20bd43467d36" containerID="c7d4c1fafa0214ca5f1a9049c2c0af8dc4b706316b70c47d67b0b4f0cc1ced03" exitCode=0
Nov 02 02:51:52.834386 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:52.834335    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nxjjl" event={"ID":"7944df8d-53d3-4c2b-b243-20bd43467d36","Type":"ContainerDied","Data":"c7d4c1fafa0214ca5f1a9049c2c0af8dc4b706316b70c47d67b0b4f0cc1ced03"}
Nov 02 02:51:54.050943 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:54.050459    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nxjjl"
Nov 02 02:51:54.065415 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:54.065347    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pbvzz"
Nov 02 02:51:54.144084 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:54.143324    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45374d9a-e4d9-49aa-9fde-f0d9e8ff3bd3-util\" (UniqueName: \"kubernetes.io/empty-dir/45374d9a-e4d9-49aa-9fde-f0d9e8ff3bd3-util\") pod \"45374d9a-e4d9-49aa-9fde-f0d9e8ff3bd3\" (UID: \"45374d9a-e4d9-49aa-9fde-f0d9e8ff3bd3\") "
Nov 02 02:51:54.144084 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:54.143397    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7944df8d-53d3-4c2b-b243-20bd43467d36-bundle\" (UniqueName: \"kubernetes.io/empty-dir/7944df8d-53d3-4c2b-b243-20bd43467d36-bundle\") pod \"7944df8d-53d3-4c2b-b243-20bd43467d36\" (UID: \"7944df8d-53d3-4c2b-b243-20bd43467d36\") "
Nov 02 02:51:54.144084 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:54.143437    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45374d9a-e4d9-49aa-9fde-f0d9e8ff3bd3-bundle\" (UniqueName: \"kubernetes.io/empty-dir/45374d9a-e4d9-49aa-9fde-f0d9e8ff3bd3-bundle\") pod \"45374d9a-e4d9-49aa-9fde-f0d9e8ff3bd3\" (UID: \"45374d9a-e4d9-49aa-9fde-f0d9e8ff3bd3\") "
Nov 02 02:51:54.144084 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:54.143521    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7944df8d-53d3-4c2b-b243-20bd43467d36-kube-api-access-s5s7k\" (UniqueName: \"kubernetes.io/projected/7944df8d-53d3-4c2b-b243-20bd43467d36-kube-api-access-s5s7k\") pod \"7944df8d-53d3-4c2b-b243-20bd43467d36\" (UID: \"7944df8d-53d3-4c2b-b243-20bd43467d36\") "
Nov 02 02:51:54.144084 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:54.143553    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45374d9a-e4d9-49aa-9fde-f0d9e8ff3bd3-kube-api-access-xdkmd\" (UniqueName: \"kubernetes.io/projected/45374d9a-e4d9-49aa-9fde-f0d9e8ff3bd3-kube-api-access-xdkmd\") pod \"45374d9a-e4d9-49aa-9fde-f0d9e8ff3bd3\" (UID: \"45374d9a-e4d9-49aa-9fde-f0d9e8ff3bd3\") "
Nov 02 02:51:54.144084 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:54.143584    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7944df8d-53d3-4c2b-b243-20bd43467d36-util\" (UniqueName: \"kubernetes.io/empty-dir/7944df8d-53d3-4c2b-b243-20bd43467d36-util\") pod \"7944df8d-53d3-4c2b-b243-20bd43467d36\" (UID: \"7944df8d-53d3-4c2b-b243-20bd43467d36\") "
Nov 02 02:51:54.144980 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:54.144925    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45374d9a-e4d9-49aa-9fde-f0d9e8ff3bd3-bundle" pod "45374d9a-e4d9-49aa-9fde-f0d9e8ff3bd3" (UID: "45374d9a-e4d9-49aa-9fde-f0d9e8ff3bd3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:51:54.146138 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:54.145085    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7944df8d-53d3-4c2b-b243-20bd43467d36-bundle" pod "7944df8d-53d3-4c2b-b243-20bd43467d36" (UID: "7944df8d-53d3-4c2b-b243-20bd43467d36"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:51:54.150353 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:54.150321    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45374d9a-e4d9-49aa-9fde-f0d9e8ff3bd3-util" pod "45374d9a-e4d9-49aa-9fde-f0d9e8ff3bd3" (UID: "45374d9a-e4d9-49aa-9fde-f0d9e8ff3bd3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:51:54.151419 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:54.151394    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7944df8d-53d3-4c2b-b243-20bd43467d36-util" pod "7944df8d-53d3-4c2b-b243-20bd43467d36" (UID: "7944df8d-53d3-4c2b-b243-20bd43467d36"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:51:54.157934 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:54.157899    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7944df8d-53d3-4c2b-b243-20bd43467d36-kube-api-access-s5s7k" pod "7944df8d-53d3-4c2b-b243-20bd43467d36" (UID: "7944df8d-53d3-4c2b-b243-20bd43467d36"). InnerVolumeSpecName "kube-api-access-s5s7k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:51:54.162787 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:54.162751    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45374d9a-e4d9-49aa-9fde-f0d9e8ff3bd3-kube-api-access-xdkmd" pod "45374d9a-e4d9-49aa-9fde-f0d9e8ff3bd3" (UID: "45374d9a-e4d9-49aa-9fde-f0d9e8ff3bd3"). InnerVolumeSpecName "kube-api-access-xdkmd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:51:54.245119 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:54.245081    4496 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/45374d9a-e4d9-49aa-9fde-f0d9e8ff3bd3-bundle\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:51:54.245119 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:54.245113    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5s7k\" (UniqueName: \"kubernetes.io/projected/7944df8d-53d3-4c2b-b243-20bd43467d36-kube-api-access-s5s7k\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:51:54.245393 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:54.245127    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdkmd\" (UniqueName: \"kubernetes.io/projected/45374d9a-e4d9-49aa-9fde-f0d9e8ff3bd3-kube-api-access-xdkmd\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:51:54.245393 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:54.245140    4496 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7944df8d-53d3-4c2b-b243-20bd43467d36-util\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:51:54.245393 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:54.245151    4496 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/45374d9a-e4d9-49aa-9fde-f0d9e8ff3bd3-util\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:51:54.245393 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:54.245162    4496 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7944df8d-53d3-4c2b-b243-20bd43467d36-bundle\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:51:54.850051 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:54.848957    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pbvzz" event={"ID":"45374d9a-e4d9-49aa-9fde-f0d9e8ff3bd3","Type":"ContainerDied","Data":"9d6f8330ce798b0f14492da5d1abcd1c62adaa6446def43af9fa46fe6b382d40"}
Nov 02 02:51:54.850051 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:54.849029    4496 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d6f8330ce798b0f14492da5d1abcd1c62adaa6446def43af9fa46fe6b382d40"
Nov 02 02:51:54.850051 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:54.849127    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pbvzz"
Nov 02 02:51:54.854919 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:54.854890    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nxjjl"
Nov 02 02:51:54.855085 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:54.854889    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nxjjl" event={"ID":"7944df8d-53d3-4c2b-b243-20bd43467d36","Type":"ContainerDied","Data":"f9e7fb8ede950ec59f7024045f3ea24c1dffab9a854ceda8c4bedb847af53a0e"}
Nov 02 02:51:54.855085 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:54.855044    4496 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f9e7fb8ede950ec59f7024045f3ea24c1dffab9a854ceda8c4bedb847af53a0e"
Nov 02 02:51:55.862657 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:55.862510    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2jh6l" event={"ID":"62c5db55-67e4-4be7-bf51-d6ce0d45303d","Type":"ContainerStarted","Data":"38c81507858f579fa97abb9772c68148b28728fab0d20198890c5c08ad3319b3"}
Nov 02 02:51:55.865026 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:55.864986    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwz5jl" event={"ID":"e87f4778-4d9a-4e2b-87f4-5bea8bc17156","Type":"ContainerStarted","Data":"016863018b6ad307101bc3139ef51aa3c8e0d931be2203cdcacf3bf7c473b9e8"}
Nov 02 02:51:55.867228 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:55.867206    4496 generic.go:356] "Generic (PLEG): container finished" podID="629a2e2b-9caa-4544-9653-a612a79d065f" containerID="f7bfc0ec7080319abd3b64e94a678d16325423f713b69c859fae27796d0ad4e5" exitCode=0
Nov 02 02:51:55.867349 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:55.867272    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mrd6c" event={"ID":"629a2e2b-9caa-4544-9653-a612a79d065f","Type":"ContainerDied","Data":"f7bfc0ec7080319abd3b64e94a678d16325423f713b69c859fae27796d0ad4e5"}
Nov 02 02:51:55.869594 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:55.869573    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbbbct" event={"ID":"ae3f087c-5607-44c2-aa1b-f616cc5def3d","Type":"ContainerStarted","Data":"df79d51d30066b777e3bf320666c99dabc337e7c3e2ecf0ad53582baf6009bf1"}
Nov 02 02:51:55.871594 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:55.871576    4496 generic.go:356] "Generic (PLEG): container finished" podID="72665a20-d25f-4519-963b-94f562ad524d" containerID="1463d889765c496adcbcbd3df5b52239e0ae503f9423444e14631614d9769b47" exitCode=0
Nov 02 02:51:55.871678 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:55.871614    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mq29t" event={"ID":"72665a20-d25f-4519-963b-94f562ad524d","Type":"ContainerDied","Data":"1463d889765c496adcbcbd3df5b52239e0ae503f9423444e14631614d9769b47"}
Nov 02 02:51:55.979033 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:55.978969    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2jh6l" podStartSLOduration=2.313814135 podStartE2EDuration="33.978956054s" totalImagesPullingTime="31.665141919s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:51:22 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:51:55.977969754 +0000 UTC m=+256.556577134" watchObservedRunningTime="2026-11-02 02:51:55.978956054 +0000 UTC m=+256.557563434"
Nov 02 02:51:56.095385 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:56.095339    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbbbct" podStartSLOduration=3.494356918 podStartE2EDuration="32.095325655s" totalImagesPullingTime="28.600968737s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:51:24 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:51:56.095056604 +0000 UTC m=+256.673663983" watchObservedRunningTime="2026-11-02 02:51:56.095325655 +0000 UTC m=+256.673933035"
Nov 02 02:51:56.211336 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:56.211292    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwz5jl" podStartSLOduration=3.339214396 podStartE2EDuration="34.211278652s" totalImagesPullingTime="30.872064256s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:51:22 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:51:56.21066541 +0000 UTC m=+256.789272789" watchObservedRunningTime="2026-11-02 02:51:56.211278652 +0000 UTC m=+256.789886032"
Nov 02 02:51:56.878098 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:56.878064    4496 generic.go:356] "Generic (PLEG): container finished" podID="62c5db55-67e4-4be7-bf51-d6ce0d45303d" containerID="38c81507858f579fa97abb9772c68148b28728fab0d20198890c5c08ad3319b3" exitCode=0
Nov 02 02:51:56.878529 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:56.878146    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2jh6l" event={"ID":"62c5db55-67e4-4be7-bf51-d6ce0d45303d","Type":"ContainerDied","Data":"38c81507858f579fa97abb9772c68148b28728fab0d20198890c5c08ad3319b3"}
Nov 02 02:51:56.880220 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:56.880199    4496 generic.go:356] "Generic (PLEG): container finished" podID="e87f4778-4d9a-4e2b-87f4-5bea8bc17156" containerID="016863018b6ad307101bc3139ef51aa3c8e0d931be2203cdcacf3bf7c473b9e8" exitCode=0
Nov 02 02:51:56.880357 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:56.880272    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwz5jl" event={"ID":"e87f4778-4d9a-4e2b-87f4-5bea8bc17156","Type":"ContainerDied","Data":"016863018b6ad307101bc3139ef51aa3c8e0d931be2203cdcacf3bf7c473b9e8"}
Nov 02 02:51:56.887670 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:56.887630    4496 generic.go:356] "Generic (PLEG): container finished" podID="ae3f087c-5607-44c2-aa1b-f616cc5def3d" containerID="df79d51d30066b777e3bf320666c99dabc337e7c3e2ecf0ad53582baf6009bf1" exitCode=0
Nov 02 02:51:56.887783 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:56.887719    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbbbct" event={"ID":"ae3f087c-5607-44c2-aa1b-f616cc5def3d","Type":"ContainerDied","Data":"df79d51d30066b777e3bf320666c99dabc337e7c3e2ecf0ad53582baf6009bf1"}
Nov 02 02:51:57.010590 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:57.010566    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mq29t"
Nov 02 02:51:57.032425 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:57.032405    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mrd6c"
Nov 02 02:51:57.068559 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:57.068536    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/629a2e2b-9caa-4544-9653-a612a79d065f-kube-api-access-78x49\" (UniqueName: \"kubernetes.io/projected/629a2e2b-9caa-4544-9653-a612a79d065f-kube-api-access-78x49\") pod \"629a2e2b-9caa-4544-9653-a612a79d065f\" (UID: \"629a2e2b-9caa-4544-9653-a612a79d065f\") "
Nov 02 02:51:57.068697 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:57.068596    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72665a20-d25f-4519-963b-94f562ad524d-kube-api-access-k99qq\" (UniqueName: \"kubernetes.io/projected/72665a20-d25f-4519-963b-94f562ad524d-kube-api-access-k99qq\") pod \"72665a20-d25f-4519-963b-94f562ad524d\" (UID: \"72665a20-d25f-4519-963b-94f562ad524d\") "
Nov 02 02:51:57.068697 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:57.068625    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72665a20-d25f-4519-963b-94f562ad524d-util\" (UniqueName: \"kubernetes.io/empty-dir/72665a20-d25f-4519-963b-94f562ad524d-util\") pod \"72665a20-d25f-4519-963b-94f562ad524d\" (UID: \"72665a20-d25f-4519-963b-94f562ad524d\") "
Nov 02 02:51:57.068697 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:57.068684    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/629a2e2b-9caa-4544-9653-a612a79d065f-util\" (UniqueName: \"kubernetes.io/empty-dir/629a2e2b-9caa-4544-9653-a612a79d065f-util\") pod \"629a2e2b-9caa-4544-9653-a612a79d065f\" (UID: \"629a2e2b-9caa-4544-9653-a612a79d065f\") "
Nov 02 02:51:57.068857 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:57.068719    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/629a2e2b-9caa-4544-9653-a612a79d065f-bundle\" (UniqueName: \"kubernetes.io/empty-dir/629a2e2b-9caa-4544-9653-a612a79d065f-bundle\") pod \"629a2e2b-9caa-4544-9653-a612a79d065f\" (UID: \"629a2e2b-9caa-4544-9653-a612a79d065f\") "
Nov 02 02:51:57.068857 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:57.068766    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72665a20-d25f-4519-963b-94f562ad524d-bundle\" (UniqueName: \"kubernetes.io/empty-dir/72665a20-d25f-4519-963b-94f562ad524d-bundle\") pod \"72665a20-d25f-4519-963b-94f562ad524d\" (UID: \"72665a20-d25f-4519-963b-94f562ad524d\") "
Nov 02 02:51:57.069272 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:57.069238    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/629a2e2b-9caa-4544-9653-a612a79d065f-bundle" pod "629a2e2b-9caa-4544-9653-a612a79d065f" (UID: "629a2e2b-9caa-4544-9653-a612a79d065f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:51:57.069360 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:57.069289    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72665a20-d25f-4519-963b-94f562ad524d-bundle" pod "72665a20-d25f-4519-963b-94f562ad524d" (UID: "72665a20-d25f-4519-963b-94f562ad524d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:51:57.070733 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:57.070714    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72665a20-d25f-4519-963b-94f562ad524d-kube-api-access-k99qq" pod "72665a20-d25f-4519-963b-94f562ad524d" (UID: "72665a20-d25f-4519-963b-94f562ad524d"). InnerVolumeSpecName "kube-api-access-k99qq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:51:57.070996 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:57.070980    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/629a2e2b-9caa-4544-9653-a612a79d065f-kube-api-access-78x49" pod "629a2e2b-9caa-4544-9653-a612a79d065f" (UID: "629a2e2b-9caa-4544-9653-a612a79d065f"). InnerVolumeSpecName "kube-api-access-78x49". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:51:57.074460 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:57.074432    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72665a20-d25f-4519-963b-94f562ad524d-util" pod "72665a20-d25f-4519-963b-94f562ad524d" (UID: "72665a20-d25f-4519-963b-94f562ad524d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:51:57.074641 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:57.074620    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/629a2e2b-9caa-4544-9653-a612a79d065f-util" pod "629a2e2b-9caa-4544-9653-a612a79d065f" (UID: "629a2e2b-9caa-4544-9653-a612a79d065f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:51:57.169584 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:57.169527    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78x49\" (UniqueName: \"kubernetes.io/projected/629a2e2b-9caa-4544-9653-a612a79d065f-kube-api-access-78x49\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:51:57.169584 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:57.169550    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k99qq\" (UniqueName: \"kubernetes.io/projected/72665a20-d25f-4519-963b-94f562ad524d-kube-api-access-k99qq\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:51:57.169584 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:57.169562    4496 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/72665a20-d25f-4519-963b-94f562ad524d-util\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:51:57.169584 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:57.169570    4496 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/629a2e2b-9caa-4544-9653-a612a79d065f-util\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:51:57.169584 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:57.169578    4496 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/629a2e2b-9caa-4544-9653-a612a79d065f-bundle\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:51:57.169584 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:57.169586    4496 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/72665a20-d25f-4519-963b-94f562ad524d-bundle\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:51:57.894852 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:57.894778    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mrd6c"
Nov 02 02:51:57.894852 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:57.894784    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mrd6c" event={"ID":"629a2e2b-9caa-4544-9653-a612a79d065f","Type":"ContainerDied","Data":"78989107c868a9c85b489b9b57273b0d9969518d7afad918b4b72974938e5c95"}
Nov 02 02:51:57.894852 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:57.894840    4496 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="78989107c868a9c85b489b9b57273b0d9969518d7afad918b4b72974938e5c95"
Nov 02 02:51:57.897282 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:57.897204    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mq29t" event={"ID":"72665a20-d25f-4519-963b-94f562ad524d","Type":"ContainerDied","Data":"8b2667a69bab011c90bbbe415a99c67152fc7c79ba699f6b0904f7a380898fdd"}
Nov 02 02:51:57.897282 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:57.897247    4496 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8b2667a69bab011c90bbbe415a99c67152fc7c79ba699f6b0904f7a380898fdd"
Nov 02 02:51:57.897537 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:57.897517    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mq29t"
Nov 02 02:51:58.047449 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:58.047422    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2jh6l"
Nov 02 02:51:58.076096 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:58.076059    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62c5db55-67e4-4be7-bf51-d6ce0d45303d-util\" (UniqueName: \"kubernetes.io/empty-dir/62c5db55-67e4-4be7-bf51-d6ce0d45303d-util\") pod \"62c5db55-67e4-4be7-bf51-d6ce0d45303d\" (UID: \"62c5db55-67e4-4be7-bf51-d6ce0d45303d\") "
Nov 02 02:51:58.076173 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:58.076113    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62c5db55-67e4-4be7-bf51-d6ce0d45303d-bundle\" (UniqueName: \"kubernetes.io/empty-dir/62c5db55-67e4-4be7-bf51-d6ce0d45303d-bundle\") pod \"62c5db55-67e4-4be7-bf51-d6ce0d45303d\" (UID: \"62c5db55-67e4-4be7-bf51-d6ce0d45303d\") "
Nov 02 02:51:58.076173 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:58.076146    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62c5db55-67e4-4be7-bf51-d6ce0d45303d-kube-api-access-8kz2l\" (UniqueName: \"kubernetes.io/projected/62c5db55-67e4-4be7-bf51-d6ce0d45303d-kube-api-access-8kz2l\") pod \"62c5db55-67e4-4be7-bf51-d6ce0d45303d\" (UID: \"62c5db55-67e4-4be7-bf51-d6ce0d45303d\") "
Nov 02 02:51:58.077357 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:58.077331    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62c5db55-67e4-4be7-bf51-d6ce0d45303d-bundle" pod "62c5db55-67e4-4be7-bf51-d6ce0d45303d" (UID: "62c5db55-67e4-4be7-bf51-d6ce0d45303d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:51:58.078319 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:58.078298    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62c5db55-67e4-4be7-bf51-d6ce0d45303d-kube-api-access-8kz2l" pod "62c5db55-67e4-4be7-bf51-d6ce0d45303d" (UID: "62c5db55-67e4-4be7-bf51-d6ce0d45303d"). InnerVolumeSpecName "kube-api-access-8kz2l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:51:58.082485 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:58.082461    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62c5db55-67e4-4be7-bf51-d6ce0d45303d-util" pod "62c5db55-67e4-4be7-bf51-d6ce0d45303d" (UID: "62c5db55-67e4-4be7-bf51-d6ce0d45303d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:51:58.084592 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:58.084567    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwz5jl"
Nov 02 02:51:58.086852 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:58.086834    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbbbct"
Nov 02 02:51:58.177270 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:58.177199    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e87f4778-4d9a-4e2b-87f4-5bea8bc17156-util\" (UniqueName: \"kubernetes.io/empty-dir/e87f4778-4d9a-4e2b-87f4-5bea8bc17156-util\") pod \"e87f4778-4d9a-4e2b-87f4-5bea8bc17156\" (UID: \"e87f4778-4d9a-4e2b-87f4-5bea8bc17156\") "
Nov 02 02:51:58.177270 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:58.177241    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e87f4778-4d9a-4e2b-87f4-5bea8bc17156-bundle\" (UniqueName: \"kubernetes.io/empty-dir/e87f4778-4d9a-4e2b-87f4-5bea8bc17156-bundle\") pod \"e87f4778-4d9a-4e2b-87f4-5bea8bc17156\" (UID: \"e87f4778-4d9a-4e2b-87f4-5bea8bc17156\") "
Nov 02 02:51:58.177906 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:58.177281    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae3f087c-5607-44c2-aa1b-f616cc5def3d-kube-api-access-snrwm\" (UniqueName: \"kubernetes.io/projected/ae3f087c-5607-44c2-aa1b-f616cc5def3d-kube-api-access-snrwm\") pod \"ae3f087c-5607-44c2-aa1b-f616cc5def3d\" (UID: \"ae3f087c-5607-44c2-aa1b-f616cc5def3d\") "
Nov 02 02:51:58.177906 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:58.177301    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae3f087c-5607-44c2-aa1b-f616cc5def3d-util\" (UniqueName: \"kubernetes.io/empty-dir/ae3f087c-5607-44c2-aa1b-f616cc5def3d-util\") pod \"ae3f087c-5607-44c2-aa1b-f616cc5def3d\" (UID: \"ae3f087c-5607-44c2-aa1b-f616cc5def3d\") "
Nov 02 02:51:58.177906 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:58.177362    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae3f087c-5607-44c2-aa1b-f616cc5def3d-bundle\" (UniqueName: \"kubernetes.io/empty-dir/ae3f087c-5607-44c2-aa1b-f616cc5def3d-bundle\") pod \"ae3f087c-5607-44c2-aa1b-f616cc5def3d\" (UID: \"ae3f087c-5607-44c2-aa1b-f616cc5def3d\") "
Nov 02 02:51:58.177906 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:58.177403    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e87f4778-4d9a-4e2b-87f4-5bea8bc17156-kube-api-access-5zb4s\" (UniqueName: \"kubernetes.io/projected/e87f4778-4d9a-4e2b-87f4-5bea8bc17156-kube-api-access-5zb4s\") pod \"e87f4778-4d9a-4e2b-87f4-5bea8bc17156\" (UID: \"e87f4778-4d9a-4e2b-87f4-5bea8bc17156\") "
Nov 02 02:51:58.177906 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:58.177618    4496 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/62c5db55-67e4-4be7-bf51-d6ce0d45303d-util\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:51:58.177906 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:58.177638    4496 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/62c5db55-67e4-4be7-bf51-d6ce0d45303d-bundle\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:51:58.177906 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:58.177653    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kz2l\" (UniqueName: \"kubernetes.io/projected/62c5db55-67e4-4be7-bf51-d6ce0d45303d-kube-api-access-8kz2l\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:51:58.178285 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:58.177991    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e87f4778-4d9a-4e2b-87f4-5bea8bc17156-bundle" pod "e87f4778-4d9a-4e2b-87f4-5bea8bc17156" (UID: "e87f4778-4d9a-4e2b-87f4-5bea8bc17156"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:51:58.178285 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:58.178245    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae3f087c-5607-44c2-aa1b-f616cc5def3d-bundle" pod "ae3f087c-5607-44c2-aa1b-f616cc5def3d" (UID: "ae3f087c-5607-44c2-aa1b-f616cc5def3d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:51:58.179660 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:58.179629    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e87f4778-4d9a-4e2b-87f4-5bea8bc17156-kube-api-access-5zb4s" pod "e87f4778-4d9a-4e2b-87f4-5bea8bc17156" (UID: "e87f4778-4d9a-4e2b-87f4-5bea8bc17156"). InnerVolumeSpecName "kube-api-access-5zb4s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:51:58.179791 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:58.179770    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae3f087c-5607-44c2-aa1b-f616cc5def3d-kube-api-access-snrwm" pod "ae3f087c-5607-44c2-aa1b-f616cc5def3d" (UID: "ae3f087c-5607-44c2-aa1b-f616cc5def3d"). InnerVolumeSpecName "kube-api-access-snrwm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:51:58.182466 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:58.182441    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e87f4778-4d9a-4e2b-87f4-5bea8bc17156-util" pod "e87f4778-4d9a-4e2b-87f4-5bea8bc17156" (UID: "e87f4778-4d9a-4e2b-87f4-5bea8bc17156"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:51:58.183631 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:58.183607    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae3f087c-5607-44c2-aa1b-f616cc5def3d-util" pod "ae3f087c-5607-44c2-aa1b-f616cc5def3d" (UID: "ae3f087c-5607-44c2-aa1b-f616cc5def3d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:51:58.278385 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:58.278347    4496 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e87f4778-4d9a-4e2b-87f4-5bea8bc17156-util\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:51:58.278385 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:58.278379    4496 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e87f4778-4d9a-4e2b-87f4-5bea8bc17156-bundle\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:51:58.278585 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:58.278393    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snrwm\" (UniqueName: \"kubernetes.io/projected/ae3f087c-5607-44c2-aa1b-f616cc5def3d-kube-api-access-snrwm\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:51:58.278585 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:58.278406    4496 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ae3f087c-5607-44c2-aa1b-f616cc5def3d-util\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:51:58.278585 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:58.278420    4496 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ae3f087c-5607-44c2-aa1b-f616cc5def3d-bundle\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:51:58.278585 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:58.278431    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zb4s\" (UniqueName: \"kubernetes.io/projected/e87f4778-4d9a-4e2b-87f4-5bea8bc17156-kube-api-access-5zb4s\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:51:58.903926 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:58.903900    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2jh6l"
Nov 02 02:51:58.904408 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:58.903897    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2jh6l" event={"ID":"62c5db55-67e4-4be7-bf51-d6ce0d45303d","Type":"ContainerDied","Data":"5b1bb1b07a30b477996fb2752f1d1fad5e79815d477de1384d3d36792d612182"}
Nov 02 02:51:58.904408 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:58.904030    4496 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5b1bb1b07a30b477996fb2752f1d1fad5e79815d477de1384d3d36792d612182"
Nov 02 02:51:58.905929 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:58.905898    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwz5jl"
Nov 02 02:51:58.905929 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:58.905914    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwz5jl" event={"ID":"e87f4778-4d9a-4e2b-87f4-5bea8bc17156","Type":"ContainerDied","Data":"e4f7f318a67e448d9db435353ebb738a9e7150b5b4a94f94c8ad866bac694154"}
Nov 02 02:51:58.906144 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:58.905956    4496 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e4f7f318a67e448d9db435353ebb738a9e7150b5b4a94f94c8ad866bac694154"
Nov 02 02:51:58.908238 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:58.908213    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbbbct" event={"ID":"ae3f087c-5607-44c2-aa1b-f616cc5def3d","Type":"ContainerDied","Data":"cc2acaa5592e74d4493c2a3d74ae64e1fd1b10d244367245a4ed4498556a3b4f"}
Nov 02 02:51:58.908238 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:58.908228    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbbbct"
Nov 02 02:51:58.908392 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:58.908255    4496 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cc2acaa5592e74d4493c2a3d74ae64e1fd1b10d244367245a4ed4498556a3b4f"
Nov 02 02:51:59.048250 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:59.048185    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5b55749cb7-qmk67"]
Nov 02 02:51:59.048941 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:59.048924    4496 cpu_manager.go:414] "containerMap: removing container" podUID="7944df8d-53d3-4c2b-b243-20bd43467d36" containerName="util"
Nov 02 02:51:59.048941 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:59.048939    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7944df8d-53d3-4c2b-b243-20bd43467d36" containerName="util"
Nov 02 02:51:59.049124 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:59.048950    4496 cpu_manager.go:414] "containerMap: removing container" podUID="62c5db55-67e4-4be7-bf51-d6ce0d45303d" containerName="util"
Nov 02 02:51:59.049124 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:59.048956    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62c5db55-67e4-4be7-bf51-d6ce0d45303d" containerName="util"
Nov 02 02:51:59.049124 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:59.048963    4496 cpu_manager.go:414] "containerMap: removing container" podUID="e87f4778-4d9a-4e2b-87f4-5bea8bc17156" containerName="extract"
Nov 02 02:51:59.049124 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:59.048968    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e87f4778-4d9a-4e2b-87f4-5bea8bc17156" containerName="extract"
Nov 02 02:51:59.049124 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:59.048979    4496 cpu_manager.go:414] "containerMap: removing container" podUID="e87f4778-4d9a-4e2b-87f4-5bea8bc17156" containerName="util"
Nov 02 02:51:59.049124 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:59.048984    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e87f4778-4d9a-4e2b-87f4-5bea8bc17156" containerName="util"
Nov 02 02:51:59.049124 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:59.048990    4496 cpu_manager.go:414] "containerMap: removing container" podUID="72665a20-d25f-4519-963b-94f562ad524d" containerName="extract"
Nov 02 02:51:59.049124 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:59.048994    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72665a20-d25f-4519-963b-94f562ad524d" containerName="extract"
Nov 02 02:51:59.049124 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:59.049004    4496 cpu_manager.go:414] "containerMap: removing container" podUID="629a2e2b-9caa-4544-9653-a612a79d065f" containerName="pull"
Nov 02 02:51:59.049124 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:59.049008    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="629a2e2b-9caa-4544-9653-a612a79d065f" containerName="pull"
Nov 02 02:51:59.049124 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:59.049034    4496 cpu_manager.go:414] "containerMap: removing container" podUID="72665a20-d25f-4519-963b-94f562ad524d" containerName="pull"
Nov 02 02:51:59.049124 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:59.049039    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72665a20-d25f-4519-963b-94f562ad524d" containerName="pull"
Nov 02 02:51:59.049124 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:59.049046    4496 cpu_manager.go:414] "containerMap: removing container" podUID="45374d9a-e4d9-49aa-9fde-f0d9e8ff3bd3" containerName="extract"
Nov 02 02:51:59.049124 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:59.049051    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45374d9a-e4d9-49aa-9fde-f0d9e8ff3bd3" containerName="extract"
Nov 02 02:51:59.049124 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:59.049057    4496 cpu_manager.go:414] "containerMap: removing container" podUID="ae3f087c-5607-44c2-aa1b-f616cc5def3d" containerName="extract"
Nov 02 02:51:59.049124 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:59.049061    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae3f087c-5607-44c2-aa1b-f616cc5def3d" containerName="extract"
Nov 02 02:51:59.049124 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:59.049067    4496 cpu_manager.go:414] "containerMap: removing container" podUID="629a2e2b-9caa-4544-9653-a612a79d065f" containerName="extract"
Nov 02 02:51:59.049124 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:59.049071    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="629a2e2b-9caa-4544-9653-a612a79d065f" containerName="extract"
Nov 02 02:51:59.049124 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:59.049077    4496 cpu_manager.go:414] "containerMap: removing container" podUID="7944df8d-53d3-4c2b-b243-20bd43467d36" containerName="pull"
Nov 02 02:51:59.049124 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:59.049082    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7944df8d-53d3-4c2b-b243-20bd43467d36" containerName="pull"
Nov 02 02:51:59.049124 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:59.049088    4496 cpu_manager.go:414] "containerMap: removing container" podUID="62c5db55-67e4-4be7-bf51-d6ce0d45303d" containerName="extract"
Nov 02 02:51:59.049124 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:59.049092    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62c5db55-67e4-4be7-bf51-d6ce0d45303d" containerName="extract"
Nov 02 02:51:59.049124 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:59.049098    4496 cpu_manager.go:414] "containerMap: removing container" podUID="7944df8d-53d3-4c2b-b243-20bd43467d36" containerName="extract"
Nov 02 02:51:59.049124 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:59.049102    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7944df8d-53d3-4c2b-b243-20bd43467d36" containerName="extract"
Nov 02 02:51:59.049124 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:59.049109    4496 cpu_manager.go:414] "containerMap: removing container" podUID="ae3f087c-5607-44c2-aa1b-f616cc5def3d" containerName="pull"
Nov 02 02:51:59.049124 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:59.049113    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae3f087c-5607-44c2-aa1b-f616cc5def3d" containerName="pull"
Nov 02 02:51:59.049124 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:59.049122    4496 cpu_manager.go:414] "containerMap: removing container" podUID="62c5db55-67e4-4be7-bf51-d6ce0d45303d" containerName="pull"
Nov 02 02:51:59.049124 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:59.049126    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62c5db55-67e4-4be7-bf51-d6ce0d45303d" containerName="pull"
Nov 02 02:51:59.049124 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:59.049132    4496 cpu_manager.go:414] "containerMap: removing container" podUID="45374d9a-e4d9-49aa-9fde-f0d9e8ff3bd3" containerName="util"
Nov 02 02:51:59.049124 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:59.049137    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45374d9a-e4d9-49aa-9fde-f0d9e8ff3bd3" containerName="util"
Nov 02 02:51:59.049124 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:59.049144    4496 cpu_manager.go:414] "containerMap: removing container" podUID="45374d9a-e4d9-49aa-9fde-f0d9e8ff3bd3" containerName="pull"
Nov 02 02:51:59.050451 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:59.049149    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45374d9a-e4d9-49aa-9fde-f0d9e8ff3bd3" containerName="pull"
Nov 02 02:51:59.050451 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:59.049155    4496 cpu_manager.go:414] "containerMap: removing container" podUID="72665a20-d25f-4519-963b-94f562ad524d" containerName="util"
Nov 02 02:51:59.050451 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:59.049160    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72665a20-d25f-4519-963b-94f562ad524d" containerName="util"
Nov 02 02:51:59.050451 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:59.049165    4496 cpu_manager.go:414] "containerMap: removing container" podUID="e87f4778-4d9a-4e2b-87f4-5bea8bc17156" containerName="pull"
Nov 02 02:51:59.050451 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:59.049170    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e87f4778-4d9a-4e2b-87f4-5bea8bc17156" containerName="pull"
Nov 02 02:51:59.050451 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:59.049178    4496 cpu_manager.go:414] "containerMap: removing container" podUID="ae3f087c-5607-44c2-aa1b-f616cc5def3d" containerName="util"
Nov 02 02:51:59.050451 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:59.049182    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae3f087c-5607-44c2-aa1b-f616cc5def3d" containerName="util"
Nov 02 02:51:59.050451 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:59.049188    4496 cpu_manager.go:414] "containerMap: removing container" podUID="629a2e2b-9caa-4544-9653-a612a79d065f" containerName="util"
Nov 02 02:51:59.050451 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:59.049192    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="629a2e2b-9caa-4544-9653-a612a79d065f" containerName="util"
Nov 02 02:51:59.050451 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:59.049239    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="72665a20-d25f-4519-963b-94f562ad524d" containerName="extract"
Nov 02 02:51:59.050451 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:59.049248    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae3f087c-5607-44c2-aa1b-f616cc5def3d" containerName="extract"
Nov 02 02:51:59.050451 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:59.049254    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="45374d9a-e4d9-49aa-9fde-f0d9e8ff3bd3" containerName="extract"
Nov 02 02:51:59.050451 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:59.049258    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="629a2e2b-9caa-4544-9653-a612a79d065f" containerName="extract"
Nov 02 02:51:59.050451 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:59.049264    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="62c5db55-67e4-4be7-bf51-d6ce0d45303d" containerName="extract"
Nov 02 02:51:59.050451 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:59.049269    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="7944df8d-53d3-4c2b-b243-20bd43467d36" containerName="extract"
Nov 02 02:51:59.050451 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:59.049277    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="e87f4778-4d9a-4e2b-87f4-5bea8bc17156" containerName="extract"
Nov 02 02:51:59.051815 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:59.051801    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b55749cb7-qmk67"
Nov 02 02:51:59.086260 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:59.086234    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/56ebc10e-dbea-45e9-8bb0-a329209832a6-metrics-cert\") pod \"opendatahub-operator-controller-manager-5b55749cb7-qmk67\" (UID: \"56ebc10e-dbea-45e9-8bb0-a329209832a6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b55749cb7-qmk67"
Nov 02 02:51:59.086369 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:59.086276    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/56ebc10e-dbea-45e9-8bb0-a329209832a6-manifests\") pod \"opendatahub-operator-controller-manager-5b55749cb7-qmk67\" (UID: \"56ebc10e-dbea-45e9-8bb0-a329209832a6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b55749cb7-qmk67"
Nov 02 02:51:59.086369 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:59.086292    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/56ebc10e-dbea-45e9-8bb0-a329209832a6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5b55749cb7-qmk67\" (UID: \"56ebc10e-dbea-45e9-8bb0-a329209832a6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b55749cb7-qmk67"
Nov 02 02:51:59.086486 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:59.086424    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9z55\" (UniqueName: \"kubernetes.io/projected/56ebc10e-dbea-45e9-8bb0-a329209832a6-kube-api-access-m9z55\") pod \"opendatahub-operator-controller-manager-5b55749cb7-qmk67\" (UID: \"56ebc10e-dbea-45e9-8bb0-a329209832a6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b55749cb7-qmk67"
Nov 02 02:51:59.086486 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:59.086473    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/56ebc10e-dbea-45e9-8bb0-a329209832a6-webhook-cert\") pod \"opendatahub-operator-controller-manager-5b55749cb7-qmk67\" (UID: \"56ebc10e-dbea-45e9-8bb0-a329209832a6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b55749cb7-qmk67"
Nov 02 02:51:59.086566 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:59.086547    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/56ebc10e-dbea-45e9-8bb0-a329209832a6-tmp\") pod \"opendatahub-operator-controller-manager-5b55749cb7-qmk67\" (UID: \"56ebc10e-dbea-45e9-8bb0-a329209832a6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b55749cb7-qmk67"
Nov 02 02:51:59.108828 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:59.108803    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Nov 02 02:51:59.108828 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:59.108818    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Nov 02 02:51:59.124278 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:59.124253    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Nov 02 02:51:59.124393 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:59.124375    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Nov 02 02:51:59.124558 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:59.124531    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-2r6bp\""
Nov 02 02:51:59.124681 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:59.124664    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Nov 02 02:51:59.187353 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:59.187290    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/56ebc10e-dbea-45e9-8bb0-a329209832a6-metrics-cert\") pod \"opendatahub-operator-controller-manager-5b55749cb7-qmk67\" (UID: \"56ebc10e-dbea-45e9-8bb0-a329209832a6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b55749cb7-qmk67"
Nov 02 02:51:59.187353 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:59.187335    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/56ebc10e-dbea-45e9-8bb0-a329209832a6-manifests\") pod \"opendatahub-operator-controller-manager-5b55749cb7-qmk67\" (UID: \"56ebc10e-dbea-45e9-8bb0-a329209832a6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b55749cb7-qmk67"
Nov 02 02:51:59.187353 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:59.187353    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/56ebc10e-dbea-45e9-8bb0-a329209832a6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5b55749cb7-qmk67\" (UID: \"56ebc10e-dbea-45e9-8bb0-a329209832a6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b55749cb7-qmk67"
Nov 02 02:51:59.187572 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:59.187454    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9z55\" (UniqueName: \"kubernetes.io/projected/56ebc10e-dbea-45e9-8bb0-a329209832a6-kube-api-access-m9z55\") pod \"opendatahub-operator-controller-manager-5b55749cb7-qmk67\" (UID: \"56ebc10e-dbea-45e9-8bb0-a329209832a6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b55749cb7-qmk67"
Nov 02 02:51:59.187572 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:59.187485    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/56ebc10e-dbea-45e9-8bb0-a329209832a6-webhook-cert\") pod \"opendatahub-operator-controller-manager-5b55749cb7-qmk67\" (UID: \"56ebc10e-dbea-45e9-8bb0-a329209832a6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b55749cb7-qmk67"
Nov 02 02:51:59.187572 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:59.187527    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/56ebc10e-dbea-45e9-8bb0-a329209832a6-tmp\") pod \"opendatahub-operator-controller-manager-5b55749cb7-qmk67\" (UID: \"56ebc10e-dbea-45e9-8bb0-a329209832a6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b55749cb7-qmk67"
Nov 02 02:51:59.187754 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:59.187725    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/56ebc10e-dbea-45e9-8bb0-a329209832a6-manifests\") pod \"opendatahub-operator-controller-manager-5b55749cb7-qmk67\" (UID: \"56ebc10e-dbea-45e9-8bb0-a329209832a6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b55749cb7-qmk67"
Nov 02 02:51:59.189760 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:59.189732    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/56ebc10e-dbea-45e9-8bb0-a329209832a6-tmp\") pod \"opendatahub-operator-controller-manager-5b55749cb7-qmk67\" (UID: \"56ebc10e-dbea-45e9-8bb0-a329209832a6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b55749cb7-qmk67"
Nov 02 02:51:59.189868 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:59.189831    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/56ebc10e-dbea-45e9-8bb0-a329209832a6-metrics-cert\") pod \"opendatahub-operator-controller-manager-5b55749cb7-qmk67\" (UID: \"56ebc10e-dbea-45e9-8bb0-a329209832a6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b55749cb7-qmk67"
Nov 02 02:51:59.189868 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:59.189847    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/56ebc10e-dbea-45e9-8bb0-a329209832a6-webhook-cert\") pod \"opendatahub-operator-controller-manager-5b55749cb7-qmk67\" (UID: \"56ebc10e-dbea-45e9-8bb0-a329209832a6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b55749cb7-qmk67"
Nov 02 02:51:59.189952 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:59.189939    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/56ebc10e-dbea-45e9-8bb0-a329209832a6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5b55749cb7-qmk67\" (UID: \"56ebc10e-dbea-45e9-8bb0-a329209832a6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b55749cb7-qmk67"
Nov 02 02:51:59.201648 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:59.201625    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5b55749cb7-qmk67"]
Nov 02 02:51:59.253785 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:59.253756    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9z55\" (UniqueName: \"kubernetes.io/projected/56ebc10e-dbea-45e9-8bb0-a329209832a6-kube-api-access-m9z55\") pod \"opendatahub-operator-controller-manager-5b55749cb7-qmk67\" (UID: \"56ebc10e-dbea-45e9-8bb0-a329209832a6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b55749cb7-qmk67"
Nov 02 02:51:59.364528 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:59.364500    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b55749cb7-qmk67"
Nov 02 02:51:59.850684 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:59.850659    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5b55749cb7-qmk67"]
Nov 02 02:51:59.852379 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:51:59.852352    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56ebc10e_dbea_45e9_8bb0_a329209832a6.slice/crio-eeb107e7831144a1fe4cbb188d6f4812160d10a26a61333be91589cad1ca9fcf WatchSource:0}: Error finding container eeb107e7831144a1fe4cbb188d6f4812160d10a26a61333be91589cad1ca9fcf: Status 404 returned error can't find the container with id eeb107e7831144a1fe4cbb188d6f4812160d10a26a61333be91589cad1ca9fcf
Nov 02 02:51:59.914638 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:51:59.914576    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b55749cb7-qmk67" event={"ID":"56ebc10e-dbea-45e9-8bb0-a329209832a6","Type":"ContainerStarted","Data":"eeb107e7831144a1fe4cbb188d6f4812160d10a26a61333be91589cad1ca9fcf"}
Nov 02 02:52:03.528826 ip-10-0-133-163 kubenswrapper[4496]: E1102 02:52:03.528791    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 02:52:03.941317 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:03.941229    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b55749cb7-qmk67" event={"ID":"56ebc10e-dbea-45e9-8bb0-a329209832a6","Type":"ContainerStarted","Data":"da3890254b7aa765381dc50d0d448934649c20e3ae4624ef19e792d74c4f57f6"}
Nov 02 02:52:04.947491 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:04.947457    4496 generic.go:356] "Generic (PLEG): container finished" podID="56ebc10e-dbea-45e9-8bb0-a329209832a6" containerID="da3890254b7aa765381dc50d0d448934649c20e3ae4624ef19e792d74c4f57f6" exitCode=0
Nov 02 02:52:04.947882 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:04.947520    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b55749cb7-qmk67" event={"ID":"56ebc10e-dbea-45e9-8bb0-a329209832a6","Type":"ContainerDied","Data":"da3890254b7aa765381dc50d0d448934649c20e3ae4624ef19e792d74c4f57f6"}
Nov 02 02:52:05.954346 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:05.954314    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b55749cb7-qmk67" event={"ID":"56ebc10e-dbea-45e9-8bb0-a329209832a6","Type":"ContainerStarted","Data":"f0c012dff2301a9806ef2806fcc58ef2de271499c6567cb913d5d5ce6b4f2b1b"}
Nov 02 02:52:05.954717 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:05.954389    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b55749cb7-qmk67"
Nov 02 02:52:06.183480 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:06.183377    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b55749cb7-qmk67" podStartSLOduration=3.116061003 podStartE2EDuration="8.183361329s" totalImagesPullingTime="4.568841021s" totalInitContainerRuntime="498.459305ms" isStatefulPod=false podCreationTimestamp="2026-11-02 02:51:58 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:52:06.132461654 +0000 UTC m=+266.711069072" watchObservedRunningTime="2026-11-02 02:52:06.183361329 +0000 UTC m=+266.761968709"
Nov 02 02:52:08.650147 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:08.650114    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-xgxnp"]
Nov 02 02:52:08.652427 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:08.652408    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-xgxnp"
Nov 02 02:52:08.756479 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:08.756440    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Nov 02 02:52:08.756479 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:08.756464    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-7gb6j\""
Nov 02 02:52:08.756479 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:08.756477    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Nov 02 02:52:08.756716 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:08.756622    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Nov 02 02:52:08.758349 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:08.758331    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-xgxnp"]
Nov 02 02:52:08.771898 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:08.771872    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7r4n\" (UniqueName: \"kubernetes.io/projected/a8b89697-14b6-4df5-a8d0-4c195599a4b4-kube-api-access-r7r4n\") pod \"dns-operator-controller-manager-8557cbdb84-xgxnp\" (UID: \"a8b89697-14b6-4df5-a8d0-4c195599a4b4\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-xgxnp"
Nov 02 02:52:08.872463 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:08.872441    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7r4n\" (UniqueName: \"kubernetes.io/projected/a8b89697-14b6-4df5-a8d0-4c195599a4b4-kube-api-access-r7r4n\") pod \"dns-operator-controller-manager-8557cbdb84-xgxnp\" (UID: \"a8b89697-14b6-4df5-a8d0-4c195599a4b4\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-xgxnp"
Nov 02 02:52:08.931984 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:08.931927    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7r4n\" (UniqueName: \"kubernetes.io/projected/a8b89697-14b6-4df5-a8d0-4c195599a4b4-kube-api-access-r7r4n\") pod \"dns-operator-controller-manager-8557cbdb84-xgxnp\" (UID: \"a8b89697-14b6-4df5-a8d0-4c195599a4b4\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-xgxnp"
Nov 02 02:52:08.965482 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:08.965462    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-xgxnp"
Nov 02 02:52:09.334848 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:09.334822    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-xgxnp"]
Nov 02 02:52:09.337678 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:52:09.337647    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8b89697_14b6_4df5_a8d0_4c195599a4b4.slice/crio-73c37c1f1bcd93d196e06102249d9034e230e0d65098c3d8bf9c6dc064d25886 WatchSource:0}: Error finding container 73c37c1f1bcd93d196e06102249d9034e230e0d65098c3d8bf9c6dc064d25886: Status 404 returned error can't find the container with id 73c37c1f1bcd93d196e06102249d9034e230e0d65098c3d8bf9c6dc064d25886
Nov 02 02:52:09.357289 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:09.357262    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-9587p"]
Nov 02 02:52:09.361075 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:09.361059    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-9587p"
Nov 02 02:52:09.414142 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:09.414119    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\""
Nov 02 02:52:09.414254 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:09.414180    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\""
Nov 02 02:52:09.414329 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:09.414265    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-lshq5\""
Nov 02 02:52:09.444455 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:09.444432    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-9587p"]
Nov 02 02:52:09.477668 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:09.477641    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwsrt\" (UniqueName: \"kubernetes.io/projected/e4e26dc7-747d-49d0-b5aa-f6a8d22c5a31-kube-api-access-rwsrt\") pod \"jobset-operator-674cd59dc5-9587p\" (UID: \"e4e26dc7-747d-49d0-b5aa-f6a8d22c5a31\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-9587p"
Nov 02 02:52:09.477862 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:09.477724    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e4e26dc7-747d-49d0-b5aa-f6a8d22c5a31-tmp\") pod \"jobset-operator-674cd59dc5-9587p\" (UID: \"e4e26dc7-747d-49d0-b5aa-f6a8d22c5a31\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-9587p"
Nov 02 02:52:09.578368 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:09.578335    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwsrt\" (UniqueName: \"kubernetes.io/projected/e4e26dc7-747d-49d0-b5aa-f6a8d22c5a31-kube-api-access-rwsrt\") pod \"jobset-operator-674cd59dc5-9587p\" (UID: \"e4e26dc7-747d-49d0-b5aa-f6a8d22c5a31\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-9587p"
Nov 02 02:52:09.578510 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:09.578394    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e4e26dc7-747d-49d0-b5aa-f6a8d22c5a31-tmp\") pod \"jobset-operator-674cd59dc5-9587p\" (UID: \"e4e26dc7-747d-49d0-b5aa-f6a8d22c5a31\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-9587p"
Nov 02 02:52:09.578761 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:09.578743    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e4e26dc7-747d-49d0-b5aa-f6a8d22c5a31-tmp\") pod \"jobset-operator-674cd59dc5-9587p\" (UID: \"e4e26dc7-747d-49d0-b5aa-f6a8d22c5a31\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-9587p"
Nov 02 02:52:09.635912 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:09.635843    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwsrt\" (UniqueName: \"kubernetes.io/projected/e4e26dc7-747d-49d0-b5aa-f6a8d22c5a31-kube-api-access-rwsrt\") pod \"jobset-operator-674cd59dc5-9587p\" (UID: \"e4e26dc7-747d-49d0-b5aa-f6a8d22c5a31\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-9587p"
Nov 02 02:52:09.684507 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:09.684477    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-9587p"
Nov 02 02:52:09.973469 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:09.973388    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-xgxnp" event={"ID":"a8b89697-14b6-4df5-a8d0-4c195599a4b4","Type":"ContainerStarted","Data":"73c37c1f1bcd93d196e06102249d9034e230e0d65098c3d8bf9c6dc064d25886"}
Nov 02 02:52:10.057780 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:10.057756    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-9587p"]
Nov 02 02:52:10.059509 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:52:10.059481    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode4e26dc7_747d_49d0_b5aa_f6a8d22c5a31.slice/crio-da4d0adf4e83fced94a5f764c1a945597b121c6878aad1fe563d347529ae1f45 WatchSource:0}: Error finding container da4d0adf4e83fced94a5f764c1a945597b121c6878aad1fe563d347529ae1f45: Status 404 returned error can't find the container with id da4d0adf4e83fced94a5f764c1a945597b121c6878aad1fe563d347529ae1f45
Nov 02 02:52:10.979329 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:10.979294    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-9587p" event={"ID":"e4e26dc7-747d-49d0-b5aa-f6a8d22c5a31","Type":"ContainerStarted","Data":"da4d0adf4e83fced94a5f764c1a945597b121c6878aad1fe563d347529ae1f45"}
Nov 02 02:52:13.816223 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:13.816189    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-tlh72"]
Nov 02 02:52:13.818696 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:13.818676    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-tlh72"
Nov 02 02:52:13.874934 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:13.874912    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-pmdpv\""
Nov 02 02:52:13.914701 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:13.914673    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcpgg\" (UniqueName: \"kubernetes.io/projected/475f7114-488a-487c-98f7-316ef5b2a6dd-kube-api-access-gcpgg\") pod \"authorino-operator-7b7bb4bc77-tlh72\" (UID: \"475f7114-488a-487c-98f7-316ef5b2a6dd\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-tlh72"
Nov 02 02:52:13.927367 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:13.927342    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-tlh72"]
Nov 02 02:52:14.002066 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:14.002007    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-xgxnp" event={"ID":"a8b89697-14b6-4df5-a8d0-4c195599a4b4","Type":"ContainerStarted","Data":"edab98ab79386f04855b6a53eaab12379f9d415aa1e4c74a0197b950cf29e020"}
Nov 02 02:52:14.002233 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:14.002105    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-xgxnp"
Nov 02 02:52:14.015304 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:14.015281    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcpgg\" (UniqueName: \"kubernetes.io/projected/475f7114-488a-487c-98f7-316ef5b2a6dd-kube-api-access-gcpgg\") pod \"authorino-operator-7b7bb4bc77-tlh72\" (UID: \"475f7114-488a-487c-98f7-316ef5b2a6dd\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-tlh72"
Nov 02 02:52:14.075709 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:14.075642    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcpgg\" (UniqueName: \"kubernetes.io/projected/475f7114-488a-487c-98f7-316ef5b2a6dd-kube-api-access-gcpgg\") pod \"authorino-operator-7b7bb4bc77-tlh72\" (UID: \"475f7114-488a-487c-98f7-316ef5b2a6dd\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-tlh72"
Nov 02 02:52:14.122122 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:14.122064    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-xgxnp" podStartSLOduration=2.360629824 podStartE2EDuration="6.12205089s" totalImagesPullingTime="3.761421066s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:52:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:52:14.121324921 +0000 UTC m=+274.699932302" watchObservedRunningTime="2026-11-02 02:52:14.12205089 +0000 UTC m=+274.700658270"
Nov 02 02:52:14.132719 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:14.132692    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-tlh72"
Nov 02 02:52:14.510494 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:14.510470    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-tlh72"]
Nov 02 02:52:14.513307 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:52:14.513267    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod475f7114_488a_487c_98f7_316ef5b2a6dd.slice/crio-c88ad4ff9bd22fca8f54e4a04c7ffa3c28f13d639259d38c489ea3cdcdddc6ba WatchSource:0}: Error finding container c88ad4ff9bd22fca8f54e4a04c7ffa3c28f13d639259d38c489ea3cdcdddc6ba: Status 404 returned error can't find the container with id c88ad4ff9bd22fca8f54e4a04c7ffa3c28f13d639259d38c489ea3cdcdddc6ba
Nov 02 02:52:15.008331 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:15.008291    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-9587p" event={"ID":"e4e26dc7-747d-49d0-b5aa-f6a8d22c5a31","Type":"ContainerStarted","Data":"053ee4693a256d704d141da8ab2e0a15a4a84c96c632634574a7fb00491859d6"}
Nov 02 02:52:15.009720 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:15.009691    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-tlh72" event={"ID":"475f7114-488a-487c-98f7-316ef5b2a6dd","Type":"ContainerStarted","Data":"c88ad4ff9bd22fca8f54e4a04c7ffa3c28f13d639259d38c489ea3cdcdddc6ba"}
Nov 02 02:52:15.133050 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:15.131456    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-9587p" podStartSLOduration=1.5788815619999998 podStartE2EDuration="6.131436179s" totalImagesPullingTime="4.552554617s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:52:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:52:15.129287842 +0000 UTC m=+275.707895223" watchObservedRunningTime="2026-11-02 02:52:15.131436179 +0000 UTC m=+275.710043560"
Nov 02 02:52:16.167096 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:16.167061    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-sq8jf"]
Nov 02 02:52:16.169354 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:16.169337    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-sq8jf"
Nov 02 02:52:16.221615 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:16.221590    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-sq8jf"]
Nov 02 02:52:16.233434 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:16.233406    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d6e477a2-6f9a-4caa-8212-75c16c2c135e-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-sq8jf\" (UID: \"d6e477a2-6f9a-4caa-8212-75c16c2c135e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-sq8jf"
Nov 02 02:52:16.233528 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:16.233498    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvwdd\" (UniqueName: \"kubernetes.io/projected/d6e477a2-6f9a-4caa-8212-75c16c2c135e-kube-api-access-bvwdd\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-sq8jf\" (UID: \"d6e477a2-6f9a-4caa-8212-75c16c2c135e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-sq8jf"
Nov 02 02:52:16.270533 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:16.270501    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Nov 02 02:52:16.270533 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:16.270521    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-vc5nx\""
Nov 02 02:52:16.270686 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:16.270562    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Nov 02 02:52:16.334782 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:16.334757    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvwdd\" (UniqueName: \"kubernetes.io/projected/d6e477a2-6f9a-4caa-8212-75c16c2c135e-kube-api-access-bvwdd\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-sq8jf\" (UID: \"d6e477a2-6f9a-4caa-8212-75c16c2c135e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-sq8jf"
Nov 02 02:52:16.334896 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:16.334790    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d6e477a2-6f9a-4caa-8212-75c16c2c135e-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-sq8jf\" (UID: \"d6e477a2-6f9a-4caa-8212-75c16c2c135e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-sq8jf"
Nov 02 02:52:16.335229 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:16.335212    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d6e477a2-6f9a-4caa-8212-75c16c2c135e-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-sq8jf\" (UID: \"d6e477a2-6f9a-4caa-8212-75c16c2c135e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-sq8jf"
Nov 02 02:52:16.403603 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:16.403583    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvwdd\" (UniqueName: \"kubernetes.io/projected/d6e477a2-6f9a-4caa-8212-75c16c2c135e-kube-api-access-bvwdd\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-sq8jf\" (UID: \"d6e477a2-6f9a-4caa-8212-75c16c2c135e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-sq8jf"
Nov 02 02:52:16.479333 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:16.479306    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-sq8jf"
Nov 02 02:52:16.962113 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:16.962034    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b55749cb7-qmk67"
Nov 02 02:52:16.968437 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:52:16.967214    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6e477a2_6f9a_4caa_8212_75c16c2c135e.slice/crio-e56b87247e7f6b59c852096040973a9d044e401df905a9f6daf65ef21d66f51d WatchSource:0}: Error finding container e56b87247e7f6b59c852096040973a9d044e401df905a9f6daf65ef21d66f51d: Status 404 returned error can't find the container with id e56b87247e7f6b59c852096040973a9d044e401df905a9f6daf65ef21d66f51d
Nov 02 02:52:16.968651 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:16.968628    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-sq8jf"]
Nov 02 02:52:17.021496 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:17.021467    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-sq8jf" event={"ID":"d6e477a2-6f9a-4caa-8212-75c16c2c135e","Type":"ContainerStarted","Data":"e56b87247e7f6b59c852096040973a9d044e401df905a9f6daf65ef21d66f51d"}
Nov 02 02:52:18.028160 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:18.028125    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-tlh72" event={"ID":"475f7114-488a-487c-98f7-316ef5b2a6dd","Type":"ContainerStarted","Data":"2deaa25e73e915a118ccb539ee64c5dd4b9a57b6d422f93586f0909a9ed4eb0d"}
Nov 02 02:52:18.028651 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:18.028264    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-tlh72"
Nov 02 02:52:18.155235 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:18.155183    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-tlh72" podStartSLOduration=2.442259123 podStartE2EDuration="5.155169964s" totalImagesPullingTime="2.712910841s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:52:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:52:18.153527997 +0000 UTC m=+278.732135377" watchObservedRunningTime="2026-11-02 02:52:18.155169964 +0000 UTC m=+278.733777344"
Nov 02 02:52:20.041344 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:20.041306    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-sq8jf" event={"ID":"d6e477a2-6f9a-4caa-8212-75c16c2c135e","Type":"ContainerStarted","Data":"7bc05b92d5722286ab13cfab9b0bfadcee5d242d2cf2ff35bbc4683ec7fc94c9"}
Nov 02 02:52:20.266671 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:20.266611    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-sq8jf" podStartSLOduration=1.654958575 podStartE2EDuration="4.266598243s" totalImagesPullingTime="2.611639668s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:52:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:52:20.265005318 +0000 UTC m=+280.843612699" watchObservedRunningTime="2026-11-02 02:52:20.266598243 +0000 UTC m=+280.845205623"
Nov 02 02:52:25.012649 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:25.012617    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-xgxnp"
Nov 02 02:52:25.625215 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:25.625179    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lmphn"]
Nov 02 02:52:25.628603 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:25.628582    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lmphn"
Nov 02 02:52:25.684934 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:25.684908    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-kkl4g\""
Nov 02 02:52:25.716262 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:25.716236    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/76b2812d-c926-4aff-a519-fa327f7e4695-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-lmphn\" (UID: \"76b2812d-c926-4aff-a519-fa327f7e4695\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lmphn"
Nov 02 02:52:25.716359 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:25.716310    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7r6gm\" (UniqueName: \"kubernetes.io/projected/76b2812d-c926-4aff-a519-fa327f7e4695-kube-api-access-7r6gm\") pod \"kuadrant-operator-controller-manager-678d87766f-lmphn\" (UID: \"76b2812d-c926-4aff-a519-fa327f7e4695\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lmphn"
Nov 02 02:52:25.738966 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:25.738938    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lmphn"]
Nov 02 02:52:25.816776 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:25.816742    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7r6gm\" (UniqueName: \"kubernetes.io/projected/76b2812d-c926-4aff-a519-fa327f7e4695-kube-api-access-7r6gm\") pod \"kuadrant-operator-controller-manager-678d87766f-lmphn\" (UID: \"76b2812d-c926-4aff-a519-fa327f7e4695\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lmphn"
Nov 02 02:52:25.816925 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:25.816794    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/76b2812d-c926-4aff-a519-fa327f7e4695-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-lmphn\" (UID: \"76b2812d-c926-4aff-a519-fa327f7e4695\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lmphn"
Nov 02 02:52:25.817249 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:25.817229    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/76b2812d-c926-4aff-a519-fa327f7e4695-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-lmphn\" (UID: \"76b2812d-c926-4aff-a519-fa327f7e4695\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lmphn"
Nov 02 02:52:25.881843 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:25.881779    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7r6gm\" (UniqueName: \"kubernetes.io/projected/76b2812d-c926-4aff-a519-fa327f7e4695-kube-api-access-7r6gm\") pod \"kuadrant-operator-controller-manager-678d87766f-lmphn\" (UID: \"76b2812d-c926-4aff-a519-fa327f7e4695\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lmphn"
Nov 02 02:52:25.941224 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:25.941193    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lmphn"
Nov 02 02:52:26.368732 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:26.368710    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lmphn"]
Nov 02 02:52:26.370851 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:52:26.370825    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76b2812d_c926_4aff_a519_fa327f7e4695.slice/crio-a01b308069089b1b2222e87cbf9c474d84b7c2f60b490a62a8ba93b6abb43ed3 WatchSource:0}: Error finding container a01b308069089b1b2222e87cbf9c474d84b7c2f60b490a62a8ba93b6abb43ed3: Status 404 returned error can't find the container with id a01b308069089b1b2222e87cbf9c474d84b7c2f60b490a62a8ba93b6abb43ed3
Nov 02 02:52:27.074953 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:27.074918    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lmphn" event={"ID":"76b2812d-c926-4aff-a519-fa327f7e4695","Type":"ContainerStarted","Data":"a01b308069089b1b2222e87cbf9c474d84b7c2f60b490a62a8ba93b6abb43ed3"}
Nov 02 02:52:29.037027 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:29.036990    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-tlh72"
Nov 02 02:52:31.829636 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:31.829591    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfb8rqj"]
Nov 02 02:52:31.833490 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:31.833468    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfb8rqj"
Nov 02 02:52:31.887176 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:31.887144    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-f8bvl\""
Nov 02 02:52:31.940124 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:31.940084    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfb8rqj"]
Nov 02 02:52:31.971195 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:31.971159    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/099ee094-c411-47c4-a050-c63bcea57bf7-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfb8rqj\" (UID: \"099ee094-c411-47c4-a050-c63bcea57bf7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfb8rqj"
Nov 02 02:52:31.971336 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:31.971209    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87lwt\" (UniqueName: \"kubernetes.io/projected/099ee094-c411-47c4-a050-c63bcea57bf7-kube-api-access-87lwt\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfb8rqj\" (UID: \"099ee094-c411-47c4-a050-c63bcea57bf7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfb8rqj"
Nov 02 02:52:31.971336 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:31.971321    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/099ee094-c411-47c4-a050-c63bcea57bf7-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfb8rqj\" (UID: \"099ee094-c411-47c4-a050-c63bcea57bf7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfb8rqj"
Nov 02 02:52:31.971425 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:31.971365    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/099ee094-c411-47c4-a050-c63bcea57bf7-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfb8rqj\" (UID: \"099ee094-c411-47c4-a050-c63bcea57bf7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfb8rqj"
Nov 02 02:52:31.971470 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:31.971446    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/099ee094-c411-47c4-a050-c63bcea57bf7-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfb8rqj\" (UID: \"099ee094-c411-47c4-a050-c63bcea57bf7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfb8rqj"
Nov 02 02:52:31.971507 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:31.971495    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/099ee094-c411-47c4-a050-c63bcea57bf7-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfb8rqj\" (UID: \"099ee094-c411-47c4-a050-c63bcea57bf7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfb8rqj"
Nov 02 02:52:31.971584 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:31.971569    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/099ee094-c411-47c4-a050-c63bcea57bf7-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfb8rqj\" (UID: \"099ee094-c411-47c4-a050-c63bcea57bf7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfb8rqj"
Nov 02 02:52:31.971633 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:31.971617    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/099ee094-c411-47c4-a050-c63bcea57bf7-istio-token\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfb8rqj\" (UID: \"099ee094-c411-47c4-a050-c63bcea57bf7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfb8rqj"
Nov 02 02:52:31.971669 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:31.971649    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/099ee094-c411-47c4-a050-c63bcea57bf7-istio-data\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfb8rqj\" (UID: \"099ee094-c411-47c4-a050-c63bcea57bf7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfb8rqj"
Nov 02 02:52:32.072651 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:32.072616    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/099ee094-c411-47c4-a050-c63bcea57bf7-istio-token\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfb8rqj\" (UID: \"099ee094-c411-47c4-a050-c63bcea57bf7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfb8rqj"
Nov 02 02:52:32.072651 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:32.072653    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/099ee094-c411-47c4-a050-c63bcea57bf7-istio-data\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfb8rqj\" (UID: \"099ee094-c411-47c4-a050-c63bcea57bf7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfb8rqj"
Nov 02 02:52:32.072835 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:32.072683    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/099ee094-c411-47c4-a050-c63bcea57bf7-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfb8rqj\" (UID: \"099ee094-c411-47c4-a050-c63bcea57bf7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfb8rqj"
Nov 02 02:52:32.072835 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:32.072809    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87lwt\" (UniqueName: \"kubernetes.io/projected/099ee094-c411-47c4-a050-c63bcea57bf7-kube-api-access-87lwt\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfb8rqj\" (UID: \"099ee094-c411-47c4-a050-c63bcea57bf7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfb8rqj"
Nov 02 02:52:32.072902 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:32.072876    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/099ee094-c411-47c4-a050-c63bcea57bf7-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfb8rqj\" (UID: \"099ee094-c411-47c4-a050-c63bcea57bf7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfb8rqj"
Nov 02 02:52:32.072952 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:32.072937    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/099ee094-c411-47c4-a050-c63bcea57bf7-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfb8rqj\" (UID: \"099ee094-c411-47c4-a050-c63bcea57bf7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfb8rqj"
Nov 02 02:52:32.072994 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:32.072973    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/099ee094-c411-47c4-a050-c63bcea57bf7-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfb8rqj\" (UID: \"099ee094-c411-47c4-a050-c63bcea57bf7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfb8rqj"
Nov 02 02:52:32.073078 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:32.073037    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/099ee094-c411-47c4-a050-c63bcea57bf7-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfb8rqj\" (UID: \"099ee094-c411-47c4-a050-c63bcea57bf7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfb8rqj"
Nov 02 02:52:32.073078 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:32.073062    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/099ee094-c411-47c4-a050-c63bcea57bf7-istio-data\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfb8rqj\" (UID: \"099ee094-c411-47c4-a050-c63bcea57bf7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfb8rqj"
Nov 02 02:52:32.073190 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:32.073087    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/099ee094-c411-47c4-a050-c63bcea57bf7-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfb8rqj\" (UID: \"099ee094-c411-47c4-a050-c63bcea57bf7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfb8rqj"
Nov 02 02:52:32.073304 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:32.073274    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/099ee094-c411-47c4-a050-c63bcea57bf7-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfb8rqj\" (UID: \"099ee094-c411-47c4-a050-c63bcea57bf7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfb8rqj"
Nov 02 02:52:32.073447 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:32.073333    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/099ee094-c411-47c4-a050-c63bcea57bf7-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfb8rqj\" (UID: \"099ee094-c411-47c4-a050-c63bcea57bf7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfb8rqj"
Nov 02 02:52:32.073447 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:32.073426    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/099ee094-c411-47c4-a050-c63bcea57bf7-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfb8rqj\" (UID: \"099ee094-c411-47c4-a050-c63bcea57bf7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfb8rqj"
Nov 02 02:52:32.073639 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:32.073620    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/099ee094-c411-47c4-a050-c63bcea57bf7-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfb8rqj\" (UID: \"099ee094-c411-47c4-a050-c63bcea57bf7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfb8rqj"
Nov 02 02:52:32.074959 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:32.074936    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/099ee094-c411-47c4-a050-c63bcea57bf7-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfb8rqj\" (UID: \"099ee094-c411-47c4-a050-c63bcea57bf7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfb8rqj"
Nov 02 02:52:32.075298 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:32.075282    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/099ee094-c411-47c4-a050-c63bcea57bf7-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfb8rqj\" (UID: \"099ee094-c411-47c4-a050-c63bcea57bf7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfb8rqj"
Nov 02 02:52:32.133039 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:32.132984    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/099ee094-c411-47c4-a050-c63bcea57bf7-istio-token\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfb8rqj\" (UID: \"099ee094-c411-47c4-a050-c63bcea57bf7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfb8rqj"
Nov 02 02:52:32.133149 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:32.133133    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87lwt\" (UniqueName: \"kubernetes.io/projected/099ee094-c411-47c4-a050-c63bcea57bf7-kube-api-access-87lwt\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfb8rqj\" (UID: \"099ee094-c411-47c4-a050-c63bcea57bf7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfb8rqj"
Nov 02 02:52:32.147810 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:32.147789    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfb8rqj"
Nov 02 02:52:32.573040 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:32.572962    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfb8rqj"]
Nov 02 02:52:32.578367 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:52:32.578331    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod099ee094_c411_47c4_a050_c63bcea57bf7.slice/crio-980b49963be712103fcf8421b8c8b58a61f4f135da3acab10ab55fa9a51b7ad9 WatchSource:0}: Error finding container 980b49963be712103fcf8421b8c8b58a61f4f135da3acab10ab55fa9a51b7ad9: Status 404 returned error can't find the container with id 980b49963be712103fcf8421b8c8b58a61f4f135da3acab10ab55fa9a51b7ad9
Nov 02 02:52:32.580645 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:32.580594    4496 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 02:52:32.580766 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:32.580684    4496 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 02:52:32.580766 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:32.580732    4496 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 02:52:33.108401 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:33.108364    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfb8rqj" event={"ID":"099ee094-c411-47c4-a050-c63bcea57bf7","Type":"ContainerStarted","Data":"f5e9aacf494186cd2fb7a6d11eac9739c19568174d425eeab18357d29ebbc258"}
Nov 02 02:52:33.108873 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:33.108419    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfb8rqj" event={"ID":"099ee094-c411-47c4-a050-c63bcea57bf7","Type":"ContainerStarted","Data":"980b49963be712103fcf8421b8c8b58a61f4f135da3acab10ab55fa9a51b7ad9"}
Nov 02 02:52:33.110185 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:33.110160    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lmphn" event={"ID":"76b2812d-c926-4aff-a519-fa327f7e4695","Type":"ContainerStarted","Data":"f1f8180bd92a106678cb940c456a132a359bc706e8558f7de443f09b586fc253"}
Nov 02 02:52:33.110323 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:33.110303    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lmphn"
Nov 02 02:52:33.148198 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:33.148166    4496 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfb8rqj"
Nov 02 02:52:33.152593 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:33.152568    4496 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfb8rqj"
Nov 02 02:52:33.229732 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:33.229678    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfb8rqj" podStartSLOduration=2.229663571 podStartE2EDuration="2.229663571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:52:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:52:33.228857319 +0000 UTC m=+293.807464699" watchObservedRunningTime="2026-11-02 02:52:33.229663571 +0000 UTC m=+293.808270950"
Nov 02 02:52:33.469398 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:33.469293    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lmphn" podStartSLOduration=2.739809273 podStartE2EDuration="8.46927846s" totalImagesPullingTime="5.729469187s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:52:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:52:33.348941375 +0000 UTC m=+293.927548756" watchObservedRunningTime="2026-11-02 02:52:33.46927846 +0000 UTC m=+294.047885839"
Nov 02 02:52:33.525639 ip-10-0-133-163 kubenswrapper[4496]: E1102 02:52:33.525599    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 02:52:34.115600 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:34.115571    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfb8rqj"
Nov 02 02:52:34.116588 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:34.116565    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfb8rqj"
Nov 02 02:52:34.123989 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:34.123963    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-htj4h"]
Nov 02 02:52:34.133268 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:34.133248    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-htj4h"
Nov 02 02:52:34.183473 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:34.183451    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-htj4h"]
Nov 02 02:52:34.186318 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:34.186294    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Nov 02 02:52:34.186409 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:34.186353    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Nov 02 02:52:34.186409 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:34.186405    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-ggcwd\""
Nov 02 02:52:34.290836 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:34.290798    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nvmj\" (UniqueName: \"kubernetes.io/projected/e7633725-0594-4c8c-aa79-bc828131c665-kube-api-access-7nvmj\") pod \"cert-manager-597b95cb84-htj4h\" (UID: \"e7633725-0594-4c8c-aa79-bc828131c665\") " pod="cert-manager/cert-manager-597b95cb84-htj4h"
Nov 02 02:52:34.290836 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:34.290839    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e7633725-0594-4c8c-aa79-bc828131c665-bound-sa-token\") pod \"cert-manager-597b95cb84-htj4h\" (UID: \"e7633725-0594-4c8c-aa79-bc828131c665\") " pod="cert-manager/cert-manager-597b95cb84-htj4h"
Nov 02 02:52:34.391717 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:34.391630    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nvmj\" (UniqueName: \"kubernetes.io/projected/e7633725-0594-4c8c-aa79-bc828131c665-kube-api-access-7nvmj\") pod \"cert-manager-597b95cb84-htj4h\" (UID: \"e7633725-0594-4c8c-aa79-bc828131c665\") " pod="cert-manager/cert-manager-597b95cb84-htj4h"
Nov 02 02:52:34.391717 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:34.391669    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e7633725-0594-4c8c-aa79-bc828131c665-bound-sa-token\") pod \"cert-manager-597b95cb84-htj4h\" (UID: \"e7633725-0594-4c8c-aa79-bc828131c665\") " pod="cert-manager/cert-manager-597b95cb84-htj4h"
Nov 02 02:52:34.449290 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:34.449256    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e7633725-0594-4c8c-aa79-bc828131c665-bound-sa-token\") pod \"cert-manager-597b95cb84-htj4h\" (UID: \"e7633725-0594-4c8c-aa79-bc828131c665\") " pod="cert-manager/cert-manager-597b95cb84-htj4h"
Nov 02 02:52:34.449429 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:34.449340    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nvmj\" (UniqueName: \"kubernetes.io/projected/e7633725-0594-4c8c-aa79-bc828131c665-kube-api-access-7nvmj\") pod \"cert-manager-597b95cb84-htj4h\" (UID: \"e7633725-0594-4c8c-aa79-bc828131c665\") " pod="cert-manager/cert-manager-597b95cb84-htj4h"
Nov 02 02:52:34.744949 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:34.744917    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-htj4h"
Nov 02 02:52:35.145902 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:35.145878    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-htj4h"]
Nov 02 02:52:35.147479 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:52:35.147454    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode7633725_0594_4c8c_aa79_bc828131c665.slice/crio-801b10694d8ae3187dc84cab5046b79480b1474231297b507ecff10fc5e70f9e WatchSource:0}: Error finding container 801b10694d8ae3187dc84cab5046b79480b1474231297b507ecff10fc5e70f9e: Status 404 returned error can't find the container with id 801b10694d8ae3187dc84cab5046b79480b1474231297b507ecff10fc5e70f9e
Nov 02 02:52:36.126396 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:36.126363    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-htj4h" event={"ID":"e7633725-0594-4c8c-aa79-bc828131c665","Type":"ContainerStarted","Data":"801b10694d8ae3187dc84cab5046b79480b1474231297b507ecff10fc5e70f9e"}
Nov 02 02:52:39.144100 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:39.144056    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-htj4h" event={"ID":"e7633725-0594-4c8c-aa79-bc828131c665","Type":"ContainerStarted","Data":"fac76ee2b3dafa536aa1190341471cc322a57a30ff1aa7e26c0582c2c6595f9b"}
Nov 02 02:52:39.260609 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:39.260549    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-htj4h" podStartSLOduration=1.932134102 podStartE2EDuration="5.260535947s" totalImagesPullingTime="3.328401845s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:52:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:52:39.260235978 +0000 UTC m=+299.838843359" watchObservedRunningTime="2026-11-02 02:52:39.260535947 +0000 UTC m=+299.839143326"
Nov 02 02:52:39.599310 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:39.599278    4496 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hww8q_5c797b08-a314-44ed-8026-8a0ce1f6c36f/ovn-acl-logging/0.log"
Nov 02 02:52:39.599707 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:39.599688    4496 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hww8q_5c797b08-a314-44ed-8026-8a0ce1f6c36f/ovn-acl-logging/0.log"
Nov 02 02:52:44.117719 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:52:44.117685    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lmphn"
Nov 02 02:53:03.527761 ip-10-0-133-163 kubenswrapper[4496]: E1102 02:53:03.527724    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 02:53:06.542652 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:53:06.542621    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-58db569bd8-bmb6f"]
Nov 02 02:53:06.548103 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:53:06.546540    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-58db569bd8-bmb6f"
Nov 02 02:53:06.651699 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:53:06.651662    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\""
Nov 02 02:53:06.651699 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:53:06.651669    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\""
Nov 02 02:53:06.651902 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:53:06.651738    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-pvnqk\""
Nov 02 02:53:06.651902 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:53:06.651810    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\""
Nov 02 02:53:06.653732 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:53:06.653711    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-58db569bd8-bmb6f"]
Nov 02 02:53:06.686249 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:53:06.686221    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53836971-8046-4f1e-be04-6bc816398b2f-metrics-certs\") pod \"jobset-controller-manager-58db569bd8-bmb6f\" (UID: \"53836971-8046-4f1e-be04-6bc816398b2f\") " pod="openshift-jobset-operator/jobset-controller-manager-58db569bd8-bmb6f"
Nov 02 02:53:06.686398 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:53:06.686297    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v275h\" (UniqueName: \"kubernetes.io/projected/53836971-8046-4f1e-be04-6bc816398b2f-kube-api-access-v275h\") pod \"jobset-controller-manager-58db569bd8-bmb6f\" (UID: \"53836971-8046-4f1e-be04-6bc816398b2f\") " pod="openshift-jobset-operator/jobset-controller-manager-58db569bd8-bmb6f"
Nov 02 02:53:06.686398 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:53:06.686355    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/53836971-8046-4f1e-be04-6bc816398b2f-manager-config\") pod \"jobset-controller-manager-58db569bd8-bmb6f\" (UID: \"53836971-8046-4f1e-be04-6bc816398b2f\") " pod="openshift-jobset-operator/jobset-controller-manager-58db569bd8-bmb6f"
Nov 02 02:53:06.686398 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:53:06.686378    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/53836971-8046-4f1e-be04-6bc816398b2f-cert\") pod \"jobset-controller-manager-58db569bd8-bmb6f\" (UID: \"53836971-8046-4f1e-be04-6bc816398b2f\") " pod="openshift-jobset-operator/jobset-controller-manager-58db569bd8-bmb6f"
Nov 02 02:53:06.787638 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:53:06.787608    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53836971-8046-4f1e-be04-6bc816398b2f-metrics-certs\") pod \"jobset-controller-manager-58db569bd8-bmb6f\" (UID: \"53836971-8046-4f1e-be04-6bc816398b2f\") " pod="openshift-jobset-operator/jobset-controller-manager-58db569bd8-bmb6f"
Nov 02 02:53:06.787808 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:53:06.787643    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v275h\" (UniqueName: \"kubernetes.io/projected/53836971-8046-4f1e-be04-6bc816398b2f-kube-api-access-v275h\") pod \"jobset-controller-manager-58db569bd8-bmb6f\" (UID: \"53836971-8046-4f1e-be04-6bc816398b2f\") " pod="openshift-jobset-operator/jobset-controller-manager-58db569bd8-bmb6f"
Nov 02 02:53:06.787808 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:53:06.787666    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/53836971-8046-4f1e-be04-6bc816398b2f-manager-config\") pod \"jobset-controller-manager-58db569bd8-bmb6f\" (UID: \"53836971-8046-4f1e-be04-6bc816398b2f\") " pod="openshift-jobset-operator/jobset-controller-manager-58db569bd8-bmb6f"
Nov 02 02:53:06.787808 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:53:06.787694    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/53836971-8046-4f1e-be04-6bc816398b2f-cert\") pod \"jobset-controller-manager-58db569bd8-bmb6f\" (UID: \"53836971-8046-4f1e-be04-6bc816398b2f\") " pod="openshift-jobset-operator/jobset-controller-manager-58db569bd8-bmb6f"
Nov 02 02:53:06.788379 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:53:06.788354    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/53836971-8046-4f1e-be04-6bc816398b2f-manager-config\") pod \"jobset-controller-manager-58db569bd8-bmb6f\" (UID: \"53836971-8046-4f1e-be04-6bc816398b2f\") " pod="openshift-jobset-operator/jobset-controller-manager-58db569bd8-bmb6f"
Nov 02 02:53:06.790117 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:53:06.790095    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/53836971-8046-4f1e-be04-6bc816398b2f-cert\") pod \"jobset-controller-manager-58db569bd8-bmb6f\" (UID: \"53836971-8046-4f1e-be04-6bc816398b2f\") " pod="openshift-jobset-operator/jobset-controller-manager-58db569bd8-bmb6f"
Nov 02 02:53:06.790203 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:53:06.790122    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53836971-8046-4f1e-be04-6bc816398b2f-metrics-certs\") pod \"jobset-controller-manager-58db569bd8-bmb6f\" (UID: \"53836971-8046-4f1e-be04-6bc816398b2f\") " pod="openshift-jobset-operator/jobset-controller-manager-58db569bd8-bmb6f"
Nov 02 02:53:06.849146 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:53:06.849084    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v275h\" (UniqueName: \"kubernetes.io/projected/53836971-8046-4f1e-be04-6bc816398b2f-kube-api-access-v275h\") pod \"jobset-controller-manager-58db569bd8-bmb6f\" (UID: \"53836971-8046-4f1e-be04-6bc816398b2f\") " pod="openshift-jobset-operator/jobset-controller-manager-58db569bd8-bmb6f"
Nov 02 02:53:06.858941 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:53:06.858916    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-58db569bd8-bmb6f"
Nov 02 02:53:07.224710 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:53:07.224685    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-58db569bd8-bmb6f"]
Nov 02 02:53:07.226940 ip-10-0-133-163 kubenswrapper[4496]: W1102 02:53:07.226905    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53836971_8046_4f1e_be04_6bc816398b2f.slice/crio-ff4794dd3c348d88ebc06c79531c7f5f691074fb8e6c5b68776b76253d21b53e WatchSource:0}: Error finding container ff4794dd3c348d88ebc06c79531c7f5f691074fb8e6c5b68776b76253d21b53e: Status 404 returned error can't find the container with id ff4794dd3c348d88ebc06c79531c7f5f691074fb8e6c5b68776b76253d21b53e
Nov 02 02:53:07.228803 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:53:07.228784    4496 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 02:53:07.293196 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:53:07.293159    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-58db569bd8-bmb6f" event={"ID":"53836971-8046-4f1e-be04-6bc816398b2f","Type":"ContainerStarted","Data":"ff4794dd3c348d88ebc06c79531c7f5f691074fb8e6c5b68776b76253d21b53e"}
Nov 02 02:53:12.320464 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:53:12.320421    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-58db569bd8-bmb6f" event={"ID":"53836971-8046-4f1e-be04-6bc816398b2f","Type":"ContainerStarted","Data":"7bdb5ca9e6ecf4af17648c1fc3d3a6335f8051a6df28e3accd3791778cbbc1c4"}
Nov 02 02:53:12.320898 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:53:12.320522    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-58db569bd8-bmb6f"
Nov 02 02:53:12.438344 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:53:12.438295    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-58db569bd8-bmb6f" podStartSLOduration=2.353299403 podStartE2EDuration="6.438281584s" totalImagesPullingTime="4.084982181s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:53:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:53:12.436191729 +0000 UTC m=+333.014799111" watchObservedRunningTime="2026-11-02 02:53:12.438281584 +0000 UTC m=+333.016888964"
Nov 02 02:53:23.331240 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:53:23.331205    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-58db569bd8-bmb6f"
Nov 02 02:53:33.525516 ip-10-0-133-163 kubenswrapper[4496]: E1102 02:53:33.525480    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 02:54:03.524873 ip-10-0-133-163 kubenswrapper[4496]: E1102 02:54:03.524837    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 02:54:33.525256 ip-10-0-133-163 kubenswrapper[4496]: E1102 02:54:33.525218    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 02:55:03.525347 ip-10-0-133-163 kubenswrapper[4496]: E1102 02:55:03.525311    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 02:55:33.525405 ip-10-0-133-163 kubenswrapper[4496]: E1102 02:55:33.525363    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 02:56:03.525147 ip-10-0-133-163 kubenswrapper[4496]: E1102 02:56:03.525110    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 02:56:33.524041 ip-10-0-133-163 kubenswrapper[4496]: E1102 02:56:33.523992    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 02:57:03.524712 ip-10-0-133-163 kubenswrapper[4496]: E1102 02:57:03.524677    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 02:57:33.524825 ip-10-0-133-163 kubenswrapper[4496]: E1102 02:57:33.524732    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 02:57:39.634447 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:57:39.634406    4496 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hww8q_5c797b08-a314-44ed-8026-8a0ce1f6c36f/ovn-acl-logging/0.log"
Nov 02 02:57:39.635593 ip-10-0-133-163 kubenswrapper[4496]: I1102 02:57:39.635568    4496 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hww8q_5c797b08-a314-44ed-8026-8a0ce1f6c36f/ovn-acl-logging/0.log"
Nov 02 02:58:03.524077 ip-10-0-133-163 kubenswrapper[4496]: E1102 02:58:03.524042    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 02:58:33.525582 ip-10-0-133-163 kubenswrapper[4496]: E1102 02:58:33.525547    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 02:59:03.525221 ip-10-0-133-163 kubenswrapper[4496]: E1102 02:59:03.525185    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 02:59:33.527601 ip-10-0-133-163 kubenswrapper[4496]: E1102 02:59:33.527558    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 03:00:03.525142 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:00:03.525096    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 03:00:33.524736 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:00:33.524660    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 03:01:03.524999 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:01:03.524961    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 03:01:33.524765 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:01:33.524731    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 03:02:03.524985 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:02:03.524883    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 03:02:33.525254 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:02:33.525214    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 03:02:39.667654 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:02:39.667624    4496 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hww8q_5c797b08-a314-44ed-8026-8a0ce1f6c36f/ovn-acl-logging/0.log"
Nov 02 03:02:39.670831 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:02:39.670807    4496 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hww8q_5c797b08-a314-44ed-8026-8a0ce1f6c36f/ovn-acl-logging/0.log"
Nov 02 03:02:47.969677 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:02:47.969643    4496 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-lmphn_76b2812d-c926-4aff-a519-fa327f7e4695/manager/0.log"
Nov 02 03:02:48.349930 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:02:48.349893    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lmphn"]
Nov 02 03:02:48.350147 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:02:48.350107    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lmphn" podUID="76b2812d-c926-4aff-a519-fa327f7e4695" containerName="manager" containerID="cri-o://f1f8180bd92a106678cb940c456a132a359bc706e8558f7de443f09b586fc253" gracePeriod=10
Nov 02 03:02:48.351779 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:02:48.351757    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fxd5p"]
Nov 02 03:02:48.355401 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:02:48.355382    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fxd5p"
Nov 02 03:02:48.384373 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:02:48.384350    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/31b4528f-969b-41a0-b61d-4d4d882e2e8c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-fxd5p\" (UID: \"31b4528f-969b-41a0-b61d-4d4d882e2e8c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fxd5p"
Nov 02 03:02:48.384475 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:02:48.384409    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swjt2\" (UniqueName: \"kubernetes.io/projected/31b4528f-969b-41a0-b61d-4d4d882e2e8c-kube-api-access-swjt2\") pod \"kuadrant-operator-controller-manager-678d87766f-fxd5p\" (UID: \"31b4528f-969b-41a0-b61d-4d4d882e2e8c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fxd5p"
Nov 02 03:02:48.473736 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:02:48.473698    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fxd5p"]
Nov 02 03:02:48.485530 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:02:48.485498    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swjt2\" (UniqueName: \"kubernetes.io/projected/31b4528f-969b-41a0-b61d-4d4d882e2e8c-kube-api-access-swjt2\") pod \"kuadrant-operator-controller-manager-678d87766f-fxd5p\" (UID: \"31b4528f-969b-41a0-b61d-4d4d882e2e8c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fxd5p"
Nov 02 03:02:48.485663 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:02:48.485590    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/31b4528f-969b-41a0-b61d-4d4d882e2e8c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-fxd5p\" (UID: \"31b4528f-969b-41a0-b61d-4d4d882e2e8c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fxd5p"
Nov 02 03:02:48.486061 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:02:48.486027    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/31b4528f-969b-41a0-b61d-4d4d882e2e8c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-fxd5p\" (UID: \"31b4528f-969b-41a0-b61d-4d4d882e2e8c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fxd5p"
Nov 02 03:02:48.543818 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:02:48.543791    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swjt2\" (UniqueName: \"kubernetes.io/projected/31b4528f-969b-41a0-b61d-4d4d882e2e8c-kube-api-access-swjt2\") pod \"kuadrant-operator-controller-manager-678d87766f-fxd5p\" (UID: \"31b4528f-969b-41a0-b61d-4d4d882e2e8c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fxd5p"
Nov 02 03:02:48.605661 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:02:48.605603    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lmphn"
Nov 02 03:02:48.687556 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:02:48.687525    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76b2812d-c926-4aff-a519-fa327f7e4695-kube-api-access-7r6gm\" (UniqueName: \"kubernetes.io/projected/76b2812d-c926-4aff-a519-fa327f7e4695-kube-api-access-7r6gm\") pod \"76b2812d-c926-4aff-a519-fa327f7e4695\" (UID: \"76b2812d-c926-4aff-a519-fa327f7e4695\") "
Nov 02 03:02:48.687697 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:02:48.687618    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76b2812d-c926-4aff-a519-fa327f7e4695-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/76b2812d-c926-4aff-a519-fa327f7e4695-extensions-socket-volume\") pod \"76b2812d-c926-4aff-a519-fa327f7e4695\" (UID: \"76b2812d-c926-4aff-a519-fa327f7e4695\") "
Nov 02 03:02:48.687969 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:02:48.687941    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76b2812d-c926-4aff-a519-fa327f7e4695-extensions-socket-volume" pod "76b2812d-c926-4aff-a519-fa327f7e4695" (UID: "76b2812d-c926-4aff-a519-fa327f7e4695"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:02:48.689664 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:02:48.689637    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76b2812d-c926-4aff-a519-fa327f7e4695-kube-api-access-7r6gm" pod "76b2812d-c926-4aff-a519-fa327f7e4695" (UID: "76b2812d-c926-4aff-a519-fa327f7e4695"). InnerVolumeSpecName "kube-api-access-7r6gm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:02:48.726856 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:02:48.726832    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fxd5p"
Nov 02 03:02:48.788894 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:02:48.788859    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7r6gm\" (UniqueName: \"kubernetes.io/projected/76b2812d-c926-4aff-a519-fa327f7e4695-kube-api-access-7r6gm\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:02:48.788985 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:02:48.788903    4496 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/76b2812d-c926-4aff-a519-fa327f7e4695-extensions-socket-volume\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:02:49.139315 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:02:49.139279    4496 generic.go:356] "Generic (PLEG): container finished" podID="76b2812d-c926-4aff-a519-fa327f7e4695" containerID="f1f8180bd92a106678cb940c456a132a359bc706e8558f7de443f09b586fc253" exitCode=0
Nov 02 03:02:49.139702 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:02:49.139323    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lmphn"
Nov 02 03:02:49.139702 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:02:49.139331    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lmphn" event={"ID":"76b2812d-c926-4aff-a519-fa327f7e4695","Type":"ContainerDied","Data":"f1f8180bd92a106678cb940c456a132a359bc706e8558f7de443f09b586fc253"}
Nov 02 03:02:49.139702 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:02:49.139381    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lmphn" event={"ID":"76b2812d-c926-4aff-a519-fa327f7e4695","Type":"ContainerDied","Data":"a01b308069089b1b2222e87cbf9c474d84b7c2f60b490a62a8ba93b6abb43ed3"}
Nov 02 03:02:49.139702 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:02:49.139409    4496 scope.go:122] "RemoveContainer" containerID="f1f8180bd92a106678cb940c456a132a359bc706e8558f7de443f09b586fc253"
Nov 02 03:02:49.148244 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:02:49.148228    4496 scope.go:122] "RemoveContainer" containerID="f1f8180bd92a106678cb940c456a132a359bc706e8558f7de443f09b586fc253"
Nov 02 03:02:49.148488 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:02:49.148469    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1f8180bd92a106678cb940c456a132a359bc706e8558f7de443f09b586fc253\": container with ID starting with f1f8180bd92a106678cb940c456a132a359bc706e8558f7de443f09b586fc253 not found: ID does not exist" containerID="f1f8180bd92a106678cb940c456a132a359bc706e8558f7de443f09b586fc253"
Nov 02 03:02:49.148531 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:02:49.148499    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1f8180bd92a106678cb940c456a132a359bc706e8558f7de443f09b586fc253"} err="failed to get container status \"f1f8180bd92a106678cb940c456a132a359bc706e8558f7de443f09b586fc253\": rpc error: code = NotFound desc = could not find container \"f1f8180bd92a106678cb940c456a132a359bc706e8558f7de443f09b586fc253\": container with ID starting with f1f8180bd92a106678cb940c456a132a359bc706e8558f7de443f09b586fc253 not found: ID does not exist"
Nov 02 03:02:49.154976 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:02:49.154956    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fxd5p"]
Nov 02 03:02:49.156466 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:02:49.156443    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31b4528f_969b_41a0_b61d_4d4d882e2e8c.slice/crio-629f3bd343aa6378d1be835ecb641097496614d087474ec826d65e528f03d1df WatchSource:0}: Error finding container 629f3bd343aa6378d1be835ecb641097496614d087474ec826d65e528f03d1df: Status 404 returned error can't find the container with id 629f3bd343aa6378d1be835ecb641097496614d087474ec826d65e528f03d1df
Nov 02 03:02:49.313344 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:02:49.313310    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lmphn"]
Nov 02 03:02:49.316890 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:02:49.316863    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lmphn"]
Nov 02 03:02:49.571560 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:02:49.571525    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76b2812d-c926-4aff-a519-fa327f7e4695" path="/var/lib/kubelet/pods/76b2812d-c926-4aff-a519-fa327f7e4695/volumes"
Nov 02 03:02:50.146278 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:02:50.146240    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fxd5p" event={"ID":"31b4528f-969b-41a0-b61d-4d4d882e2e8c","Type":"ContainerStarted","Data":"2d8dc416b7660c754eedaf2c1e6fcdaf08987a60a9c24644902cf935115566d5"}
Nov 02 03:02:50.146734 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:02:50.146398    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fxd5p"
Nov 02 03:02:50.146734 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:02:50.146447    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fxd5p" event={"ID":"31b4528f-969b-41a0-b61d-4d4d882e2e8c","Type":"ContainerStarted","Data":"629f3bd343aa6378d1be835ecb641097496614d087474ec826d65e528f03d1df"}
Nov 02 03:02:50.367648 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:02:50.367595    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fxd5p" podStartSLOduration=2.367579761 podStartE2EDuration="2.367579761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:02:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:02:50.316567866 +0000 UTC m=+910.895175245" watchObservedRunningTime="2026-11-02 03:02:50.367579761 +0000 UTC m=+910.946187138"
Nov 02 03:03:01.155488 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:03:01.155455    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fxd5p"
Nov 02 03:03:03.525557 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:03:03.525521    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 03:03:12.973719 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:03:12.973684    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-rckpv"]
Nov 02 03:03:12.974127 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:03:12.974115    4496 cpu_manager.go:414] "containerMap: removing container" podUID="76b2812d-c926-4aff-a519-fa327f7e4695" containerName="manager"
Nov 02 03:03:12.974176 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:03:12.974129    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76b2812d-c926-4aff-a519-fa327f7e4695" containerName="manager"
Nov 02 03:03:12.974213 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:03:12.974195    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="76b2812d-c926-4aff-a519-fa327f7e4695" containerName="manager"
Nov 02 03:03:12.977184 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:03:12.977168    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-rckpv"
Nov 02 03:03:13.032874 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:03:13.032839    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-rckpv"]
Nov 02 03:03:13.080927 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:03:13.080899    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-lcdwz\""
Nov 02 03:03:13.099613 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:03:13.099582    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5d4rz\" (UniqueName: \"kubernetes.io/projected/c20860de-e53a-4b3f-b5f3-9199a030d05c-kube-api-access-5d4rz\") pod \"authorino-57f7f8659f-rckpv\" (UID: \"c20860de-e53a-4b3f-b5f3-9199a030d05c\") " pod="kuadrant-system/authorino-57f7f8659f-rckpv"
Nov 02 03:03:13.200876 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:03:13.200842    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5d4rz\" (UniqueName: \"kubernetes.io/projected/c20860de-e53a-4b3f-b5f3-9199a030d05c-kube-api-access-5d4rz\") pod \"authorino-57f7f8659f-rckpv\" (UID: \"c20860de-e53a-4b3f-b5f3-9199a030d05c\") " pod="kuadrant-system/authorino-57f7f8659f-rckpv"
Nov 02 03:03:13.259731 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:03:13.259705    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5d4rz\" (UniqueName: \"kubernetes.io/projected/c20860de-e53a-4b3f-b5f3-9199a030d05c-kube-api-access-5d4rz\") pod \"authorino-57f7f8659f-rckpv\" (UID: \"c20860de-e53a-4b3f-b5f3-9199a030d05c\") " pod="kuadrant-system/authorino-57f7f8659f-rckpv"
Nov 02 03:03:13.288670 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:03:13.288645    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-rckpv"
Nov 02 03:03:13.656648 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:03:13.656626    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-rckpv"]
Nov 02 03:03:13.658500 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:03:13.658465    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc20860de_e53a_4b3f_b5f3_9199a030d05c.slice/crio-25a3cd5022aa4bab85d15f24dc125e95169e86b8e0d69296f839978e745519c4 WatchSource:0}: Error finding container 25a3cd5022aa4bab85d15f24dc125e95169e86b8e0d69296f839978e745519c4: Status 404 returned error can't find the container with id 25a3cd5022aa4bab85d15f24dc125e95169e86b8e0d69296f839978e745519c4
Nov 02 03:03:13.659764 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:03:13.659746    4496 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 03:03:14.268947 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:03:14.268910    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-rckpv" event={"ID":"c20860de-e53a-4b3f-b5f3-9199a030d05c","Type":"ContainerStarted","Data":"25a3cd5022aa4bab85d15f24dc125e95169e86b8e0d69296f839978e745519c4"}
Nov 02 03:03:16.063316 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:03:16.063281    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-nxjmt"]
Nov 02 03:03:16.067085 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:03:16.067063    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-nxjmt"
Nov 02 03:03:16.123624 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:03:16.123591    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-nxjmt"]
Nov 02 03:03:16.229330 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:03:16.229296    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wr7pc\" (UniqueName: \"kubernetes.io/projected/a24f74eb-7859-4958-b4d1-8fff3f1371e6-kube-api-access-wr7pc\") pod \"authorino-56c6c9ddd5-nxjmt\" (UID: \"a24f74eb-7859-4958-b4d1-8fff3f1371e6\") " pod="kuadrant-system/authorino-56c6c9ddd5-nxjmt"
Nov 02 03:03:16.330188 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:03:16.330103    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wr7pc\" (UniqueName: \"kubernetes.io/projected/a24f74eb-7859-4958-b4d1-8fff3f1371e6-kube-api-access-wr7pc\") pod \"authorino-56c6c9ddd5-nxjmt\" (UID: \"a24f74eb-7859-4958-b4d1-8fff3f1371e6\") " pod="kuadrant-system/authorino-56c6c9ddd5-nxjmt"
Nov 02 03:03:16.388437 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:03:16.388406    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wr7pc\" (UniqueName: \"kubernetes.io/projected/a24f74eb-7859-4958-b4d1-8fff3f1371e6-kube-api-access-wr7pc\") pod \"authorino-56c6c9ddd5-nxjmt\" (UID: \"a24f74eb-7859-4958-b4d1-8fff3f1371e6\") " pod="kuadrant-system/authorino-56c6c9ddd5-nxjmt"
Nov 02 03:03:16.680733 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:03:16.680655    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-nxjmt"
Nov 02 03:03:17.054851 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:03:17.054818    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-nxjmt"]
Nov 02 03:03:17.056115 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:03:17.056081    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda24f74eb_7859_4958_b4d1_8fff3f1371e6.slice/crio-032a25b7ce4ee6d1c8cb1b01f95a4d59fa583239de46b6363be0988a6a6995db WatchSource:0}: Error finding container 032a25b7ce4ee6d1c8cb1b01f95a4d59fa583239de46b6363be0988a6a6995db: Status 404 returned error can't find the container with id 032a25b7ce4ee6d1c8cb1b01f95a4d59fa583239de46b6363be0988a6a6995db
Nov 02 03:03:17.284854 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:03:17.284815    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-nxjmt" event={"ID":"a24f74eb-7859-4958-b4d1-8fff3f1371e6","Type":"ContainerStarted","Data":"032a25b7ce4ee6d1c8cb1b01f95a4d59fa583239de46b6363be0988a6a6995db"}
Nov 02 03:03:17.286306 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:03:17.286280    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-rckpv" event={"ID":"c20860de-e53a-4b3f-b5f3-9199a030d05c","Type":"ContainerStarted","Data":"8dd7c82e3af15db6b51a250e21626d8ebe7612981f5e168d48fac0e993157512"}
Nov 02 03:03:17.401209 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:03:17.401111    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-rckpv" podStartSLOduration=2.611914808 podStartE2EDuration="5.401097902s" totalImagesPullingTime="2.789183094s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:03:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:03:17.40035106 +0000 UTC m=+937.978958422" watchObservedRunningTime="2026-11-02 03:03:17.401097902 +0000 UTC m=+937.979705282"
Nov 02 03:03:19.298946 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:03:19.298907    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-nxjmt" event={"ID":"a24f74eb-7859-4958-b4d1-8fff3f1371e6","Type":"ContainerStarted","Data":"89811980c760fbe44ea89568a2f14fd92e7d3d5877a8da9d848f7f7c3de37b5e"}
Nov 02 03:03:19.415954 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:03:19.415903    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-nxjmt" podStartSLOduration=2.026458087 podStartE2EDuration="3.415889215s" totalImagesPullingTime="1.389431128s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:03:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:03:19.414220399 +0000 UTC m=+939.992827778" watchObservedRunningTime="2026-11-02 03:03:19.415889215 +0000 UTC m=+939.994496594"
Nov 02 03:03:19.467341 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:03:19.467307    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-rckpv"]
Nov 02 03:03:19.467515 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:03:19.467492    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-rckpv" podUID="c20860de-e53a-4b3f-b5f3-9199a030d05c" containerName="authorino" containerID="cri-o://8dd7c82e3af15db6b51a250e21626d8ebe7612981f5e168d48fac0e993157512" gracePeriod=30
Nov 02 03:03:19.709388 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:03:19.709363    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-rckpv"
Nov 02 03:03:19.861177 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:03:19.861105    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c20860de-e53a-4b3f-b5f3-9199a030d05c-kube-api-access-5d4rz\" (UniqueName: \"kubernetes.io/projected/c20860de-e53a-4b3f-b5f3-9199a030d05c-kube-api-access-5d4rz\") pod \"c20860de-e53a-4b3f-b5f3-9199a030d05c\" (UID: \"c20860de-e53a-4b3f-b5f3-9199a030d05c\") "
Nov 02 03:03:19.863205 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:03:19.863186    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c20860de-e53a-4b3f-b5f3-9199a030d05c-kube-api-access-5d4rz" pod "c20860de-e53a-4b3f-b5f3-9199a030d05c" (UID: "c20860de-e53a-4b3f-b5f3-9199a030d05c"). InnerVolumeSpecName "kube-api-access-5d4rz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:03:19.962119 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:03:19.962084    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5d4rz\" (UniqueName: \"kubernetes.io/projected/c20860de-e53a-4b3f-b5f3-9199a030d05c-kube-api-access-5d4rz\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:03:20.305588 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:03:20.305549    4496 generic.go:356] "Generic (PLEG): container finished" podID="c20860de-e53a-4b3f-b5f3-9199a030d05c" containerID="8dd7c82e3af15db6b51a250e21626d8ebe7612981f5e168d48fac0e993157512" exitCode=0
Nov 02 03:03:20.306037 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:03:20.305595    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-rckpv"
Nov 02 03:03:20.306037 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:03:20.305626    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-rckpv" event={"ID":"c20860de-e53a-4b3f-b5f3-9199a030d05c","Type":"ContainerDied","Data":"8dd7c82e3af15db6b51a250e21626d8ebe7612981f5e168d48fac0e993157512"}
Nov 02 03:03:20.306037 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:03:20.305670    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-rckpv" event={"ID":"c20860de-e53a-4b3f-b5f3-9199a030d05c","Type":"ContainerDied","Data":"25a3cd5022aa4bab85d15f24dc125e95169e86b8e0d69296f839978e745519c4"}
Nov 02 03:03:20.306037 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:03:20.305688    4496 scope.go:122] "RemoveContainer" containerID="8dd7c82e3af15db6b51a250e21626d8ebe7612981f5e168d48fac0e993157512"
Nov 02 03:03:20.315055 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:03:20.315028    4496 scope.go:122] "RemoveContainer" containerID="8dd7c82e3af15db6b51a250e21626d8ebe7612981f5e168d48fac0e993157512"
Nov 02 03:03:20.315336 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:03:20.315317    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8dd7c82e3af15db6b51a250e21626d8ebe7612981f5e168d48fac0e993157512\": container with ID starting with 8dd7c82e3af15db6b51a250e21626d8ebe7612981f5e168d48fac0e993157512 not found: ID does not exist" containerID="8dd7c82e3af15db6b51a250e21626d8ebe7612981f5e168d48fac0e993157512"
Nov 02 03:03:20.315384 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:03:20.315347    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dd7c82e3af15db6b51a250e21626d8ebe7612981f5e168d48fac0e993157512"} err="failed to get container status \"8dd7c82e3af15db6b51a250e21626d8ebe7612981f5e168d48fac0e993157512\": rpc error: code = NotFound desc = could not find container \"8dd7c82e3af15db6b51a250e21626d8ebe7612981f5e168d48fac0e993157512\": container with ID starting with 8dd7c82e3af15db6b51a250e21626d8ebe7612981f5e168d48fac0e993157512 not found: ID does not exist"
Nov 02 03:03:20.471646 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:03:20.471606    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-rckpv"]
Nov 02 03:03:20.475628 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:03:20.475603    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-rckpv"]
Nov 02 03:03:21.572041 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:03:21.571990    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c20860de-e53a-4b3f-b5f3-9199a030d05c" path="/var/lib/kubelet/pods/c20860de-e53a-4b3f-b5f3-9199a030d05c/volumes"
Nov 02 03:03:33.524582 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:03:33.524499    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 03:04:03.526804 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:04:03.526769    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 03:04:06.841453 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:04:06.841416    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-fj5pj"]
Nov 02 03:04:06.841922 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:04:06.841879    4496 cpu_manager.go:414] "containerMap: removing container" podUID="c20860de-e53a-4b3f-b5f3-9199a030d05c" containerName="authorino"
Nov 02 03:04:06.841922 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:04:06.841896    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c20860de-e53a-4b3f-b5f3-9199a030d05c" containerName="authorino"
Nov 02 03:04:06.842028 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:04:06.841983    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="c20860de-e53a-4b3f-b5f3-9199a030d05c" containerName="authorino"
Nov 02 03:04:06.845047 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:04:06.845029    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-fj5pj"
Nov 02 03:04:06.950898 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:04:06.950861    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\""
Nov 02 03:04:06.950898 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:04:06.950876    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Nov 02 03:04:06.951148 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:04:06.950895    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Nov 02 03:04:06.951148 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:04:06.950914    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-cfvjn\""
Nov 02 03:04:06.951148 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:04:06.950923    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\""
Nov 02 03:04:06.951148 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:04:06.951072    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\""
Nov 02 03:04:06.953605 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:04:06.953578    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-fj5pj"]
Nov 02 03:04:06.956420 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:04:06.956396    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26lcq\" (UniqueName: \"kubernetes.io/projected/ced04bab-431b-4207-a07b-647649f4419e-kube-api-access-26lcq\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-fj5pj\" (UID: \"ced04bab-431b-4207-a07b-647649f4419e\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-fj5pj"
Nov 02 03:04:06.956525 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:04:06.956435    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ced04bab-431b-4207-a07b-647649f4419e-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-fj5pj\" (UID: \"ced04bab-431b-4207-a07b-647649f4419e\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-fj5pj"
Nov 02 03:04:06.956525 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:04:06.956459    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/ced04bab-431b-4207-a07b-647649f4419e-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-fj5pj\" (UID: \"ced04bab-431b-4207-a07b-647649f4419e\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-fj5pj"
Nov 02 03:04:06.956606 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:04:06.956528    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ced04bab-431b-4207-a07b-647649f4419e-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-fj5pj\" (UID: \"ced04bab-431b-4207-a07b-647649f4419e\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-fj5pj"
Nov 02 03:04:07.057184 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:04:07.057149    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26lcq\" (UniqueName: \"kubernetes.io/projected/ced04bab-431b-4207-a07b-647649f4419e-kube-api-access-26lcq\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-fj5pj\" (UID: \"ced04bab-431b-4207-a07b-647649f4419e\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-fj5pj"
Nov 02 03:04:07.057367 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:04:07.057198    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ced04bab-431b-4207-a07b-647649f4419e-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-fj5pj\" (UID: \"ced04bab-431b-4207-a07b-647649f4419e\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-fj5pj"
Nov 02 03:04:07.057367 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:04:07.057219    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/ced04bab-431b-4207-a07b-647649f4419e-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-fj5pj\" (UID: \"ced04bab-431b-4207-a07b-647649f4419e\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-fj5pj"
Nov 02 03:04:07.057367 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:04:07.057244    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ced04bab-431b-4207-a07b-647649f4419e-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-fj5pj\" (UID: \"ced04bab-431b-4207-a07b-647649f4419e\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-fj5pj"
Nov 02 03:04:07.057367 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:04:07.057308    4496 configmap.go:193] Couldn't get configMap opendatahub/usage-tenancy-proxy-source: configmap "usage-tenancy-proxy-source" not found
Nov 02 03:04:07.057367 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:04:07.057328    4496 secret.go:189] Couldn't get secret opendatahub/usage-logs-tenancy-proxy-tls: secret "usage-logs-tenancy-proxy-tls" not found
Nov 02 03:04:07.057611 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:04:07.057399    4496 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ced04bab-431b-4207-a07b-647649f4419e-proxy-source podName:ced04bab-431b-4207-a07b-647649f4419e nodeName:}" failed. No retries permitted until 2026-11-02 03:04:07.557381634 +0000 UTC m=+988.135988991 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-source" (UniqueName: "kubernetes.io/configmap/ced04bab-431b-4207-a07b-647649f4419e-proxy-source") pod "usage-logs-tenancy-proxy-9c5d49bcd-fj5pj" (UID: "ced04bab-431b-4207-a07b-647649f4419e") : configmap "usage-tenancy-proxy-source" not found
Nov 02 03:04:07.057611 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:04:07.057421    4496 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ced04bab-431b-4207-a07b-647649f4419e-tls-cert podName:ced04bab-431b-4207-a07b-647649f4419e nodeName:}" failed. No retries permitted until 2026-11-02 03:04:07.557409194 +0000 UTC m=+988.136016552 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-cert" (UniqueName: "kubernetes.io/secret/ced04bab-431b-4207-a07b-647649f4419e-tls-cert") pod "usage-logs-tenancy-proxy-9c5d49bcd-fj5pj" (UID: "ced04bab-431b-4207-a07b-647649f4419e") : secret "usage-logs-tenancy-proxy-tls" not found
Nov 02 03:04:07.057817 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:04:07.057801    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ced04bab-431b-4207-a07b-647649f4419e-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-fj5pj\" (UID: \"ced04bab-431b-4207-a07b-647649f4419e\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-fj5pj"
Nov 02 03:04:07.117616 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:04:07.117554    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26lcq\" (UniqueName: \"kubernetes.io/projected/ced04bab-431b-4207-a07b-647649f4419e-kube-api-access-26lcq\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-fj5pj\" (UID: \"ced04bab-431b-4207-a07b-647649f4419e\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-fj5pj"
Nov 02 03:04:07.561295 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:04:07.561264    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ced04bab-431b-4207-a07b-647649f4419e-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-fj5pj\" (UID: \"ced04bab-431b-4207-a07b-647649f4419e\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-fj5pj"
Nov 02 03:04:07.561471 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:04:07.561302    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/ced04bab-431b-4207-a07b-647649f4419e-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-fj5pj\" (UID: \"ced04bab-431b-4207-a07b-647649f4419e\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-fj5pj"
Nov 02 03:04:07.562094 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:04:07.562066    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/ced04bab-431b-4207-a07b-647649f4419e-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-fj5pj\" (UID: \"ced04bab-431b-4207-a07b-647649f4419e\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-fj5pj"
Nov 02 03:04:07.563654 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:04:07.563627    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ced04bab-431b-4207-a07b-647649f4419e-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-fj5pj\" (UID: \"ced04bab-431b-4207-a07b-647649f4419e\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-fj5pj"
Nov 02 03:04:07.757936 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:04:07.757901    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-fj5pj"
Nov 02 03:04:08.133379 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:04:08.133348    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-fj5pj"]
Nov 02 03:04:08.135048 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:04:08.135007    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podced04bab_431b_4207_a07b_647649f4419e.slice/crio-bc398cfa1917a5eef69fa76c0020be7349079253eac759a55d6c31e1d50b29f3 WatchSource:0}: Error finding container bc398cfa1917a5eef69fa76c0020be7349079253eac759a55d6c31e1d50b29f3: Status 404 returned error can't find the container with id bc398cfa1917a5eef69fa76c0020be7349079253eac759a55d6c31e1d50b29f3
Nov 02 03:04:08.550947 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:04:08.550913    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-fj5pj" event={"ID":"ced04bab-431b-4207-a07b-647649f4419e","Type":"ContainerStarted","Data":"bc398cfa1917a5eef69fa76c0020be7349079253eac759a55d6c31e1d50b29f3"}
Nov 02 03:04:22.632370 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:04:22.632288    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-fj5pj" event={"ID":"ced04bab-431b-4207-a07b-647649f4419e","Type":"ContainerStarted","Data":"9c552bf4b1d3a2788f4c91e6422403970f033a690623dfc6f79f0a028715c735"}
Nov 02 03:04:22.632370 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:04:22.632367    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-fj5pj"
Nov 02 03:04:22.751044 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:04:22.750984    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-fj5pj" podStartSLOduration=2.617770433 podStartE2EDuration="16.750970961s" totalImagesPullingTime="14.133200528s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:04:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:04:22.749259924 +0000 UTC m=+1003.327867306" watchObservedRunningTime="2026-11-02 03:04:22.750970961 +0000 UTC m=+1003.329578340"
Nov 02 03:04:33.526349 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:04:33.526304    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 03:04:33.642861 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:04:33.642832    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-fj5pj"
Nov 02 03:04:54.733146 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:04:54.733110    4496 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-cdwxx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 03:04:54.733604 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:04:54.733185    4496 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cdwxx" podUID="41697786-2b0f-4cb2-8975-1942d727c987" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 03:04:55.235622 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:04:55.235589    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-79j5r"]
Nov 02 03:04:55.237812 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:04:55.237795    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-79j5r"
Nov 02 03:04:55.295837 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:04:55.295806    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-79j5r"]
Nov 02 03:04:55.405932 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:04:55.405908    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqh76\" (UniqueName: \"kubernetes.io/projected/27e9997a-0ffc-47e0-94c4-2c3352e52a8f-kube-api-access-nqh76\") pod \"authorino-6f6944bcdd-79j5r\" (UID: \"27e9997a-0ffc-47e0-94c4-2c3352e52a8f\") " pod="kuadrant-system/authorino-6f6944bcdd-79j5r"
Nov 02 03:04:55.507122 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:04:55.507056    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqh76\" (UniqueName: \"kubernetes.io/projected/27e9997a-0ffc-47e0-94c4-2c3352e52a8f-kube-api-access-nqh76\") pod \"authorino-6f6944bcdd-79j5r\" (UID: \"27e9997a-0ffc-47e0-94c4-2c3352e52a8f\") " pod="kuadrant-system/authorino-6f6944bcdd-79j5r"
Nov 02 03:04:55.566901 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:04:55.566866    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqh76\" (UniqueName: \"kubernetes.io/projected/27e9997a-0ffc-47e0-94c4-2c3352e52a8f-kube-api-access-nqh76\") pod \"authorino-6f6944bcdd-79j5r\" (UID: \"27e9997a-0ffc-47e0-94c4-2c3352e52a8f\") " pod="kuadrant-system/authorino-6f6944bcdd-79j5r"
Nov 02 03:04:55.578673 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:04:55.578647    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-79j5r"]
Nov 02 03:04:55.578846 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:04:55.578834    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-79j5r"
Nov 02 03:04:55.601533 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:04:55.601508    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6db4498c4d-9ml7r"]
Nov 02 03:04:55.603969 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:04:55.603953    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6db4498c4d-9ml7r"
Nov 02 03:04:55.660263 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:04:55.660230    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6db4498c4d-9ml7r"]
Nov 02 03:04:55.711575 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:04:55.711547    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlwh8\" (UniqueName: \"kubernetes.io/projected/6f64e653-384e-4f15-8575-ac9602d531ee-kube-api-access-hlwh8\") pod \"authorino-6db4498c4d-9ml7r\" (UID: \"6f64e653-384e-4f15-8575-ac9602d531ee\") " pod="kuadrant-system/authorino-6db4498c4d-9ml7r"
Nov 02 03:04:55.810260 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:04:55.810191    4496 generic.go:356] "Generic (PLEG): container finished" podID="41697786-2b0f-4cb2-8975-1942d727c987" containerID="a9dec28ff6474e708ad4695c5434851a7e15a0187897222fc106592a444533b5" exitCode=0
Nov 02 03:04:55.810590 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:04:55.810264    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cdwxx" event={"ID":"41697786-2b0f-4cb2-8975-1942d727c987","Type":"ContainerDied","Data":"a9dec28ff6474e708ad4695c5434851a7e15a0187897222fc106592a444533b5"}
Nov 02 03:04:55.810634 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:04:55.810618    4496 scope.go:122] "RemoveContainer" containerID="a9dec28ff6474e708ad4695c5434851a7e15a0187897222fc106592a444533b5"
Nov 02 03:04:55.811684 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:04:55.811658    4496 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 03:04:55.811795 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:04:55.811737    4496 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 03:04:55.811795 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:04:55.811786    4496 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 03:04:55.811905 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:04:55.811893    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlwh8\" (UniqueName: \"kubernetes.io/projected/6f64e653-384e-4f15-8575-ac9602d531ee-kube-api-access-hlwh8\") pod \"authorino-6db4498c4d-9ml7r\" (UID: \"6f64e653-384e-4f15-8575-ac9602d531ee\") " pod="kuadrant-system/authorino-6db4498c4d-9ml7r"
Nov 02 03:04:55.871972 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:04:55.871943    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlwh8\" (UniqueName: \"kubernetes.io/projected/6f64e653-384e-4f15-8575-ac9602d531ee-kube-api-access-hlwh8\") pod \"authorino-6db4498c4d-9ml7r\" (UID: \"6f64e653-384e-4f15-8575-ac9602d531ee\") " pod="kuadrant-system/authorino-6db4498c4d-9ml7r"
Nov 02 03:04:55.918801 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:04:55.918770    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6db4498c4d-9ml7r"
Nov 02 03:04:55.950859 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:04:55.950820    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-79j5r"]
Nov 02 03:04:55.956489 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:04:55.956459    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod27e9997a_0ffc_47e0_94c4_2c3352e52a8f.slice/crio-648521894aa5008bba3f33319b7bd05744d5610bb5041b4ab9bfcbdd20c9dd98 WatchSource:0}: Error finding container 648521894aa5008bba3f33319b7bd05744d5610bb5041b4ab9bfcbdd20c9dd98: Status 404 returned error can't find the container with id 648521894aa5008bba3f33319b7bd05744d5610bb5041b4ab9bfcbdd20c9dd98
Nov 02 03:04:56.297526 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:04:56.297491    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6db4498c4d-9ml7r"]
Nov 02 03:04:56.298049 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:04:56.298005    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6f64e653_384e_4f15_8575_ac9602d531ee.slice/crio-74057d784e867673fecb5b61ade93c3c6c0d5c2068266405942f04e2c32da9ff WatchSource:0}: Error finding container 74057d784e867673fecb5b61ade93c3c6c0d5c2068266405942f04e2c32da9ff: Status 404 returned error can't find the container with id 74057d784e867673fecb5b61ade93c3c6c0d5c2068266405942f04e2c32da9ff
Nov 02 03:04:56.817304 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:04:56.817272    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6db4498c4d-9ml7r" event={"ID":"6f64e653-384e-4f15-8575-ac9602d531ee","Type":"ContainerStarted","Data":"74057d784e867673fecb5b61ade93c3c6c0d5c2068266405942f04e2c32da9ff"}
Nov 02 03:04:56.819135 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:04:56.819112    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cdwxx" event={"ID":"41697786-2b0f-4cb2-8975-1942d727c987","Type":"ContainerStarted","Data":"d97efdf4aed3ec1a74c908cc6aa65855bf41c4e7978e28e395833834a40d45e9"}
Nov 02 03:04:56.820339 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:04:56.820316    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-79j5r" event={"ID":"27e9997a-0ffc-47e0-94c4-2c3352e52a8f","Type":"ContainerStarted","Data":"648521894aa5008bba3f33319b7bd05744d5610bb5041b4ab9bfcbdd20c9dd98"}
Nov 02 03:04:56.892647 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:04:56.892615    4496 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cdwxx"
Nov 02 03:04:56.892883 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:04:56.892859    4496 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-cdwxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body=
Nov 02 03:04:56.892946 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:04:56.892924    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cdwxx" podUID="41697786-2b0f-4cb2-8975-1942d727c987" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused"
Nov 02 03:04:57.827010 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:04:57.826909    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-79j5r" event={"ID":"27e9997a-0ffc-47e0-94c4-2c3352e52a8f","Type":"ContainerStarted","Data":"833ee8d6e64e4457bd1c5c421472616441c04511a12cf3b81ab4b5453d17dfd3"}
Nov 02 03:04:57.827010 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:04:57.826892    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6f6944bcdd-79j5r" podUID="27e9997a-0ffc-47e0-94c4-2c3352e52a8f" containerName="authorino" containerID="cri-o://833ee8d6e64e4457bd1c5c421472616441c04511a12cf3b81ab4b5453d17dfd3" gracePeriod=30
Nov 02 03:04:57.828494 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:04:57.828464    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6db4498c4d-9ml7r" event={"ID":"6f64e653-384e-4f15-8575-ac9602d531ee","Type":"ContainerStarted","Data":"a3e7d602b3502fd52a100330ec61282a7f4dbe883ed4cbd05e081ab442991c34"}
Nov 02 03:04:57.871213 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:04:57.871181    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6db4498c4d-9ml7r"]
Nov 02 03:04:57.893381 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:04:57.893350    4496 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-cdwxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body=
Nov 02 03:04:57.893487 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:04:57.893402    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cdwxx" podUID="41697786-2b0f-4cb2-8975-1942d727c987" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused"
Nov 02 03:04:57.922186 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:04:57.922158    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-78c9c77858-nq6sx"]
Nov 02 03:04:57.924452 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:04:57.924437    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-78c9c77858-nq6sx"
Nov 02 03:04:57.948636 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:04:57.948580    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6f6944bcdd-79j5r" podStartSLOduration=1.383725764 podStartE2EDuration="2.948561624s" totalImagesPullingTime="1.56483586s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:04:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:04:57.945329585 +0000 UTC m=+1038.523936966" watchObservedRunningTime="2026-11-02 03:04:57.948561624 +0000 UTC m=+1038.527169005"
Nov 02 03:04:57.964918 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:04:57.964888    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-78c9c77858-nq6sx"]
Nov 02 03:04:57.977718 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:04:57.977693    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Nov 02 03:04:58.032683 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:04:58.032656    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/72d5b418-688f-4a96-8129-4c2f81a06638-tls-cert\") pod \"authorino-78c9c77858-nq6sx\" (UID: \"72d5b418-688f-4a96-8129-4c2f81a06638\") " pod="kuadrant-system/authorino-78c9c77858-nq6sx"
Nov 02 03:04:58.032817 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:04:58.032728    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2g2gs\" (UniqueName: \"kubernetes.io/projected/72d5b418-688f-4a96-8129-4c2f81a06638-kube-api-access-2g2gs\") pod \"authorino-78c9c77858-nq6sx\" (UID: \"72d5b418-688f-4a96-8129-4c2f81a06638\") " pod="kuadrant-system/authorino-78c9c77858-nq6sx"
Nov 02 03:04:58.061039 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:04:58.060211    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6db4498c4d-9ml7r" podStartSLOduration=1.777132192 podStartE2EDuration="3.060192511s" totalImagesPullingTime="1.283060319s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:04:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:04:58.059863664 +0000 UTC m=+1038.638471030" watchObservedRunningTime="2026-11-02 03:04:58.060192511 +0000 UTC m=+1038.638799893"
Nov 02 03:04:58.130886 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:04:58.130861    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-79j5r"
Nov 02 03:04:58.146041 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:04:58.134284    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2g2gs\" (UniqueName: \"kubernetes.io/projected/72d5b418-688f-4a96-8129-4c2f81a06638-kube-api-access-2g2gs\") pod \"authorino-78c9c77858-nq6sx\" (UID: \"72d5b418-688f-4a96-8129-4c2f81a06638\") " pod="kuadrant-system/authorino-78c9c77858-nq6sx"
Nov 02 03:04:58.146041 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:04:58.134385    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/72d5b418-688f-4a96-8129-4c2f81a06638-tls-cert\") pod \"authorino-78c9c77858-nq6sx\" (UID: \"72d5b418-688f-4a96-8129-4c2f81a06638\") " pod="kuadrant-system/authorino-78c9c77858-nq6sx"
Nov 02 03:04:58.146041 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:04:58.138207    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/72d5b418-688f-4a96-8129-4c2f81a06638-tls-cert\") pod \"authorino-78c9c77858-nq6sx\" (UID: \"72d5b418-688f-4a96-8129-4c2f81a06638\") " pod="kuadrant-system/authorino-78c9c77858-nq6sx"
Nov 02 03:04:58.193062 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:04:58.193007    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2g2gs\" (UniqueName: \"kubernetes.io/projected/72d5b418-688f-4a96-8129-4c2f81a06638-kube-api-access-2g2gs\") pod \"authorino-78c9c77858-nq6sx\" (UID: \"72d5b418-688f-4a96-8129-4c2f81a06638\") " pod="kuadrant-system/authorino-78c9c77858-nq6sx"
Nov 02 03:04:58.238035 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:04:58.237636    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27e9997a-0ffc-47e0-94c4-2c3352e52a8f-kube-api-access-nqh76\" (UniqueName: \"kubernetes.io/projected/27e9997a-0ffc-47e0-94c4-2c3352e52a8f-kube-api-access-nqh76\") pod \"27e9997a-0ffc-47e0-94c4-2c3352e52a8f\" (UID: \"27e9997a-0ffc-47e0-94c4-2c3352e52a8f\") "
Nov 02 03:04:58.252502 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:04:58.245142    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27e9997a-0ffc-47e0-94c4-2c3352e52a8f-kube-api-access-nqh76" pod "27e9997a-0ffc-47e0-94c4-2c3352e52a8f" (UID: "27e9997a-0ffc-47e0-94c4-2c3352e52a8f"). InnerVolumeSpecName "kube-api-access-nqh76". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:04:58.252502 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:04:58.252272    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-78c9c77858-nq6sx"
Nov 02 03:04:58.363271 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:04:58.363154    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqh76\" (UniqueName: \"kubernetes.io/projected/27e9997a-0ffc-47e0-94c4-2c3352e52a8f-kube-api-access-nqh76\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:04:58.651932 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:04:58.651838    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-78c9c77858-nq6sx"]
Nov 02 03:04:58.845064 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:04:58.840581    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-78c9c77858-nq6sx" event={"ID":"72d5b418-688f-4a96-8129-4c2f81a06638","Type":"ContainerStarted","Data":"d4b830b88009e93d1698f46ddf9b2e2175bb9479e0b259fb43c25c11c2113174"}
Nov 02 03:04:58.845064 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:04:58.842136    4496 generic.go:356] "Generic (PLEG): container finished" podID="27e9997a-0ffc-47e0-94c4-2c3352e52a8f" containerID="833ee8d6e64e4457bd1c5c421472616441c04511a12cf3b81ab4b5453d17dfd3" exitCode=0
Nov 02 03:04:58.845064 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:04:58.842681    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-79j5r"
Nov 02 03:04:58.855053 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:04:58.850060    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-79j5r" event={"ID":"27e9997a-0ffc-47e0-94c4-2c3352e52a8f","Type":"ContainerDied","Data":"833ee8d6e64e4457bd1c5c421472616441c04511a12cf3b81ab4b5453d17dfd3"}
Nov 02 03:04:58.855053 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:04:58.850109    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-79j5r" event={"ID":"27e9997a-0ffc-47e0-94c4-2c3352e52a8f","Type":"ContainerDied","Data":"648521894aa5008bba3f33319b7bd05744d5610bb5041b4ab9bfcbdd20c9dd98"}
Nov 02 03:04:58.855053 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:04:58.850152    4496 scope.go:122] "RemoveContainer" containerID="833ee8d6e64e4457bd1c5c421472616441c04511a12cf3b81ab4b5453d17dfd3"
Nov 02 03:04:58.870265 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:04:58.870245    4496 scope.go:122] "RemoveContainer" containerID="833ee8d6e64e4457bd1c5c421472616441c04511a12cf3b81ab4b5453d17dfd3"
Nov 02 03:04:58.870573 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:04:58.870550    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"833ee8d6e64e4457bd1c5c421472616441c04511a12cf3b81ab4b5453d17dfd3\": container with ID starting with 833ee8d6e64e4457bd1c5c421472616441c04511a12cf3b81ab4b5453d17dfd3 not found: ID does not exist" containerID="833ee8d6e64e4457bd1c5c421472616441c04511a12cf3b81ab4b5453d17dfd3"
Nov 02 03:04:58.870616 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:04:58.870588    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"833ee8d6e64e4457bd1c5c421472616441c04511a12cf3b81ab4b5453d17dfd3"} err="failed to get container status \"833ee8d6e64e4457bd1c5c421472616441c04511a12cf3b81ab4b5453d17dfd3\": rpc error: code = NotFound desc = could not find container \"833ee8d6e64e4457bd1c5c421472616441c04511a12cf3b81ab4b5453d17dfd3\": container with ID starting with 833ee8d6e64e4457bd1c5c421472616441c04511a12cf3b81ab4b5453d17dfd3 not found: ID does not exist"
Nov 02 03:04:58.893119 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:04:58.893093    4496 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-cdwxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body=
Nov 02 03:04:58.893248 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:04:58.893138    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cdwxx" podUID="41697786-2b0f-4cb2-8975-1942d727c987" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused"
Nov 02 03:04:59.016347 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:04:59.016304    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-79j5r"]
Nov 02 03:04:59.020000 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:04:59.019965    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-79j5r"]
Nov 02 03:04:59.573185 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:04:59.573151    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27e9997a-0ffc-47e0-94c4-2c3352e52a8f" path="/var/lib/kubelet/pods/27e9997a-0ffc-47e0-94c4-2c3352e52a8f/volumes"
Nov 02 03:04:59.865136 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:04:59.864638    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6db4498c4d-9ml7r" podUID="6f64e653-384e-4f15-8575-ac9602d531ee" containerName="authorino" containerID="cri-o://a3e7d602b3502fd52a100330ec61282a7f4dbe883ed4cbd05e081ab442991c34" gracePeriod=30
Nov 02 03:04:59.892915 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:04:59.892889    4496 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-cdwxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body=
Nov 02 03:04:59.893040 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:04:59.892944    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cdwxx" podUID="41697786-2b0f-4cb2-8975-1942d727c987" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused"
Nov 02 03:05:00.172708 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:05:00.172683    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6db4498c4d-9ml7r"
Nov 02 03:05:00.187592 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:05:00.187561    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f64e653-384e-4f15-8575-ac9602d531ee-kube-api-access-hlwh8\" (UniqueName: \"kubernetes.io/projected/6f64e653-384e-4f15-8575-ac9602d531ee-kube-api-access-hlwh8\") pod \"6f64e653-384e-4f15-8575-ac9602d531ee\" (UID: \"6f64e653-384e-4f15-8575-ac9602d531ee\") "
Nov 02 03:05:00.189765 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:05:00.189740    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f64e653-384e-4f15-8575-ac9602d531ee-kube-api-access-hlwh8" pod "6f64e653-384e-4f15-8575-ac9602d531ee" (UID: "6f64e653-384e-4f15-8575-ac9602d531ee"). InnerVolumeSpecName "kube-api-access-hlwh8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:05:00.288862 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:05:00.288826    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlwh8\" (UniqueName: \"kubernetes.io/projected/6f64e653-384e-4f15-8575-ac9602d531ee-kube-api-access-hlwh8\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:05:00.871849 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:05:00.871805    4496 generic.go:356] "Generic (PLEG): container finished" podID="6f64e653-384e-4f15-8575-ac9602d531ee" containerID="a3e7d602b3502fd52a100330ec61282a7f4dbe883ed4cbd05e081ab442991c34" exitCode=0
Nov 02 03:05:00.872379 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:05:00.871927    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6db4498c4d-9ml7r" event={"ID":"6f64e653-384e-4f15-8575-ac9602d531ee","Type":"ContainerDied","Data":"a3e7d602b3502fd52a100330ec61282a7f4dbe883ed4cbd05e081ab442991c34"}
Nov 02 03:05:00.872379 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:05:00.871967    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6db4498c4d-9ml7r" event={"ID":"6f64e653-384e-4f15-8575-ac9602d531ee","Type":"ContainerDied","Data":"74057d784e867673fecb5b61ade93c3c6c0d5c2068266405942f04e2c32da9ff"}
Nov 02 03:05:00.872379 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:05:00.871990    4496 scope.go:122] "RemoveContainer" containerID="a3e7d602b3502fd52a100330ec61282a7f4dbe883ed4cbd05e081ab442991c34"
Nov 02 03:05:00.872379 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:05:00.872127    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6db4498c4d-9ml7r"
Nov 02 03:05:00.876099 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:05:00.876064    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-78c9c77858-nq6sx" event={"ID":"72d5b418-688f-4a96-8129-4c2f81a06638","Type":"ContainerStarted","Data":"8d2970834ddcea14a12791eb93010d75ab00586b56d2aaad451f993690e74170"}
Nov 02 03:05:00.895381 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:05:00.894231    4496 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-cdwxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body=
Nov 02 03:05:00.895381 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:05:00.894282    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cdwxx" podUID="41697786-2b0f-4cb2-8975-1942d727c987" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused"
Nov 02 03:05:00.900144 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:05:00.900123    4496 scope.go:122] "RemoveContainer" containerID="a3e7d602b3502fd52a100330ec61282a7f4dbe883ed4cbd05e081ab442991c34"
Nov 02 03:05:00.901110 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:05:00.900830    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3e7d602b3502fd52a100330ec61282a7f4dbe883ed4cbd05e081ab442991c34\": container with ID starting with a3e7d602b3502fd52a100330ec61282a7f4dbe883ed4cbd05e081ab442991c34 not found: ID does not exist" containerID="a3e7d602b3502fd52a100330ec61282a7f4dbe883ed4cbd05e081ab442991c34"
Nov 02 03:05:00.901110 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:05:00.900940    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3e7d602b3502fd52a100330ec61282a7f4dbe883ed4cbd05e081ab442991c34"} err="failed to get container status \"a3e7d602b3502fd52a100330ec61282a7f4dbe883ed4cbd05e081ab442991c34\": rpc error: code = NotFound desc = could not find container \"a3e7d602b3502fd52a100330ec61282a7f4dbe883ed4cbd05e081ab442991c34\": container with ID starting with a3e7d602b3502fd52a100330ec61282a7f4dbe883ed4cbd05e081ab442991c34 not found: ID does not exist"
Nov 02 03:05:01.055042 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:05:01.054221    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-78c9c77858-nq6sx" podStartSLOduration=2.704726079 podStartE2EDuration="4.054201734s" totalImagesPullingTime="1.349475655s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:04:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:05:01.050487814 +0000 UTC m=+1041.629095219" watchObservedRunningTime="2026-11-02 03:05:01.054201734 +0000 UTC m=+1041.632809115"
Nov 02 03:05:01.076129 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:05:01.076094    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-nxjmt"]
Nov 02 03:05:01.076579 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:05:01.076543    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-nxjmt" podUID="a24f74eb-7859-4958-b4d1-8fff3f1371e6" containerName="authorino" containerID="cri-o://89811980c760fbe44ea89568a2f14fd92e7d3d5877a8da9d848f7f7c3de37b5e" gracePeriod=30
Nov 02 03:05:01.165731 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:05:01.165626    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6db4498c4d-9ml7r"]
Nov 02 03:05:01.168843 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:05:01.168814    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6db4498c4d-9ml7r"]
Nov 02 03:05:01.387186 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:05:01.387161    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-nxjmt"
Nov 02 03:05:01.401767 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:05:01.401158    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a24f74eb-7859-4958-b4d1-8fff3f1371e6-kube-api-access-wr7pc\" (UniqueName: \"kubernetes.io/projected/a24f74eb-7859-4958-b4d1-8fff3f1371e6-kube-api-access-wr7pc\") pod \"a24f74eb-7859-4958-b4d1-8fff3f1371e6\" (UID: \"a24f74eb-7859-4958-b4d1-8fff3f1371e6\") "
Nov 02 03:05:01.403510 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:05:01.403472    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a24f74eb-7859-4958-b4d1-8fff3f1371e6-kube-api-access-wr7pc" pod "a24f74eb-7859-4958-b4d1-8fff3f1371e6" (UID: "a24f74eb-7859-4958-b4d1-8fff3f1371e6"). InnerVolumeSpecName "kube-api-access-wr7pc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:05:01.502573 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:05:01.502536    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wr7pc\" (UniqueName: \"kubernetes.io/projected/a24f74eb-7859-4958-b4d1-8fff3f1371e6-kube-api-access-wr7pc\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:05:01.575155 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:05:01.575118    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f64e653-384e-4f15-8575-ac9602d531ee" path="/var/lib/kubelet/pods/6f64e653-384e-4f15-8575-ac9602d531ee/volumes"
Nov 02 03:05:01.889674 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:05:01.889575    4496 generic.go:356] "Generic (PLEG): container finished" podID="a24f74eb-7859-4958-b4d1-8fff3f1371e6" containerID="89811980c760fbe44ea89568a2f14fd92e7d3d5877a8da9d848f7f7c3de37b5e" exitCode=0
Nov 02 03:05:01.889674 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:05:01.889632    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-nxjmt"
Nov 02 03:05:01.889674 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:05:01.889655    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-nxjmt" event={"ID":"a24f74eb-7859-4958-b4d1-8fff3f1371e6","Type":"ContainerDied","Data":"89811980c760fbe44ea89568a2f14fd92e7d3d5877a8da9d848f7f7c3de37b5e"}
Nov 02 03:05:01.890257 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:05:01.889699    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-nxjmt" event={"ID":"a24f74eb-7859-4958-b4d1-8fff3f1371e6","Type":"ContainerDied","Data":"032a25b7ce4ee6d1c8cb1b01f95a4d59fa583239de46b6363be0988a6a6995db"}
Nov 02 03:05:01.890257 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:05:01.889716    4496 scope.go:122] "RemoveContainer" containerID="89811980c760fbe44ea89568a2f14fd92e7d3d5877a8da9d848f7f7c3de37b5e"
Nov 02 03:05:01.893590 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:05:01.893568    4496 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-cdwxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body=
Nov 02 03:05:01.893710 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:05:01.893617    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cdwxx" podUID="41697786-2b0f-4cb2-8975-1942d727c987" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused"
Nov 02 03:05:01.899646 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:05:01.899625    4496 scope.go:122] "RemoveContainer" containerID="89811980c760fbe44ea89568a2f14fd92e7d3d5877a8da9d848f7f7c3de37b5e"
Nov 02 03:05:01.899939 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:05:01.899914    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89811980c760fbe44ea89568a2f14fd92e7d3d5877a8da9d848f7f7c3de37b5e\": container with ID starting with 89811980c760fbe44ea89568a2f14fd92e7d3d5877a8da9d848f7f7c3de37b5e not found: ID does not exist" containerID="89811980c760fbe44ea89568a2f14fd92e7d3d5877a8da9d848f7f7c3de37b5e"
Nov 02 03:05:01.899998 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:05:01.899940    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89811980c760fbe44ea89568a2f14fd92e7d3d5877a8da9d848f7f7c3de37b5e"} err="failed to get container status \"89811980c760fbe44ea89568a2f14fd92e7d3d5877a8da9d848f7f7c3de37b5e\": rpc error: code = NotFound desc = could not find container \"89811980c760fbe44ea89568a2f14fd92e7d3d5877a8da9d848f7f7c3de37b5e\": container with ID starting with 89811980c760fbe44ea89568a2f14fd92e7d3d5877a8da9d848f7f7c3de37b5e not found: ID does not exist"
Nov 02 03:05:02.063132 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:05:02.063094    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-nxjmt"]
Nov 02 03:05:02.069210 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:05:02.069177    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-nxjmt"]
Nov 02 03:05:02.896446 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:05:02.896414    4496 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cdwxx"
Nov 02 03:05:02.896922 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:05:02.896709    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cdwxx"
Nov 02 03:05:02.897562 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:05:02.897543    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cdwxx"
Nov 02 03:05:03.525373 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:05:03.525342    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 03:05:03.571409 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:05:03.571380    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a24f74eb-7859-4958-b4d1-8fff3f1371e6" path="/var/lib/kubelet/pods/a24f74eb-7859-4958-b4d1-8fff3f1371e6/volumes"
Nov 02 03:05:33.525234 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:05:33.525197    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 03:05:36.774394 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:05:36.774352    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-7tpjk"]
Nov 02 03:05:36.774915 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:05:36.774897    4496 cpu_manager.go:414] "containerMap: removing container" podUID="6f64e653-384e-4f15-8575-ac9602d531ee" containerName="authorino"
Nov 02 03:05:36.774961 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:05:36.774917    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f64e653-384e-4f15-8575-ac9602d531ee" containerName="authorino"
Nov 02 03:05:36.774961 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:05:36.774942    4496 cpu_manager.go:414] "containerMap: removing container" podUID="a24f74eb-7859-4958-b4d1-8fff3f1371e6" containerName="authorino"
Nov 02 03:05:36.774961 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:05:36.774947    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a24f74eb-7859-4958-b4d1-8fff3f1371e6" containerName="authorino"
Nov 02 03:05:36.774961 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:05:36.774956    4496 cpu_manager.go:414] "containerMap: removing container" podUID="27e9997a-0ffc-47e0-94c4-2c3352e52a8f" containerName="authorino"
Nov 02 03:05:36.774961 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:05:36.774962    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27e9997a-0ffc-47e0-94c4-2c3352e52a8f" containerName="authorino"
Nov 02 03:05:36.775138 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:05:36.775043    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="a24f74eb-7859-4958-b4d1-8fff3f1371e6" containerName="authorino"
Nov 02 03:05:36.775138 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:05:36.775053    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="27e9997a-0ffc-47e0-94c4-2c3352e52a8f" containerName="authorino"
Nov 02 03:05:36.775138 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:05:36.775062    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f64e653-384e-4f15-8575-ac9602d531ee" containerName="authorino"
Nov 02 03:05:36.777164 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:05:36.777149    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-7tpjk"
Nov 02 03:05:36.888112 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:05:36.888079    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\""
Nov 02 03:05:36.888112 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:05:36.888089    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\""
Nov 02 03:05:36.888350 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:05:36.888163    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-qspdt\""
Nov 02 03:05:36.890186 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:05:36.890161    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-7tpjk"]
Nov 02 03:05:36.916587 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:05:36.916559    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c2ed470-2715-4757-851b-c9474a7d5b50-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-7tpjk\" (UID: \"2c2ed470-2715-4757-851b-c9474a7d5b50\") " pod="opendatahub/ai-gateway-controller-7d85675f98-7tpjk"
Nov 02 03:05:36.916726 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:05:36.916616    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4pjz\" (UniqueName: \"kubernetes.io/projected/2c2ed470-2715-4757-851b-c9474a7d5b50-kube-api-access-w4pjz\") pod \"ai-gateway-controller-7d85675f98-7tpjk\" (UID: \"2c2ed470-2715-4757-851b-c9474a7d5b50\") " pod="opendatahub/ai-gateway-controller-7d85675f98-7tpjk"
Nov 02 03:05:37.017182 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:05:37.017155    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c2ed470-2715-4757-851b-c9474a7d5b50-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-7tpjk\" (UID: \"2c2ed470-2715-4757-851b-c9474a7d5b50\") " pod="opendatahub/ai-gateway-controller-7d85675f98-7tpjk"
Nov 02 03:05:37.017331 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:05:37.017199    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4pjz\" (UniqueName: \"kubernetes.io/projected/2c2ed470-2715-4757-851b-c9474a7d5b50-kube-api-access-w4pjz\") pod \"ai-gateway-controller-7d85675f98-7tpjk\" (UID: \"2c2ed470-2715-4757-851b-c9474a7d5b50\") " pod="opendatahub/ai-gateway-controller-7d85675f98-7tpjk"
Nov 02 03:05:37.019538 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:05:37.019512    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c2ed470-2715-4757-851b-c9474a7d5b50-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-7tpjk\" (UID: \"2c2ed470-2715-4757-851b-c9474a7d5b50\") " pod="opendatahub/ai-gateway-controller-7d85675f98-7tpjk"
Nov 02 03:05:37.078759 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:05:37.078688    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4pjz\" (UniqueName: \"kubernetes.io/projected/2c2ed470-2715-4757-851b-c9474a7d5b50-kube-api-access-w4pjz\") pod \"ai-gateway-controller-7d85675f98-7tpjk\" (UID: \"2c2ed470-2715-4757-851b-c9474a7d5b50\") " pod="opendatahub/ai-gateway-controller-7d85675f98-7tpjk"
Nov 02 03:05:37.089067 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:05:37.089045    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-7tpjk"
Nov 02 03:05:37.464508 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:05:37.464478    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-7tpjk"]
Nov 02 03:05:37.465579 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:05:37.465547    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c2ed470_2715_4757_851b_c9474a7d5b50.slice/crio-d80e48c07bd49cb29a98f05ac7fc06d61e3c8585200b3926b83126454fa8c68c WatchSource:0}: Error finding container d80e48c07bd49cb29a98f05ac7fc06d61e3c8585200b3926b83126454fa8c68c: Status 404 returned error can't find the container with id d80e48c07bd49cb29a98f05ac7fc06d61e3c8585200b3926b83126454fa8c68c
Nov 02 03:05:38.082707 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:05:38.082666    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-7tpjk" event={"ID":"2c2ed470-2715-4757-851b-c9474a7d5b50","Type":"ContainerStarted","Data":"d80e48c07bd49cb29a98f05ac7fc06d61e3c8585200b3926b83126454fa8c68c"}
Nov 02 03:05:40.494581 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:05:40.494547    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\""
Nov 02 03:05:41.101005 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:05:41.100971    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-7tpjk" event={"ID":"2c2ed470-2715-4757-851b-c9474a7d5b50","Type":"ContainerStarted","Data":"b1db6bd14880c6005fd2c834434a6b170e72fadece29810b89530a6e896d0a7a"}
Nov 02 03:05:41.101210 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:05:41.101060    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-7d85675f98-7tpjk"
Nov 02 03:05:41.222310 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:05:41.222260    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-7d85675f98-7tpjk" podStartSLOduration=2.247754479 podStartE2EDuration="5.22224676s" totalImagesPullingTime="2.974492281s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:05:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:05:41.220449011 +0000 UTC m=+1081.799056392" watchObservedRunningTime="2026-11-02 03:05:41.22224676 +0000 UTC m=+1081.800854139"
Nov 02 03:05:44.779155 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:05:44.779121    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-xmc9j"]
Nov 02 03:05:44.781518 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:05:44.781496    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-xmc9j"
Nov 02 03:05:44.886733 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:05:44.886699    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lq9rw\" (UniqueName: \"kubernetes.io/projected/99a7ac28-5904-4047-acc6-571f7aa78e35-kube-api-access-lq9rw\") pod \"payload-pre-processing-7b98bb678c-xmc9j\" (UID: \"99a7ac28-5904-4047-acc6-571f7aa78e35\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-xmc9j"
Nov 02 03:05:44.886902 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:05:44.886744    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99a7ac28-5904-4047-acc6-571f7aa78e35-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-xmc9j\" (UID: \"99a7ac28-5904-4047-acc6-571f7aa78e35\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-xmc9j"
Nov 02 03:05:44.887152 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:05:44.887134    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Nov 02 03:05:44.887275 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:05:44.887254    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-9x25j\""
Nov 02 03:05:44.889378 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:05:44.889359    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-xmc9j"]
Nov 02 03:05:44.943266 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:05:44.943231    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-gzc2f"]
Nov 02 03:05:44.945722 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:05:44.945705    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-gzc2f"
Nov 02 03:05:44.987439 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:05:44.987399    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99a7ac28-5904-4047-acc6-571f7aa78e35-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-xmc9j\" (UID: \"99a7ac28-5904-4047-acc6-571f7aa78e35\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-xmc9j"
Nov 02 03:05:44.987641 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:05:44.987595    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd8a1b95-491e-44ea-988a-e4437c386a1a-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-gzc2f\" (UID: \"bd8a1b95-491e-44ea-988a-e4437c386a1a\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-gzc2f"
Nov 02 03:05:44.987712 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:05:44.987638    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lrd6\" (UniqueName: \"kubernetes.io/projected/bd8a1b95-491e-44ea-988a-e4437c386a1a-kube-api-access-8lrd6\") pod \"payload-processing-55fbbbf9dc-gzc2f\" (UID: \"bd8a1b95-491e-44ea-988a-e4437c386a1a\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-gzc2f"
Nov 02 03:05:44.987712 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:05:44.987668    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lq9rw\" (UniqueName: \"kubernetes.io/projected/99a7ac28-5904-4047-acc6-571f7aa78e35-kube-api-access-lq9rw\") pod \"payload-pre-processing-7b98bb678c-xmc9j\" (UID: \"99a7ac28-5904-4047-acc6-571f7aa78e35\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-xmc9j"
Nov 02 03:05:44.988264 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:05:44.988240    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99a7ac28-5904-4047-acc6-571f7aa78e35-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-xmc9j\" (UID: \"99a7ac28-5904-4047-acc6-571f7aa78e35\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-xmc9j"
Nov 02 03:05:45.004077 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:05:45.004047    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-gzc2f"]
Nov 02 03:05:45.045714 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:05:45.045635    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lq9rw\" (UniqueName: \"kubernetes.io/projected/99a7ac28-5904-4047-acc6-571f7aa78e35-kube-api-access-lq9rw\") pod \"payload-pre-processing-7b98bb678c-xmc9j\" (UID: \"99a7ac28-5904-4047-acc6-571f7aa78e35\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-xmc9j"
Nov 02 03:05:45.088524 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:05:45.088491    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd8a1b95-491e-44ea-988a-e4437c386a1a-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-gzc2f\" (UID: \"bd8a1b95-491e-44ea-988a-e4437c386a1a\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-gzc2f"
Nov 02 03:05:45.088680 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:05:45.088536    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lrd6\" (UniqueName: \"kubernetes.io/projected/bd8a1b95-491e-44ea-988a-e4437c386a1a-kube-api-access-8lrd6\") pod \"payload-processing-55fbbbf9dc-gzc2f\" (UID: \"bd8a1b95-491e-44ea-988a-e4437c386a1a\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-gzc2f"
Nov 02 03:05:45.089184 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:05:45.089164    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd8a1b95-491e-44ea-988a-e4437c386a1a-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-gzc2f\" (UID: \"bd8a1b95-491e-44ea-988a-e4437c386a1a\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-gzc2f"
Nov 02 03:05:45.093457 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:05:45.093432    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-xmc9j"
Nov 02 03:05:45.147208 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:05:45.147174    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lrd6\" (UniqueName: \"kubernetes.io/projected/bd8a1b95-491e-44ea-988a-e4437c386a1a-kube-api-access-8lrd6\") pod \"payload-processing-55fbbbf9dc-gzc2f\" (UID: \"bd8a1b95-491e-44ea-988a-e4437c386a1a\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-gzc2f"
Nov 02 03:05:45.257682 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:05:45.257653    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-gzc2f"
Nov 02 03:05:45.465247 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:05:45.465219    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-xmc9j"]
Nov 02 03:05:45.467139 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:05:45.467112    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99a7ac28_5904_4047_acc6_571f7aa78e35.slice/crio-c6b340fdd079a4c041c74bae2561454b54fce784116afdf41be7f2880ca28d2a WatchSource:0}: Error finding container c6b340fdd079a4c041c74bae2561454b54fce784116afdf41be7f2880ca28d2a: Status 404 returned error can't find the container with id c6b340fdd079a4c041c74bae2561454b54fce784116afdf41be7f2880ca28d2a
Nov 02 03:05:45.627254 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:05:45.627229    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-gzc2f"]
Nov 02 03:05:45.628975 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:05:45.628948    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd8a1b95_491e_44ea_988a_e4437c386a1a.slice/crio-ce4263eb57435836afafa768ce2ad15903e03b1b1d15e9e7aced45e1f9291929 WatchSource:0}: Error finding container ce4263eb57435836afafa768ce2ad15903e03b1b1d15e9e7aced45e1f9291929: Status 404 returned error can't find the container with id ce4263eb57435836afafa768ce2ad15903e03b1b1d15e9e7aced45e1f9291929
Nov 02 03:05:46.128319 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:05:46.128281    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-xmc9j" event={"ID":"99a7ac28-5904-4047-acc6-571f7aa78e35","Type":"ContainerStarted","Data":"c6b340fdd079a4c041c74bae2561454b54fce784116afdf41be7f2880ca28d2a"}
Nov 02 03:05:46.129453 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:05:46.129429    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-gzc2f" event={"ID":"bd8a1b95-491e-44ea-988a-e4437c386a1a","Type":"ContainerStarted","Data":"ce4263eb57435836afafa768ce2ad15903e03b1b1d15e9e7aced45e1f9291929"}
Nov 02 03:05:49.150765 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:05:49.150730    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-xmc9j" event={"ID":"99a7ac28-5904-4047-acc6-571f7aa78e35","Type":"ContainerStarted","Data":"ffce7e36ca05b737836bc07a9f7c50713eb2be7f1d85cc4231111d7bd6078fe8"}
Nov 02 03:05:49.151230 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:05:49.150837    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-xmc9j"
Nov 02 03:05:49.152251 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:05:49.152229    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-gzc2f" event={"ID":"bd8a1b95-491e-44ea-988a-e4437c386a1a","Type":"ContainerStarted","Data":"736d9b25614590c9271caec4b6b16832ac1b4beb618748ef324bb08e9a719ad1"}
Nov 02 03:05:49.152318 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:05:49.152285    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-gzc2f"
Nov 02 03:05:49.266422 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:05:49.266372    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-xmc9j" podStartSLOduration=2.442455969 podStartE2EDuration="5.266359208s" totalImagesPullingTime="2.823903239s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:05:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:05:49.264829186 +0000 UTC m=+1089.843436566" watchObservedRunningTime="2026-11-02 03:05:49.266359208 +0000 UTC m=+1089.844966587"
Nov 02 03:05:49.379503 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:05:49.379452    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-55fbbbf9dc-gzc2f" podStartSLOduration=2.36334304 podStartE2EDuration="5.379436684s" totalImagesPullingTime="3.016093644s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:05:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:05:49.378788649 +0000 UTC m=+1089.957396029" watchObservedRunningTime="2026-11-02 03:05:49.379436684 +0000 UTC m=+1089.958044065"
Nov 02 03:05:52.108223 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:05:52.108192    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-7d85675f98-7tpjk"
Nov 02 03:05:52.441116 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:05:52.441010    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-7fcb66b96b-dwcmz"]
Nov 02 03:05:52.446187 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:05:52.446163    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7fcb66b96b-dwcmz"
Nov 02 03:05:52.499182 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:05:52.499153    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7fcb66b96b-dwcmz"]
Nov 02 03:05:52.560637 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:05:52.560600    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxrhn\" (UniqueName: \"kubernetes.io/projected/9dd13471-dd39-4838-be2a-dd99db00133b-kube-api-access-zxrhn\") pod \"ai-gateway-controller-7fcb66b96b-dwcmz\" (UID: \"9dd13471-dd39-4838-be2a-dd99db00133b\") " pod="opendatahub/ai-gateway-controller-7fcb66b96b-dwcmz"
Nov 02 03:05:52.560772 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:05:52.560756    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9dd13471-dd39-4838-be2a-dd99db00133b-metrics-certs\") pod \"ai-gateway-controller-7fcb66b96b-dwcmz\" (UID: \"9dd13471-dd39-4838-be2a-dd99db00133b\") " pod="opendatahub/ai-gateway-controller-7fcb66b96b-dwcmz"
Nov 02 03:05:52.661410 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:05:52.661378    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9dd13471-dd39-4838-be2a-dd99db00133b-metrics-certs\") pod \"ai-gateway-controller-7fcb66b96b-dwcmz\" (UID: \"9dd13471-dd39-4838-be2a-dd99db00133b\") " pod="opendatahub/ai-gateway-controller-7fcb66b96b-dwcmz"
Nov 02 03:05:52.661572 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:05:52.661446    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxrhn\" (UniqueName: \"kubernetes.io/projected/9dd13471-dd39-4838-be2a-dd99db00133b-kube-api-access-zxrhn\") pod \"ai-gateway-controller-7fcb66b96b-dwcmz\" (UID: \"9dd13471-dd39-4838-be2a-dd99db00133b\") " pod="opendatahub/ai-gateway-controller-7fcb66b96b-dwcmz"
Nov 02 03:05:52.663805 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:05:52.663776    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9dd13471-dd39-4838-be2a-dd99db00133b-metrics-certs\") pod \"ai-gateway-controller-7fcb66b96b-dwcmz\" (UID: \"9dd13471-dd39-4838-be2a-dd99db00133b\") " pod="opendatahub/ai-gateway-controller-7fcb66b96b-dwcmz"
Nov 02 03:05:52.720304 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:05:52.720241    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxrhn\" (UniqueName: \"kubernetes.io/projected/9dd13471-dd39-4838-be2a-dd99db00133b-kube-api-access-zxrhn\") pod \"ai-gateway-controller-7fcb66b96b-dwcmz\" (UID: \"9dd13471-dd39-4838-be2a-dd99db00133b\") " pod="opendatahub/ai-gateway-controller-7fcb66b96b-dwcmz"
Nov 02 03:05:52.759055 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:05:52.759030    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7fcb66b96b-dwcmz"
Nov 02 03:05:53.130802 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:05:53.130769    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7fcb66b96b-dwcmz"]
Nov 02 03:05:53.131913 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:05:53.131867    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dd13471_dd39_4838_be2a_dd99db00133b.slice/crio-9d30122c34f1aa0f999c91bf41090b2ce4fc53ea240d0f41c1cd32b46bbb56ef WatchSource:0}: Error finding container 9d30122c34f1aa0f999c91bf41090b2ce4fc53ea240d0f41c1cd32b46bbb56ef: Status 404 returned error can't find the container with id 9d30122c34f1aa0f999c91bf41090b2ce4fc53ea240d0f41c1cd32b46bbb56ef
Nov 02 03:05:53.172162 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:05:53.172134    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7fcb66b96b-dwcmz" event={"ID":"9dd13471-dd39-4838-be2a-dd99db00133b","Type":"ContainerStarted","Data":"9d30122c34f1aa0f999c91bf41090b2ce4fc53ea240d0f41c1cd32b46bbb56ef"}
Nov 02 03:05:54.178637 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:05:54.178599    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7fcb66b96b-dwcmz" event={"ID":"9dd13471-dd39-4838-be2a-dd99db00133b","Type":"ContainerStarted","Data":"80fdaaeceba306982da150a73db96cee5520151991901ddfe76270f173afeee3"}
Nov 02 03:05:54.179094 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:05:54.178664    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-7fcb66b96b-dwcmz"
Nov 02 03:05:54.293514 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:05:54.293405    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-7fcb66b96b-dwcmz" podStartSLOduration=1.415351641 podStartE2EDuration="2.29339093s" totalImagesPullingTime="878.039289ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:05:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:05:54.292484661 +0000 UTC m=+1094.871092044" watchObservedRunningTime="2026-11-02 03:05:54.29339093 +0000 UTC m=+1094.871998310"
Nov 02 03:06:00.160077 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:00.160046    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-xmc9j"
Nov 02 03:06:00.160561 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:00.160124    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-gzc2f"
Nov 02 03:06:03.525206 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:06:03.525166    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 03:06:05.185984 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:05.185957    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-7fcb66b96b-dwcmz"
Nov 02 03:06:05.326417 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:05.326383    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-7tpjk"]
Nov 02 03:06:05.326581 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:05.326561    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-7d85675f98-7tpjk" podUID="2c2ed470-2715-4757-851b-c9474a7d5b50" containerName="manager" containerID="cri-o://b1db6bd14880c6005fd2c834434a6b170e72fadece29810b89530a6e896d0a7a" gracePeriod=30
Nov 02 03:06:05.575436 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:05.575409    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-7tpjk"
Nov 02 03:06:05.676185 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:05.676150    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c2ed470-2715-4757-851b-c9474a7d5b50-kube-api-access-w4pjz\" (UniqueName: \"kubernetes.io/projected/2c2ed470-2715-4757-851b-c9474a7d5b50-kube-api-access-w4pjz\") pod \"2c2ed470-2715-4757-851b-c9474a7d5b50\" (UID: \"2c2ed470-2715-4757-851b-c9474a7d5b50\") "
Nov 02 03:06:05.676353 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:05.676245    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c2ed470-2715-4757-851b-c9474a7d5b50-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c2ed470-2715-4757-851b-c9474a7d5b50-metrics-certs\") pod \"2c2ed470-2715-4757-851b-c9474a7d5b50\" (UID: \"2c2ed470-2715-4757-851b-c9474a7d5b50\") "
Nov 02 03:06:05.678402 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:05.678372    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c2ed470-2715-4757-851b-c9474a7d5b50-metrics-certs" pod "2c2ed470-2715-4757-851b-c9474a7d5b50" (UID: "2c2ed470-2715-4757-851b-c9474a7d5b50"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 03:06:05.678402 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:05.678397    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c2ed470-2715-4757-851b-c9474a7d5b50-kube-api-access-w4pjz" pod "2c2ed470-2715-4757-851b-c9474a7d5b50" (UID: "2c2ed470-2715-4757-851b-c9474a7d5b50"). InnerVolumeSpecName "kube-api-access-w4pjz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:06:05.777103 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:05.777072    4496 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c2ed470-2715-4757-851b-c9474a7d5b50-metrics-certs\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:06:05.777103 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:05.777100    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4pjz\" (UniqueName: \"kubernetes.io/projected/2c2ed470-2715-4757-851b-c9474a7d5b50-kube-api-access-w4pjz\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:06:06.248599 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:06.248568    4496 generic.go:356] "Generic (PLEG): container finished" podID="2c2ed470-2715-4757-851b-c9474a7d5b50" containerID="b1db6bd14880c6005fd2c834434a6b170e72fadece29810b89530a6e896d0a7a" exitCode=0
Nov 02 03:06:06.248983 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:06.248605    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-7tpjk" event={"ID":"2c2ed470-2715-4757-851b-c9474a7d5b50","Type":"ContainerDied","Data":"b1db6bd14880c6005fd2c834434a6b170e72fadece29810b89530a6e896d0a7a"}
Nov 02 03:06:06.248983 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:06.248615    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-7tpjk"
Nov 02 03:06:06.248983 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:06.248659    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-7tpjk" event={"ID":"2c2ed470-2715-4757-851b-c9474a7d5b50","Type":"ContainerDied","Data":"d80e48c07bd49cb29a98f05ac7fc06d61e3c8585200b3926b83126454fa8c68c"}
Nov 02 03:06:06.248983 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:06.248682    4496 scope.go:122] "RemoveContainer" containerID="b1db6bd14880c6005fd2c834434a6b170e72fadece29810b89530a6e896d0a7a"
Nov 02 03:06:06.260831 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:06.260813    4496 scope.go:122] "RemoveContainer" containerID="b1db6bd14880c6005fd2c834434a6b170e72fadece29810b89530a6e896d0a7a"
Nov 02 03:06:06.261143 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:06:06.261122    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1db6bd14880c6005fd2c834434a6b170e72fadece29810b89530a6e896d0a7a\": container with ID starting with b1db6bd14880c6005fd2c834434a6b170e72fadece29810b89530a6e896d0a7a not found: ID does not exist" containerID="b1db6bd14880c6005fd2c834434a6b170e72fadece29810b89530a6e896d0a7a"
Nov 02 03:06:06.261216 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:06.261153    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1db6bd14880c6005fd2c834434a6b170e72fadece29810b89530a6e896d0a7a"} err="failed to get container status \"b1db6bd14880c6005fd2c834434a6b170e72fadece29810b89530a6e896d0a7a\": rpc error: code = NotFound desc = could not find container \"b1db6bd14880c6005fd2c834434a6b170e72fadece29810b89530a6e896d0a7a\": container with ID starting with b1db6bd14880c6005fd2c834434a6b170e72fadece29810b89530a6e896d0a7a not found: ID does not exist"
Nov 02 03:06:06.419968 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:06.419939    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-7tpjk"]
Nov 02 03:06:06.424216 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:06.424190    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-7tpjk"]
Nov 02 03:06:07.577874 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:07.577836    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c2ed470-2715-4757-851b-c9474a7d5b50" path="/var/lib/kubelet/pods/2c2ed470-2715-4757-851b-c9474a7d5b50/volumes"
Nov 02 03:06:21.717135 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:21.717098    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-bm2wq"]
Nov 02 03:06:21.717628 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:21.717488    4496 cpu_manager.go:414] "containerMap: removing container" podUID="2c2ed470-2715-4757-851b-c9474a7d5b50" containerName="manager"
Nov 02 03:06:21.717628 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:21.717500    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c2ed470-2715-4757-851b-c9474a7d5b50" containerName="manager"
Nov 02 03:06:21.717628 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:21.717590    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c2ed470-2715-4757-851b-c9474a7d5b50" containerName="manager"
Nov 02 03:06:21.723438 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:21.723415    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bm2wq"
Nov 02 03:06:21.777975 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:21.777939    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-bm2wq"]
Nov 02 03:06:21.825369 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:21.825332    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Nov 02 03:06:21.825530 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:21.825378    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Nov 02 03:06:21.825530 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:21.825460    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Nov 02 03:06:21.827930 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:21.827911    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-n5h79\""
Nov 02 03:06:21.919652 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:21.919606    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d81db86e-2a08-4c67-add9-b7353fb0372d-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bm2wq\" (UID: \"d81db86e-2a08-4c67-add9-b7353fb0372d\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bm2wq"
Nov 02 03:06:21.919836 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:21.919674    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d81db86e-2a08-4c67-add9-b7353fb0372d-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bm2wq\" (UID: \"d81db86e-2a08-4c67-add9-b7353fb0372d\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bm2wq"
Nov 02 03:06:21.919836 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:21.919729    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lhsj\" (UniqueName: \"kubernetes.io/projected/d81db86e-2a08-4c67-add9-b7353fb0372d-kube-api-access-7lhsj\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bm2wq\" (UID: \"d81db86e-2a08-4c67-add9-b7353fb0372d\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bm2wq"
Nov 02 03:06:21.919919 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:21.919867    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d81db86e-2a08-4c67-add9-b7353fb0372d-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bm2wq\" (UID: \"d81db86e-2a08-4c67-add9-b7353fb0372d\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bm2wq"
Nov 02 03:06:21.919919 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:21.919905    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d81db86e-2a08-4c67-add9-b7353fb0372d-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bm2wq\" (UID: \"d81db86e-2a08-4c67-add9-b7353fb0372d\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bm2wq"
Nov 02 03:06:21.920007 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:21.919933    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d81db86e-2a08-4c67-add9-b7353fb0372d-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bm2wq\" (UID: \"d81db86e-2a08-4c67-add9-b7353fb0372d\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bm2wq"
Nov 02 03:06:22.021188 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:22.021148    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d81db86e-2a08-4c67-add9-b7353fb0372d-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bm2wq\" (UID: \"d81db86e-2a08-4c67-add9-b7353fb0372d\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bm2wq"
Nov 02 03:06:22.021188 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:22.021181    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lhsj\" (UniqueName: \"kubernetes.io/projected/d81db86e-2a08-4c67-add9-b7353fb0372d-kube-api-access-7lhsj\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bm2wq\" (UID: \"d81db86e-2a08-4c67-add9-b7353fb0372d\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bm2wq"
Nov 02 03:06:22.021422 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:22.021227    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d81db86e-2a08-4c67-add9-b7353fb0372d-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bm2wq\" (UID: \"d81db86e-2a08-4c67-add9-b7353fb0372d\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bm2wq"
Nov 02 03:06:22.021422 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:22.021253    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d81db86e-2a08-4c67-add9-b7353fb0372d-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bm2wq\" (UID: \"d81db86e-2a08-4c67-add9-b7353fb0372d\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bm2wq"
Nov 02 03:06:22.021422 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:22.021283    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d81db86e-2a08-4c67-add9-b7353fb0372d-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bm2wq\" (UID: \"d81db86e-2a08-4c67-add9-b7353fb0372d\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bm2wq"
Nov 02 03:06:22.021422 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:22.021339    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d81db86e-2a08-4c67-add9-b7353fb0372d-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bm2wq\" (UID: \"d81db86e-2a08-4c67-add9-b7353fb0372d\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bm2wq"
Nov 02 03:06:22.021704 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:22.021679    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d81db86e-2a08-4c67-add9-b7353fb0372d-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bm2wq\" (UID: \"d81db86e-2a08-4c67-add9-b7353fb0372d\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bm2wq"
Nov 02 03:06:22.021835 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:22.021704    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d81db86e-2a08-4c67-add9-b7353fb0372d-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bm2wq\" (UID: \"d81db86e-2a08-4c67-add9-b7353fb0372d\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bm2wq"
Nov 02 03:06:22.021835 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:22.021738    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d81db86e-2a08-4c67-add9-b7353fb0372d-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bm2wq\" (UID: \"d81db86e-2a08-4c67-add9-b7353fb0372d\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bm2wq"
Nov 02 03:06:22.023681 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:22.023658    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d81db86e-2a08-4c67-add9-b7353fb0372d-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bm2wq\" (UID: \"d81db86e-2a08-4c67-add9-b7353fb0372d\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bm2wq"
Nov 02 03:06:22.023803 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:22.023783    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d81db86e-2a08-4c67-add9-b7353fb0372d-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bm2wq\" (UID: \"d81db86e-2a08-4c67-add9-b7353fb0372d\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bm2wq"
Nov 02 03:06:22.079297 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:22.079253    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lhsj\" (UniqueName: \"kubernetes.io/projected/d81db86e-2a08-4c67-add9-b7353fb0372d-kube-api-access-7lhsj\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bm2wq\" (UID: \"d81db86e-2a08-4c67-add9-b7353fb0372d\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bm2wq"
Nov 02 03:06:22.335207 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:22.335120    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bm2wq"
Nov 02 03:06:22.707006 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:22.706981    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-bm2wq"]
Nov 02 03:06:22.707944 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:06:22.707920    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd81db86e_2a08_4c67_add9_b7353fb0372d.slice/crio-4585710bfb897436a0507448f7feb91f190db0bfc0e251e3251501bb5e6c9fbf WatchSource:0}: Error finding container 4585710bfb897436a0507448f7feb91f190db0bfc0e251e3251501bb5e6c9fbf: Status 404 returned error can't find the container with id 4585710bfb897436a0507448f7feb91f190db0bfc0e251e3251501bb5e6c9fbf
Nov 02 03:06:23.338625 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:23.338593    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bm2wq" event={"ID":"d81db86e-2a08-4c67-add9-b7353fb0372d","Type":"ContainerStarted","Data":"4585710bfb897436a0507448f7feb91f190db0bfc0e251e3251501bb5e6c9fbf"}
Nov 02 03:06:24.345708 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:24.345669    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bm2wq" event={"ID":"d81db86e-2a08-4c67-add9-b7353fb0372d","Type":"ContainerStarted","Data":"43b08626d220b065e144815235ec4d81881413aed9577ded0c5cff46b53d3ebb"}
Nov 02 03:06:24.461458 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:24.461357    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bm2wq" podStartSLOduration=2.014041659 podStartE2EDuration="3.461343063s" totalImagesPullingTime="1.447301404s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:06:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:06:24.460693548 +0000 UTC m=+1125.039300928" watchObservedRunningTime="2026-11-02 03:06:24.461343063 +0000 UTC m=+1125.039950443"
Nov 02 03:06:25.257957 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:25.257922    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-vnxgf"]
Nov 02 03:06:25.262127 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:25.262103    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vnxgf"
Nov 02 03:06:25.320405 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:25.320368    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-vnxgf"]
Nov 02 03:06:25.335280 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:25.335250    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bm2wq"
Nov 02 03:06:25.351068 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:25.351003    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bebc74da-01c1-4547-ab5d-90a0a8ec1ad7-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vnxgf\" (UID: \"bebc74da-01c1-4547-ab5d-90a0a8ec1ad7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vnxgf"
Nov 02 03:06:25.351519 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:25.351120    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bebc74da-01c1-4547-ab5d-90a0a8ec1ad7-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vnxgf\" (UID: \"bebc74da-01c1-4547-ab5d-90a0a8ec1ad7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vnxgf"
Nov 02 03:06:25.351519 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:25.351172    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbvbv\" (UniqueName: \"kubernetes.io/projected/bebc74da-01c1-4547-ab5d-90a0a8ec1ad7-kube-api-access-fbvbv\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vnxgf\" (UID: \"bebc74da-01c1-4547-ab5d-90a0a8ec1ad7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vnxgf"
Nov 02 03:06:25.351519 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:25.351206    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bebc74da-01c1-4547-ab5d-90a0a8ec1ad7-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vnxgf\" (UID: \"bebc74da-01c1-4547-ab5d-90a0a8ec1ad7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vnxgf"
Nov 02 03:06:25.351519 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:25.351377    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bebc74da-01c1-4547-ab5d-90a0a8ec1ad7-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vnxgf\" (UID: \"bebc74da-01c1-4547-ab5d-90a0a8ec1ad7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vnxgf"
Nov 02 03:06:25.351519 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:25.351411    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bebc74da-01c1-4547-ab5d-90a0a8ec1ad7-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vnxgf\" (UID: \"bebc74da-01c1-4547-ab5d-90a0a8ec1ad7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vnxgf"
Nov 02 03:06:25.366649 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:25.366622    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Nov 02 03:06:25.457881 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:25.456840    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bebc74da-01c1-4547-ab5d-90a0a8ec1ad7-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vnxgf\" (UID: \"bebc74da-01c1-4547-ab5d-90a0a8ec1ad7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vnxgf"
Nov 02 03:06:25.457881 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:25.456922    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbvbv\" (UniqueName: \"kubernetes.io/projected/bebc74da-01c1-4547-ab5d-90a0a8ec1ad7-kube-api-access-fbvbv\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vnxgf\" (UID: \"bebc74da-01c1-4547-ab5d-90a0a8ec1ad7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vnxgf"
Nov 02 03:06:25.457881 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:25.456963    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bebc74da-01c1-4547-ab5d-90a0a8ec1ad7-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vnxgf\" (UID: \"bebc74da-01c1-4547-ab5d-90a0a8ec1ad7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vnxgf"
Nov 02 03:06:25.457881 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:25.457035    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bebc74da-01c1-4547-ab5d-90a0a8ec1ad7-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vnxgf\" (UID: \"bebc74da-01c1-4547-ab5d-90a0a8ec1ad7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vnxgf"
Nov 02 03:06:25.457881 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:25.457086    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bebc74da-01c1-4547-ab5d-90a0a8ec1ad7-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vnxgf\" (UID: \"bebc74da-01c1-4547-ab5d-90a0a8ec1ad7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vnxgf"
Nov 02 03:06:25.457881 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:25.457140    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bebc74da-01c1-4547-ab5d-90a0a8ec1ad7-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vnxgf\" (UID: \"bebc74da-01c1-4547-ab5d-90a0a8ec1ad7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vnxgf"
Nov 02 03:06:25.457881 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:25.457582    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bebc74da-01c1-4547-ab5d-90a0a8ec1ad7-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vnxgf\" (UID: \"bebc74da-01c1-4547-ab5d-90a0a8ec1ad7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vnxgf"
Nov 02 03:06:25.458695 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:25.458668    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bebc74da-01c1-4547-ab5d-90a0a8ec1ad7-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vnxgf\" (UID: \"bebc74da-01c1-4547-ab5d-90a0a8ec1ad7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vnxgf"
Nov 02 03:06:25.459425 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:25.459375    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bebc74da-01c1-4547-ab5d-90a0a8ec1ad7-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vnxgf\" (UID: \"bebc74da-01c1-4547-ab5d-90a0a8ec1ad7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vnxgf"
Nov 02 03:06:25.472962 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:25.472933    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bebc74da-01c1-4547-ab5d-90a0a8ec1ad7-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vnxgf\" (UID: \"bebc74da-01c1-4547-ab5d-90a0a8ec1ad7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vnxgf"
Nov 02 03:06:25.476615 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:25.476584    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bebc74da-01c1-4547-ab5d-90a0a8ec1ad7-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vnxgf\" (UID: \"bebc74da-01c1-4547-ab5d-90a0a8ec1ad7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vnxgf"
Nov 02 03:06:25.519427 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:25.519350    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbvbv\" (UniqueName: \"kubernetes.io/projected/bebc74da-01c1-4547-ab5d-90a0a8ec1ad7-kube-api-access-fbvbv\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vnxgf\" (UID: \"bebc74da-01c1-4547-ab5d-90a0a8ec1ad7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vnxgf"
Nov 02 03:06:25.577220 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:25.577173    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vnxgf"
Nov 02 03:06:25.967087 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:25.966989    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-vnxgf"]
Nov 02 03:06:25.972619 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:06:25.972583    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbebc74da_01c1_4547_ab5d_90a0a8ec1ad7.slice/crio-cb4f78f7cbd75903132568a0d9021356c7654c128d1b17431d8f6d1d4b7bbeb5 WatchSource:0}: Error finding container cb4f78f7cbd75903132568a0d9021356c7654c128d1b17431d8f6d1d4b7bbeb5: Status 404 returned error can't find the container with id cb4f78f7cbd75903132568a0d9021356c7654c128d1b17431d8f6d1d4b7bbeb5
Nov 02 03:06:26.358351 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:26.358314    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vnxgf" event={"ID":"bebc74da-01c1-4547-ab5d-90a0a8ec1ad7","Type":"ContainerStarted","Data":"cb4f78f7cbd75903132568a0d9021356c7654c128d1b17431d8f6d1d4b7bbeb5"}
Nov 02 03:06:27.364634 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:27.364597    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vnxgf" event={"ID":"bebc74da-01c1-4547-ab5d-90a0a8ec1ad7","Type":"ContainerStarted","Data":"ab4581ad730ba5cbe86714f6b1a69528e5c824921883d675075a20e4fcb87c9b"}
Nov 02 03:06:27.482758 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:27.482703    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vnxgf" podStartSLOduration=1.895663115 podStartE2EDuration="2.48268758s" totalImagesPullingTime="587.024465ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:06:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:06:27.482325655 +0000 UTC m=+1128.060933035" watchObservedRunningTime="2026-11-02 03:06:27.48268758 +0000 UTC m=+1128.061294957"
Nov 02 03:06:27.577398 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:27.577368    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vnxgf"
Nov 02 03:06:29.174604 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:29.174565    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-qlqqk"]
Nov 02 03:06:29.178201 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:29.178185    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qlqqk"
Nov 02 03:06:29.191840 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:29.191814    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c94d962-9ac1-4c2c-a6bb-8bb264d26f3e-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qlqqk\" (UID: \"3c94d962-9ac1-4c2c-a6bb-8bb264d26f3e\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qlqqk"
Nov 02 03:06:29.191973 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:29.191860    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c94d962-9ac1-4c2c-a6bb-8bb264d26f3e-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qlqqk\" (UID: \"3c94d962-9ac1-4c2c-a6bb-8bb264d26f3e\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qlqqk"
Nov 02 03:06:29.192036 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:29.191991    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c94d962-9ac1-4c2c-a6bb-8bb264d26f3e-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qlqqk\" (UID: \"3c94d962-9ac1-4c2c-a6bb-8bb264d26f3e\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qlqqk"
Nov 02 03:06:29.192078 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:29.192041    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cq9wj\" (UniqueName: \"kubernetes.io/projected/3c94d962-9ac1-4c2c-a6bb-8bb264d26f3e-kube-api-access-cq9wj\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qlqqk\" (UID: \"3c94d962-9ac1-4c2c-a6bb-8bb264d26f3e\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qlqqk"
Nov 02 03:06:29.192142 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:29.192105    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c94d962-9ac1-4c2c-a6bb-8bb264d26f3e-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qlqqk\" (UID: \"3c94d962-9ac1-4c2c-a6bb-8bb264d26f3e\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qlqqk"
Nov 02 03:06:29.192255 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:29.192236    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c94d962-9ac1-4c2c-a6bb-8bb264d26f3e-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qlqqk\" (UID: \"3c94d962-9ac1-4c2c-a6bb-8bb264d26f3e\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qlqqk"
Nov 02 03:06:29.282928 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:29.282890    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Nov 02 03:06:29.285297 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:29.285276    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-qlqqk"]
Nov 02 03:06:29.293296 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:29.293274    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c94d962-9ac1-4c2c-a6bb-8bb264d26f3e-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qlqqk\" (UID: \"3c94d962-9ac1-4c2c-a6bb-8bb264d26f3e\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qlqqk"
Nov 02 03:06:29.293400 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:29.293334    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c94d962-9ac1-4c2c-a6bb-8bb264d26f3e-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qlqqk\" (UID: \"3c94d962-9ac1-4c2c-a6bb-8bb264d26f3e\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qlqqk"
Nov 02 03:06:29.293594 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:29.293530    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c94d962-9ac1-4c2c-a6bb-8bb264d26f3e-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qlqqk\" (UID: \"3c94d962-9ac1-4c2c-a6bb-8bb264d26f3e\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qlqqk"
Nov 02 03:06:29.293688 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:29.293624    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c94d962-9ac1-4c2c-a6bb-8bb264d26f3e-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qlqqk\" (UID: \"3c94d962-9ac1-4c2c-a6bb-8bb264d26f3e\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qlqqk"
Nov 02 03:06:29.294039 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:29.293821    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c94d962-9ac1-4c2c-a6bb-8bb264d26f3e-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qlqqk\" (UID: \"3c94d962-9ac1-4c2c-a6bb-8bb264d26f3e\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qlqqk"
Nov 02 03:06:29.294039 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:29.293882    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cq9wj\" (UniqueName: \"kubernetes.io/projected/3c94d962-9ac1-4c2c-a6bb-8bb264d26f3e-kube-api-access-cq9wj\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qlqqk\" (UID: \"3c94d962-9ac1-4c2c-a6bb-8bb264d26f3e\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qlqqk"
Nov 02 03:06:29.294039 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:29.293931    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c94d962-9ac1-4c2c-a6bb-8bb264d26f3e-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qlqqk\" (UID: \"3c94d962-9ac1-4c2c-a6bb-8bb264d26f3e\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qlqqk"
Nov 02 03:06:29.294307 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:29.294127    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c94d962-9ac1-4c2c-a6bb-8bb264d26f3e-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qlqqk\" (UID: \"3c94d962-9ac1-4c2c-a6bb-8bb264d26f3e\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qlqqk"
Nov 02 03:06:29.294307 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:29.294196    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c94d962-9ac1-4c2c-a6bb-8bb264d26f3e-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qlqqk\" (UID: \"3c94d962-9ac1-4c2c-a6bb-8bb264d26f3e\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qlqqk"
Nov 02 03:06:29.296217 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:29.296194    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c94d962-9ac1-4c2c-a6bb-8bb264d26f3e-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qlqqk\" (UID: \"3c94d962-9ac1-4c2c-a6bb-8bb264d26f3e\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qlqqk"
Nov 02 03:06:29.296901 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:29.296879    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c94d962-9ac1-4c2c-a6bb-8bb264d26f3e-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qlqqk\" (UID: \"3c94d962-9ac1-4c2c-a6bb-8bb264d26f3e\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qlqqk"
Nov 02 03:06:29.351892 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:29.351846    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cq9wj\" (UniqueName: \"kubernetes.io/projected/3c94d962-9ac1-4c2c-a6bb-8bb264d26f3e-kube-api-access-cq9wj\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qlqqk\" (UID: \"3c94d962-9ac1-4c2c-a6bb-8bb264d26f3e\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qlqqk"
Nov 02 03:06:29.490773 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:29.490738    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qlqqk"
Nov 02 03:06:29.878342 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:29.878310    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-qlqqk"]
Nov 02 03:06:29.878836 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:06:29.878812    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c94d962_9ac1_4c2c_a6bb_8bb264d26f3e.slice/crio-14222b676029bf07a4cb7985590d8fe2a17d5b4fb876cdf81f6c079b5876abba WatchSource:0}: Error finding container 14222b676029bf07a4cb7985590d8fe2a17d5b4fb876cdf81f6c079b5876abba: Status 404 returned error can't find the container with id 14222b676029bf07a4cb7985590d8fe2a17d5b4fb876cdf81f6c079b5876abba
Nov 02 03:06:30.394831 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:30.394785    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qlqqk" event={"ID":"3c94d962-9ac1-4c2c-a6bb-8bb264d26f3e","Type":"ContainerStarted","Data":"14222b676029bf07a4cb7985590d8fe2a17d5b4fb876cdf81f6c079b5876abba"}
Nov 02 03:06:31.401267 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:31.401225    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qlqqk" event={"ID":"3c94d962-9ac1-4c2c-a6bb-8bb264d26f3e","Type":"ContainerStarted","Data":"17b1a3aea1ee3a8f7b2b3af01f19d094994202e83cd4a981b3bd3630a437d48f"}
Nov 02 03:06:31.491198 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:31.491160    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qlqqk"
Nov 02 03:06:31.517257 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:31.517212    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qlqqk" podStartSLOduration=1.9306329070000001 podStartE2EDuration="2.517199369s" totalImagesPullingTime="586.566462ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:06:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:06:31.516680121 +0000 UTC m=+1132.095287503" watchObservedRunningTime="2026-11-02 03:06:31.517199369 +0000 UTC m=+1132.095806748"
Nov 02 03:06:32.335997 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:32.335961    4496 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bm2wq"
Nov 02 03:06:32.348703 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:32.348677    4496 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bm2wq"
Nov 02 03:06:32.417319 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:32.417293    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bm2wq"
Nov 02 03:06:33.525384 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:06:33.525349    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 03:06:35.578162 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:35.578126    4496 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vnxgf"
Nov 02 03:06:35.592843 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:35.592815    4496 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vnxgf"
Nov 02 03:06:36.489040 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:36.484512    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vnxgf"
Nov 02 03:06:39.491156 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:39.491122    4496 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qlqqk"
Nov 02 03:06:39.503556 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:39.503535    4496 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qlqqk"
Nov 02 03:06:40.488000 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:40.487967    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qlqqk"
Nov 02 03:06:43.160991 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:43.160939    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xlbmw"]
Nov 02 03:06:43.178855 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:43.178827    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xlbmw"
Nov 02 03:06:43.216643 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:43.216574    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xlbmw"]
Nov 02 03:06:43.261841 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:43.261805    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2baeda96-f36e-49cb-bb8a-3389c9e8d61d-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-xlbmw\" (UID: \"2baeda96-f36e-49cb-bb8a-3389c9e8d61d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xlbmw"
Nov 02 03:06:43.262026 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:43.261852    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2baeda96-f36e-49cb-bb8a-3389c9e8d61d-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-xlbmw\" (UID: \"2baeda96-f36e-49cb-bb8a-3389c9e8d61d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xlbmw"
Nov 02 03:06:43.262026 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:43.261874    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2baeda96-f36e-49cb-bb8a-3389c9e8d61d-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-xlbmw\" (UID: \"2baeda96-f36e-49cb-bb8a-3389c9e8d61d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xlbmw"
Nov 02 03:06:43.262114 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:43.262090    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2baeda96-f36e-49cb-bb8a-3389c9e8d61d-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-xlbmw\" (UID: \"2baeda96-f36e-49cb-bb8a-3389c9e8d61d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xlbmw"
Nov 02 03:06:43.262214 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:43.262195    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8m5g\" (UniqueName: \"kubernetes.io/projected/2baeda96-f36e-49cb-bb8a-3389c9e8d61d-kube-api-access-m8m5g\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-xlbmw\" (UID: \"2baeda96-f36e-49cb-bb8a-3389c9e8d61d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xlbmw"
Nov 02 03:06:43.262275 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:43.262259    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2baeda96-f36e-49cb-bb8a-3389c9e8d61d-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-xlbmw\" (UID: \"2baeda96-f36e-49cb-bb8a-3389c9e8d61d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xlbmw"
Nov 02 03:06:43.263371 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:43.263350    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Nov 02 03:06:43.363310 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:43.363270    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2baeda96-f36e-49cb-bb8a-3389c9e8d61d-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-xlbmw\" (UID: \"2baeda96-f36e-49cb-bb8a-3389c9e8d61d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xlbmw"
Nov 02 03:06:43.363517 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:43.363357    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8m5g\" (UniqueName: \"kubernetes.io/projected/2baeda96-f36e-49cb-bb8a-3389c9e8d61d-kube-api-access-m8m5g\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-xlbmw\" (UID: \"2baeda96-f36e-49cb-bb8a-3389c9e8d61d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xlbmw"
Nov 02 03:06:43.363517 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:43.363396    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2baeda96-f36e-49cb-bb8a-3389c9e8d61d-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-xlbmw\" (UID: \"2baeda96-f36e-49cb-bb8a-3389c9e8d61d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xlbmw"
Nov 02 03:06:43.363517 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:43.363446    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2baeda96-f36e-49cb-bb8a-3389c9e8d61d-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-xlbmw\" (UID: \"2baeda96-f36e-49cb-bb8a-3389c9e8d61d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xlbmw"
Nov 02 03:06:43.363517 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:43.363490    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2baeda96-f36e-49cb-bb8a-3389c9e8d61d-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-xlbmw\" (UID: \"2baeda96-f36e-49cb-bb8a-3389c9e8d61d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xlbmw"
Nov 02 03:06:43.363739 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:43.363519    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2baeda96-f36e-49cb-bb8a-3389c9e8d61d-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-xlbmw\" (UID: \"2baeda96-f36e-49cb-bb8a-3389c9e8d61d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xlbmw"
Nov 02 03:06:43.363802 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:43.363756    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2baeda96-f36e-49cb-bb8a-3389c9e8d61d-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-xlbmw\" (UID: \"2baeda96-f36e-49cb-bb8a-3389c9e8d61d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xlbmw"
Nov 02 03:06:43.363979 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:43.363957    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2baeda96-f36e-49cb-bb8a-3389c9e8d61d-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-xlbmw\" (UID: \"2baeda96-f36e-49cb-bb8a-3389c9e8d61d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xlbmw"
Nov 02 03:06:43.364084 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:43.363973    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2baeda96-f36e-49cb-bb8a-3389c9e8d61d-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-xlbmw\" (UID: \"2baeda96-f36e-49cb-bb8a-3389c9e8d61d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xlbmw"
Nov 02 03:06:43.365988 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:43.365961    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2baeda96-f36e-49cb-bb8a-3389c9e8d61d-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-xlbmw\" (UID: \"2baeda96-f36e-49cb-bb8a-3389c9e8d61d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xlbmw"
Nov 02 03:06:43.366589 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:43.366382    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2baeda96-f36e-49cb-bb8a-3389c9e8d61d-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-xlbmw\" (UID: \"2baeda96-f36e-49cb-bb8a-3389c9e8d61d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xlbmw"
Nov 02 03:06:43.422653 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:43.422577    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8m5g\" (UniqueName: \"kubernetes.io/projected/2baeda96-f36e-49cb-bb8a-3389c9e8d61d-kube-api-access-m8m5g\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-xlbmw\" (UID: \"2baeda96-f36e-49cb-bb8a-3389c9e8d61d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xlbmw"
Nov 02 03:06:43.507245 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:43.507205    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xlbmw"
Nov 02 03:06:43.893010 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:43.892977    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xlbmw"]
Nov 02 03:06:43.895669 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:06:43.895641    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2baeda96_f36e_49cb_bb8a_3389c9e8d61d.slice/crio-08bc5eac385d43e15e44efb719b6073ce9439eac53ce4827273740846ff73cb5 WatchSource:0}: Error finding container 08bc5eac385d43e15e44efb719b6073ce9439eac53ce4827273740846ff73cb5: Status 404 returned error can't find the container with id 08bc5eac385d43e15e44efb719b6073ce9439eac53ce4827273740846ff73cb5
Nov 02 03:06:44.508398 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:44.508361    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xlbmw" event={"ID":"2baeda96-f36e-49cb-bb8a-3389c9e8d61d","Type":"ContainerStarted","Data":"08bc5eac385d43e15e44efb719b6073ce9439eac53ce4827273740846ff73cb5"}
Nov 02 03:06:45.515909 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:45.515864    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xlbmw" event={"ID":"2baeda96-f36e-49cb-bb8a-3389c9e8d61d","Type":"ContainerStarted","Data":"ace2503be2dcd20b6eb31f32f062b4f86fbad61bbdf414e0838051d1381a59b3"}
Nov 02 03:06:45.641172 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:45.641122    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xlbmw" podStartSLOduration=2.054862438 podStartE2EDuration="2.641109536s" totalImagesPullingTime="586.247098ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:06:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:06:45.639005428 +0000 UTC m=+1146.217612807" watchObservedRunningTime="2026-11-02 03:06:45.641109536 +0000 UTC m=+1146.219716915"
Nov 02 03:06:46.507752 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:46.507719    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xlbmw"
Nov 02 03:06:52.297901 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:52.297859    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-kv6dd-hgn6k"]
Nov 02 03:06:52.302382 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:52.302366    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-kv6dd-hgn6k"
Nov 02 03:06:52.358165 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:52.358136    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-kv6dd-hgn6k"]
Nov 02 03:06:52.404930 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:52.404899    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 02 03:06:52.406280 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:52.406257    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-dgnd5\""
Nov 02 03:06:52.453482 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:52.453455    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d8d9948-fdb9-40bb-830f-aa6fb34e9451-service-ca-bundle\") pod \"periodic-gathering-kv6dd-hgn6k\" (UID: \"9d8d9948-fdb9-40bb-830f-aa6fb34e9451\") " pod="openshift-insights/periodic-gathering-kv6dd-hgn6k"
Nov 02 03:06:52.453612 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:52.453495    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9d8d9948-fdb9-40bb-830f-aa6fb34e9451-archives-path\") pod \"periodic-gathering-kv6dd-hgn6k\" (UID: \"9d8d9948-fdb9-40bb-830f-aa6fb34e9451\") " pod="openshift-insights/periodic-gathering-kv6dd-hgn6k"
Nov 02 03:06:52.453612 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:52.453528    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvxpw\" (UniqueName: \"kubernetes.io/projected/9d8d9948-fdb9-40bb-830f-aa6fb34e9451-kube-api-access-bvxpw\") pod \"periodic-gathering-kv6dd-hgn6k\" (UID: \"9d8d9948-fdb9-40bb-830f-aa6fb34e9451\") " pod="openshift-insights/periodic-gathering-kv6dd-hgn6k"
Nov 02 03:06:52.453714 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:52.453647    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d8d9948-fdb9-40bb-830f-aa6fb34e9451-trusted-ca-bundle\") pod \"periodic-gathering-kv6dd-hgn6k\" (UID: \"9d8d9948-fdb9-40bb-830f-aa6fb34e9451\") " pod="openshift-insights/periodic-gathering-kv6dd-hgn6k"
Nov 02 03:06:52.509513 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:52.509481    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 02 03:06:52.554565 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:52.554486    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d8d9948-fdb9-40bb-830f-aa6fb34e9451-service-ca-bundle\") pod \"periodic-gathering-kv6dd-hgn6k\" (UID: \"9d8d9948-fdb9-40bb-830f-aa6fb34e9451\") " pod="openshift-insights/periodic-gathering-kv6dd-hgn6k"
Nov 02 03:06:52.554565 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:52.554515    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9d8d9948-fdb9-40bb-830f-aa6fb34e9451-archives-path\") pod \"periodic-gathering-kv6dd-hgn6k\" (UID: \"9d8d9948-fdb9-40bb-830f-aa6fb34e9451\") " pod="openshift-insights/periodic-gathering-kv6dd-hgn6k"
Nov 02 03:06:52.554565 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:52.554537    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvxpw\" (UniqueName: \"kubernetes.io/projected/9d8d9948-fdb9-40bb-830f-aa6fb34e9451-kube-api-access-bvxpw\") pod \"periodic-gathering-kv6dd-hgn6k\" (UID: \"9d8d9948-fdb9-40bb-830f-aa6fb34e9451\") " pod="openshift-insights/periodic-gathering-kv6dd-hgn6k"
Nov 02 03:06:52.554827 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:52.554571    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d8d9948-fdb9-40bb-830f-aa6fb34e9451-trusted-ca-bundle\") pod \"periodic-gathering-kv6dd-hgn6k\" (UID: \"9d8d9948-fdb9-40bb-830f-aa6fb34e9451\") " pod="openshift-insights/periodic-gathering-kv6dd-hgn6k"
Nov 02 03:06:52.554981 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:52.554961    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9d8d9948-fdb9-40bb-830f-aa6fb34e9451-archives-path\") pod \"periodic-gathering-kv6dd-hgn6k\" (UID: \"9d8d9948-fdb9-40bb-830f-aa6fb34e9451\") " pod="openshift-insights/periodic-gathering-kv6dd-hgn6k"
Nov 02 03:06:52.555296 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:52.555271    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d8d9948-fdb9-40bb-830f-aa6fb34e9451-service-ca-bundle\") pod \"periodic-gathering-kv6dd-hgn6k\" (UID: \"9d8d9948-fdb9-40bb-830f-aa6fb34e9451\") " pod="openshift-insights/periodic-gathering-kv6dd-hgn6k"
Nov 02 03:06:52.555376 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:52.555344    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d8d9948-fdb9-40bb-830f-aa6fb34e9451-trusted-ca-bundle\") pod \"periodic-gathering-kv6dd-hgn6k\" (UID: \"9d8d9948-fdb9-40bb-830f-aa6fb34e9451\") " pod="openshift-insights/periodic-gathering-kv6dd-hgn6k"
Nov 02 03:06:52.612436 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:52.612411    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvxpw\" (UniqueName: \"kubernetes.io/projected/9d8d9948-fdb9-40bb-830f-aa6fb34e9451-kube-api-access-bvxpw\") pod \"periodic-gathering-kv6dd-hgn6k\" (UID: \"9d8d9948-fdb9-40bb-830f-aa6fb34e9451\") " pod="openshift-insights/periodic-gathering-kv6dd-hgn6k"
Nov 02 03:06:52.614148 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:52.614128    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-kv6dd-hgn6k"
Nov 02 03:06:52.984037 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:52.983997    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-kv6dd-hgn6k"]
Nov 02 03:06:52.985580 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:06:52.985544    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d8d9948_fdb9_40bb_830f_aa6fb34e9451.slice/crio-6404f125b563287dec37d99a68bf39946b6e9027b15c97a9ccc8eab6bf51ba6d WatchSource:0}: Error finding container 6404f125b563287dec37d99a68bf39946b6e9027b15c97a9ccc8eab6bf51ba6d: Status 404 returned error can't find the container with id 6404f125b563287dec37d99a68bf39946b6e9027b15c97a9ccc8eab6bf51ba6d
Nov 02 03:06:53.508343 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:53.508304    4496 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xlbmw"
Nov 02 03:06:53.520483 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:53.520457    4496 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xlbmw"
Nov 02 03:06:53.572804 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:53.572772    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-kv6dd-hgn6k" event={"ID":"9d8d9948-fdb9-40bb-830f-aa6fb34e9451","Type":"ContainerStarted","Data":"9be1cb2167112fdd91ceec1512a9cccb3d25e5353f8c850e55724f8c4b398c60"}
Nov 02 03:06:53.572991 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:53.572819    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-kv6dd-hgn6k" event={"ID":"9d8d9948-fdb9-40bb-830f-aa6fb34e9451","Type":"ContainerStarted","Data":"6404f125b563287dec37d99a68bf39946b6e9027b15c97a9ccc8eab6bf51ba6d"}
Nov 02 03:06:53.582267 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:53.582241    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xlbmw"
Nov 02 03:06:53.866931 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:06:53.866826    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-kv6dd-hgn6k" podStartSLOduration=1.866812253 podStartE2EDuration="1.866812253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:06:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:06:53.864340062 +0000 UTC m=+1154.442947444" watchObservedRunningTime="2026-11-02 03:06:53.866812253 +0000 UTC m=+1154.445419633"
Nov 02 03:07:02.799194 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:02.799154    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mfxjl"]
Nov 02 03:07:02.803009 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:02.802985    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mfxjl"
Nov 02 03:07:02.843634 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:02.843602    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2532dec7-68f1-4cf4-8cdb-a7c34c3b6438-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-mfxjl\" (UID: \"2532dec7-68f1-4cf4-8cdb-a7c34c3b6438\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mfxjl"
Nov 02 03:07:02.843838 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:02.843707    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2532dec7-68f1-4cf4-8cdb-a7c34c3b6438-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-mfxjl\" (UID: \"2532dec7-68f1-4cf4-8cdb-a7c34c3b6438\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mfxjl"
Nov 02 03:07:02.843838 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:02.843745    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2532dec7-68f1-4cf4-8cdb-a7c34c3b6438-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-mfxjl\" (UID: \"2532dec7-68f1-4cf4-8cdb-a7c34c3b6438\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mfxjl"
Nov 02 03:07:02.843838 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:02.843772    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2532dec7-68f1-4cf4-8cdb-a7c34c3b6438-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-mfxjl\" (UID: \"2532dec7-68f1-4cf4-8cdb-a7c34c3b6438\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mfxjl"
Nov 02 03:07:02.843947 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:02.843888    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kv4d\" (UniqueName: \"kubernetes.io/projected/2532dec7-68f1-4cf4-8cdb-a7c34c3b6438-kube-api-access-2kv4d\") pod \"maas-default-gateway-openshift-default-58847f9fc5-mfxjl\" (UID: \"2532dec7-68f1-4cf4-8cdb-a7c34c3b6438\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mfxjl"
Nov 02 03:07:02.843947 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:02.843934    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2532dec7-68f1-4cf4-8cdb-a7c34c3b6438-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-mfxjl\" (UID: \"2532dec7-68f1-4cf4-8cdb-a7c34c3b6438\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mfxjl"
Nov 02 03:07:02.844027 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:02.843999    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2532dec7-68f1-4cf4-8cdb-a7c34c3b6438-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-mfxjl\" (UID: \"2532dec7-68f1-4cf4-8cdb-a7c34c3b6438\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mfxjl"
Nov 02 03:07:02.844070 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:02.844057    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2532dec7-68f1-4cf4-8cdb-a7c34c3b6438-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-mfxjl\" (UID: \"2532dec7-68f1-4cf4-8cdb-a7c34c3b6438\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mfxjl"
Nov 02 03:07:02.844155 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:02.844134    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2532dec7-68f1-4cf4-8cdb-a7c34c3b6438-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-mfxjl\" (UID: \"2532dec7-68f1-4cf4-8cdb-a7c34c3b6438\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mfxjl"
Nov 02 03:07:02.908957 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:02.908922    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mfxjl"]
Nov 02 03:07:02.944751 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:02.944718    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2532dec7-68f1-4cf4-8cdb-a7c34c3b6438-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-mfxjl\" (UID: \"2532dec7-68f1-4cf4-8cdb-a7c34c3b6438\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mfxjl"
Nov 02 03:07:02.944922 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:02.944787    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2532dec7-68f1-4cf4-8cdb-a7c34c3b6438-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-mfxjl\" (UID: \"2532dec7-68f1-4cf4-8cdb-a7c34c3b6438\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mfxjl"
Nov 02 03:07:02.944922 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:02.944840    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2532dec7-68f1-4cf4-8cdb-a7c34c3b6438-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-mfxjl\" (UID: \"2532dec7-68f1-4cf4-8cdb-a7c34c3b6438\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mfxjl"
Nov 02 03:07:02.944922 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:02.944893    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2532dec7-68f1-4cf4-8cdb-a7c34c3b6438-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-mfxjl\" (UID: \"2532dec7-68f1-4cf4-8cdb-a7c34c3b6438\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mfxjl"
Nov 02 03:07:02.945103 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:02.944948    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2532dec7-68f1-4cf4-8cdb-a7c34c3b6438-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-mfxjl\" (UID: \"2532dec7-68f1-4cf4-8cdb-a7c34c3b6438\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mfxjl"
Nov 02 03:07:02.945103 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:02.944990    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2532dec7-68f1-4cf4-8cdb-a7c34c3b6438-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-mfxjl\" (UID: \"2532dec7-68f1-4cf4-8cdb-a7c34c3b6438\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mfxjl"
Nov 02 03:07:02.945195 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:02.945142    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2532dec7-68f1-4cf4-8cdb-a7c34c3b6438-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-mfxjl\" (UID: \"2532dec7-68f1-4cf4-8cdb-a7c34c3b6438\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mfxjl"
Nov 02 03:07:02.945195 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:02.945186    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2532dec7-68f1-4cf4-8cdb-a7c34c3b6438-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-mfxjl\" (UID: \"2532dec7-68f1-4cf4-8cdb-a7c34c3b6438\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mfxjl"
Nov 02 03:07:02.945304 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:02.945218    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kv4d\" (UniqueName: \"kubernetes.io/projected/2532dec7-68f1-4cf4-8cdb-a7c34c3b6438-kube-api-access-2kv4d\") pod \"maas-default-gateway-openshift-default-58847f9fc5-mfxjl\" (UID: \"2532dec7-68f1-4cf4-8cdb-a7c34c3b6438\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mfxjl"
Nov 02 03:07:02.954685 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:02.946100    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2532dec7-68f1-4cf4-8cdb-a7c34c3b6438-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-mfxjl\" (UID: \"2532dec7-68f1-4cf4-8cdb-a7c34c3b6438\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mfxjl"
Nov 02 03:07:02.954685 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:02.946156    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2532dec7-68f1-4cf4-8cdb-a7c34c3b6438-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-mfxjl\" (UID: \"2532dec7-68f1-4cf4-8cdb-a7c34c3b6438\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mfxjl"
Nov 02 03:07:02.954685 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:02.946278    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2532dec7-68f1-4cf4-8cdb-a7c34c3b6438-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-mfxjl\" (UID: \"2532dec7-68f1-4cf4-8cdb-a7c34c3b6438\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mfxjl"
Nov 02 03:07:02.954685 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:02.946280    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2532dec7-68f1-4cf4-8cdb-a7c34c3b6438-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-mfxjl\" (UID: \"2532dec7-68f1-4cf4-8cdb-a7c34c3b6438\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mfxjl"
Nov 02 03:07:02.954685 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:02.946512    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2532dec7-68f1-4cf4-8cdb-a7c34c3b6438-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-mfxjl\" (UID: \"2532dec7-68f1-4cf4-8cdb-a7c34c3b6438\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mfxjl"
Nov 02 03:07:02.954685 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:02.947892    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2532dec7-68f1-4cf4-8cdb-a7c34c3b6438-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-mfxjl\" (UID: \"2532dec7-68f1-4cf4-8cdb-a7c34c3b6438\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mfxjl"
Nov 02 03:07:02.956524 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:02.956495    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2532dec7-68f1-4cf4-8cdb-a7c34c3b6438-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-mfxjl\" (UID: \"2532dec7-68f1-4cf4-8cdb-a7c34c3b6438\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mfxjl"
Nov 02 03:07:03.002735 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:03.002702    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2532dec7-68f1-4cf4-8cdb-a7c34c3b6438-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-mfxjl\" (UID: \"2532dec7-68f1-4cf4-8cdb-a7c34c3b6438\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mfxjl"
Nov 02 03:07:03.002869 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:03.002831    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kv4d\" (UniqueName: \"kubernetes.io/projected/2532dec7-68f1-4cf4-8cdb-a7c34c3b6438-kube-api-access-2kv4d\") pod \"maas-default-gateway-openshift-default-58847f9fc5-mfxjl\" (UID: \"2532dec7-68f1-4cf4-8cdb-a7c34c3b6438\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mfxjl"
Nov 02 03:07:03.117284 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:03.117202    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mfxjl"
Nov 02 03:07:03.526072 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:07:03.526010    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 03:07:03.592611 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:03.592576    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mfxjl"]
Nov 02 03:07:03.632463 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:07:03.632431    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2532dec7_68f1_4cf4_8cdb_a7c34c3b6438.slice/crio-e2d2d2738a597a31d9ddf35119df19f6cfe0b9cfd983ac40980cdd22b05d818b WatchSource:0}: Error finding container e2d2d2738a597a31d9ddf35119df19f6cfe0b9cfd983ac40980cdd22b05d818b: Status 404 returned error can't find the container with id e2d2d2738a597a31d9ddf35119df19f6cfe0b9cfd983ac40980cdd22b05d818b
Nov 02 03:07:03.634425 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:03.634392    4496 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 03:07:03.634519 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:03.634474    4496 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 03:07:03.634519 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:03.634506    4496 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 03:07:04.632209 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:04.632174    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mfxjl" event={"ID":"2532dec7-68f1-4cf4-8cdb-a7c34c3b6438","Type":"ContainerStarted","Data":"684757fb4233cd1f6a1d5d82185f03bbe9657835e0ab3e5781658c8cf99056cf"}
Nov 02 03:07:04.632641 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:04.632227    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mfxjl" event={"ID":"2532dec7-68f1-4cf4-8cdb-a7c34c3b6438","Type":"ContainerStarted","Data":"e2d2d2738a597a31d9ddf35119df19f6cfe0b9cfd983ac40980cdd22b05d818b"}
Nov 02 03:07:04.851047 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:04.847921    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mfxjl" podStartSLOduration=2.847898358 podStartE2EDuration="2.847898358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:07:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:07:04.802849707 +0000 UTC m=+1165.381457091" watchObservedRunningTime="2026-11-02 03:07:04.847898358 +0000 UTC m=+1165.426505742"
Nov 02 03:07:05.117740 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:05.117708    4496 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mfxjl"
Nov 02 03:07:05.118923 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:05.118896    4496 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-mfxjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Nov 02 03:07:05.119067 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:05.118956    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mfxjl" podUID="2532dec7-68f1-4cf4-8cdb-a7c34c3b6438" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Nov 02 03:07:06.117805 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:06.117769    4496 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-mfxjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Nov 02 03:07:06.118236 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:06.117831    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mfxjl" podUID="2532dec7-68f1-4cf4-8cdb-a7c34c3b6438" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Nov 02 03:07:07.117814 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:07.117777    4496 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-mfxjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Nov 02 03:07:07.118203 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:07.117836    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mfxjl" podUID="2532dec7-68f1-4cf4-8cdb-a7c34c3b6438" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Nov 02 03:07:08.118065 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:08.118002    4496 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-mfxjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Nov 02 03:07:08.118419 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:08.118086    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mfxjl" podUID="2532dec7-68f1-4cf4-8cdb-a7c34c3b6438" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Nov 02 03:07:09.117787 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:09.117745    4496 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-mfxjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Nov 02 03:07:09.118000 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:09.117805    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mfxjl" podUID="2532dec7-68f1-4cf4-8cdb-a7c34c3b6438" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Nov 02 03:07:10.117539 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:10.117498    4496 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-mfxjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Nov 02 03:07:10.117911 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:10.117560    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mfxjl" podUID="2532dec7-68f1-4cf4-8cdb-a7c34c3b6438" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Nov 02 03:07:11.869203 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:11.869171    4496 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mfxjl"
Nov 02 03:07:11.869785 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:11.869552    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mfxjl"
Nov 02 03:07:11.870467 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:11.870448    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mfxjl"
Nov 02 03:07:12.042607 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:12.042567    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5ccd9d7cb9-hg9ft"]
Nov 02 03:07:12.046298 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:12.046281    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5ccd9d7cb9-hg9ft"
Nov 02 03:07:12.099028 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:12.098994    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5ccd9d7cb9-hg9ft"]
Nov 02 03:07:12.140427 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:12.140349    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fbfd5bbe-5d78-41b3-90b2-9e0565607d40-tls-cert\") pod \"authorino-5ccd9d7cb9-hg9ft\" (UID: \"fbfd5bbe-5d78-41b3-90b2-9e0565607d40\") " pod="kuadrant-system/authorino-5ccd9d7cb9-hg9ft"
Nov 02 03:07:12.140427 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:12.140395    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxqpp\" (UniqueName: \"kubernetes.io/projected/fbfd5bbe-5d78-41b3-90b2-9e0565607d40-kube-api-access-kxqpp\") pod \"authorino-5ccd9d7cb9-hg9ft\" (UID: \"fbfd5bbe-5d78-41b3-90b2-9e0565607d40\") " pod="kuadrant-system/authorino-5ccd9d7cb9-hg9ft"
Nov 02 03:07:12.241377 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:12.241346    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fbfd5bbe-5d78-41b3-90b2-9e0565607d40-tls-cert\") pod \"authorino-5ccd9d7cb9-hg9ft\" (UID: \"fbfd5bbe-5d78-41b3-90b2-9e0565607d40\") " pod="kuadrant-system/authorino-5ccd9d7cb9-hg9ft"
Nov 02 03:07:12.241532 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:12.241387    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxqpp\" (UniqueName: \"kubernetes.io/projected/fbfd5bbe-5d78-41b3-90b2-9e0565607d40-kube-api-access-kxqpp\") pod \"authorino-5ccd9d7cb9-hg9ft\" (UID: \"fbfd5bbe-5d78-41b3-90b2-9e0565607d40\") " pod="kuadrant-system/authorino-5ccd9d7cb9-hg9ft"
Nov 02 03:07:12.243659 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:12.243630    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fbfd5bbe-5d78-41b3-90b2-9e0565607d40-tls-cert\") pod \"authorino-5ccd9d7cb9-hg9ft\" (UID: \"fbfd5bbe-5d78-41b3-90b2-9e0565607d40\") " pod="kuadrant-system/authorino-5ccd9d7cb9-hg9ft"
Nov 02 03:07:12.298772 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:12.298742    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxqpp\" (UniqueName: \"kubernetes.io/projected/fbfd5bbe-5d78-41b3-90b2-9e0565607d40-kube-api-access-kxqpp\") pod \"authorino-5ccd9d7cb9-hg9ft\" (UID: \"fbfd5bbe-5d78-41b3-90b2-9e0565607d40\") " pod="kuadrant-system/authorino-5ccd9d7cb9-hg9ft"
Nov 02 03:07:12.358036 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:12.357994    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5ccd9d7cb9-hg9ft"
Nov 02 03:07:12.731051 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:12.731011    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5ccd9d7cb9-hg9ft"]
Nov 02 03:07:12.731614 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:07:12.731591    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfbfd5bbe_5d78_41b3_90b2_9e0565607d40.slice/crio-5fd49e877d1fa34ef2073ac3a03cb81c97b5c2e2e674720b98379840ba77564f WatchSource:0}: Error finding container 5fd49e877d1fa34ef2073ac3a03cb81c97b5c2e2e674720b98379840ba77564f: Status 404 returned error can't find the container with id 5fd49e877d1fa34ef2073ac3a03cb81c97b5c2e2e674720b98379840ba77564f
Nov 02 03:07:13.708334 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:13.708296    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5ccd9d7cb9-hg9ft" event={"ID":"fbfd5bbe-5d78-41b3-90b2-9e0565607d40","Type":"ContainerStarted","Data":"5fd49e877d1fa34ef2073ac3a03cb81c97b5c2e2e674720b98379840ba77564f"}
Nov 02 03:07:16.728638 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:16.728603    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5ccd9d7cb9-hg9ft" event={"ID":"fbfd5bbe-5d78-41b3-90b2-9e0565607d40","Type":"ContainerStarted","Data":"0a878b939778db3aea96e2f73228a0860467214ee6b90437e0fd63201785a522"}
Nov 02 03:07:16.844131 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:16.844077    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-5ccd9d7cb9-hg9ft" podStartSLOduration=2.533276923 podStartE2EDuration="5.844063487s" totalImagesPullingTime="3.310786564s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:07:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:07:16.842289227 +0000 UTC m=+1177.420896606" watchObservedRunningTime="2026-11-02 03:07:16.844063487 +0000 UTC m=+1177.422670867"
Nov 02 03:07:16.896932 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:16.896883    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-78c9c77858-nq6sx"]
Nov 02 03:07:16.897153 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:16.897111    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-78c9c77858-nq6sx" podUID="72d5b418-688f-4a96-8129-4c2f81a06638" containerName="authorino" containerID="cri-o://8d2970834ddcea14a12791eb93010d75ab00586b56d2aaad451f993690e74170" gracePeriod=30
Nov 02 03:07:17.170930 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:17.170882    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-78c9c77858-nq6sx"
Nov 02 03:07:17.284637 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:17.284555    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72d5b418-688f-4a96-8129-4c2f81a06638-kube-api-access-2g2gs\" (UniqueName: \"kubernetes.io/projected/72d5b418-688f-4a96-8129-4c2f81a06638-kube-api-access-2g2gs\") pod \"72d5b418-688f-4a96-8129-4c2f81a06638\" (UID: \"72d5b418-688f-4a96-8129-4c2f81a06638\") "
Nov 02 03:07:17.284788 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:17.284639    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72d5b418-688f-4a96-8129-4c2f81a06638-tls-cert\" (UniqueName: \"kubernetes.io/secret/72d5b418-688f-4a96-8129-4c2f81a06638-tls-cert\") pod \"72d5b418-688f-4a96-8129-4c2f81a06638\" (UID: \"72d5b418-688f-4a96-8129-4c2f81a06638\") "
Nov 02 03:07:17.286935 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:17.286905    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72d5b418-688f-4a96-8129-4c2f81a06638-kube-api-access-2g2gs" pod "72d5b418-688f-4a96-8129-4c2f81a06638" (UID: "72d5b418-688f-4a96-8129-4c2f81a06638"). InnerVolumeSpecName "kube-api-access-2g2gs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:07:17.302508 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:17.302439    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72d5b418-688f-4a96-8129-4c2f81a06638-tls-cert" pod "72d5b418-688f-4a96-8129-4c2f81a06638" (UID: "72d5b418-688f-4a96-8129-4c2f81a06638"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 03:07:17.389730 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:17.389657    4496 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/72d5b418-688f-4a96-8129-4c2f81a06638-tls-cert\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:07:17.389730 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:17.389699    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2g2gs\" (UniqueName: \"kubernetes.io/projected/72d5b418-688f-4a96-8129-4c2f81a06638-kube-api-access-2g2gs\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:07:17.741245 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:17.741206    4496 generic.go:356] "Generic (PLEG): container finished" podID="72d5b418-688f-4a96-8129-4c2f81a06638" containerID="8d2970834ddcea14a12791eb93010d75ab00586b56d2aaad451f993690e74170" exitCode=0
Nov 02 03:07:17.741724 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:17.741253    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-78c9c77858-nq6sx"
Nov 02 03:07:17.741724 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:17.741299    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-78c9c77858-nq6sx" event={"ID":"72d5b418-688f-4a96-8129-4c2f81a06638","Type":"ContainerDied","Data":"8d2970834ddcea14a12791eb93010d75ab00586b56d2aaad451f993690e74170"}
Nov 02 03:07:17.741724 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:17.741348    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-78c9c77858-nq6sx" event={"ID":"72d5b418-688f-4a96-8129-4c2f81a06638","Type":"ContainerDied","Data":"d4b830b88009e93d1698f46ddf9b2e2175bb9479e0b259fb43c25c11c2113174"}
Nov 02 03:07:17.741724 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:17.741372    4496 scope.go:122] "RemoveContainer" containerID="8d2970834ddcea14a12791eb93010d75ab00586b56d2aaad451f993690e74170"
Nov 02 03:07:17.751993 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:17.751974    4496 scope.go:122] "RemoveContainer" containerID="8d2970834ddcea14a12791eb93010d75ab00586b56d2aaad451f993690e74170"
Nov 02 03:07:17.752340 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:07:17.752320    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d2970834ddcea14a12791eb93010d75ab00586b56d2aaad451f993690e74170\": container with ID starting with 8d2970834ddcea14a12791eb93010d75ab00586b56d2aaad451f993690e74170 not found: ID does not exist" containerID="8d2970834ddcea14a12791eb93010d75ab00586b56d2aaad451f993690e74170"
Nov 02 03:07:17.752440 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:17.752347    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d2970834ddcea14a12791eb93010d75ab00586b56d2aaad451f993690e74170"} err="failed to get container status \"8d2970834ddcea14a12791eb93010d75ab00586b56d2aaad451f993690e74170\": rpc error: code = NotFound desc = could not find container \"8d2970834ddcea14a12791eb93010d75ab00586b56d2aaad451f993690e74170\": container with ID starting with 8d2970834ddcea14a12791eb93010d75ab00586b56d2aaad451f993690e74170 not found: ID does not exist"
Nov 02 03:07:17.913867 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:17.913785    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-78c9c77858-nq6sx"]
Nov 02 03:07:17.917542 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:17.917514    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-78c9c77858-nq6sx"]
Nov 02 03:07:19.573592 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:19.573560    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72d5b418-688f-4a96-8129-4c2f81a06638" path="/var/lib/kubelet/pods/72d5b418-688f-4a96-8129-4c2f81a06638/volumes"
Nov 02 03:07:33.525713 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:07:33.525675    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 03:07:39.710301 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:39.710267    4496 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hww8q_5c797b08-a314-44ed-8026-8a0ce1f6c36f/ovn-acl-logging/0.log"
Nov 02 03:07:39.715302 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:39.715279    4496 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hww8q_5c797b08-a314-44ed-8026-8a0ce1f6c36f/ovn-acl-logging/0.log"
Nov 02 03:07:51.995692 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:51.995648    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-935e4abf-57498b49f5-qkrxc"]
Nov 02 03:07:51.996155 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:51.996100    4496 cpu_manager.go:414] "containerMap: removing container" podUID="72d5b418-688f-4a96-8129-4c2f81a06638" containerName="authorino"
Nov 02 03:07:51.996155 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:51.996113    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72d5b418-688f-4a96-8129-4c2f81a06638" containerName="authorino"
Nov 02 03:07:51.996233 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:51.996216    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="72d5b418-688f-4a96-8129-4c2f81a06638" containerName="authorino"
Nov 02 03:07:52.000693 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:52.000669    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-935e4abf-57498b49f5-qkrxc"
Nov 02 03:07:52.068040 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:52.067260    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-935e4abf-57498b49f5-qkrxc"]
Nov 02 03:07:52.076584 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:52.076550    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6nqp\" (UniqueName: \"kubernetes.io/projected/7665e3a4-191c-4de6-b99e-3ae4b4e78b9b-kube-api-access-m6nqp\") pod \"payload-pre-processing-e2e-migrate-935e4abf-57498b49f5-qkrxc\" (UID: \"7665e3a4-191c-4de6-b99e-3ae4b4e78b9b\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-935e4abf-57498b49f5-qkrxc"
Nov 02 03:07:52.076777 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:52.076745    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7665e3a4-191c-4de6-b99e-3ae4b4e78b9b-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-935e4abf-57498b49f5-qkrxc\" (UID: \"7665e3a4-191c-4de6-b99e-3ae4b4e78b9b\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-935e4abf-57498b49f5-qkrxc"
Nov 02 03:07:52.112759 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:52.112730    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-935e4abf\""
Nov 02 03:07:52.112900 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:52.112831    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-935e4abf-dockercfg-w4vmz\""
Nov 02 03:07:52.165856 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:52.165829    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-935e4abf-7d747dd7d7-gs66f"]
Nov 02 03:07:52.169482 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:52.169467    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-935e4abf-7d747dd7d7-gs66f"
Nov 02 03:07:52.177860 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:52.177836    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6nqp\" (UniqueName: \"kubernetes.io/projected/7665e3a4-191c-4de6-b99e-3ae4b4e78b9b-kube-api-access-m6nqp\") pod \"payload-pre-processing-e2e-migrate-935e4abf-57498b49f5-qkrxc\" (UID: \"7665e3a4-191c-4de6-b99e-3ae4b4e78b9b\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-935e4abf-57498b49f5-qkrxc"
Nov 02 03:07:52.177961 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:52.177882    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7665e3a4-191c-4de6-b99e-3ae4b4e78b9b-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-935e4abf-57498b49f5-qkrxc\" (UID: \"7665e3a4-191c-4de6-b99e-3ae4b4e78b9b\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-935e4abf-57498b49f5-qkrxc"
Nov 02 03:07:52.178572 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:52.178555    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7665e3a4-191c-4de6-b99e-3ae4b4e78b9b-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-935e4abf-57498b49f5-qkrxc\" (UID: \"7665e3a4-191c-4de6-b99e-3ae4b4e78b9b\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-935e4abf-57498b49f5-qkrxc"
Nov 02 03:07:52.225511 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:52.225474    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-935e4abf-7d747dd7d7-gs66f"]
Nov 02 03:07:52.237711 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:52.237687    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6nqp\" (UniqueName: \"kubernetes.io/projected/7665e3a4-191c-4de6-b99e-3ae4b4e78b9b-kube-api-access-m6nqp\") pod \"payload-pre-processing-e2e-migrate-935e4abf-57498b49f5-qkrxc\" (UID: \"7665e3a4-191c-4de6-b99e-3ae4b4e78b9b\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-935e4abf-57498b49f5-qkrxc"
Nov 02 03:07:52.278925 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:52.278854    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcc64ce5-28af-4a53-b5f2-6574758ae27c-plugins-config-volume\") pod \"payload-processing-e2e-migrate-935e4abf-7d747dd7d7-gs66f\" (UID: \"fcc64ce5-28af-4a53-b5f2-6574758ae27c\") " pod="openshift-ingress/payload-processing-e2e-migrate-935e4abf-7d747dd7d7-gs66f"
Nov 02 03:07:52.279086 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:52.278966    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqjjt\" (UniqueName: \"kubernetes.io/projected/fcc64ce5-28af-4a53-b5f2-6574758ae27c-kube-api-access-lqjjt\") pod \"payload-processing-e2e-migrate-935e4abf-7d747dd7d7-gs66f\" (UID: \"fcc64ce5-28af-4a53-b5f2-6574758ae27c\") " pod="openshift-ingress/payload-processing-e2e-migrate-935e4abf-7d747dd7d7-gs66f"
Nov 02 03:07:52.313944 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:52.313914    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-935e4abf-57498b49f5-qkrxc"
Nov 02 03:07:52.380814 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:52.380777    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcc64ce5-28af-4a53-b5f2-6574758ae27c-plugins-config-volume\") pod \"payload-processing-e2e-migrate-935e4abf-7d747dd7d7-gs66f\" (UID: \"fcc64ce5-28af-4a53-b5f2-6574758ae27c\") " pod="openshift-ingress/payload-processing-e2e-migrate-935e4abf-7d747dd7d7-gs66f"
Nov 02 03:07:52.380951 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:52.380911    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqjjt\" (UniqueName: \"kubernetes.io/projected/fcc64ce5-28af-4a53-b5f2-6574758ae27c-kube-api-access-lqjjt\") pod \"payload-processing-e2e-migrate-935e4abf-7d747dd7d7-gs66f\" (UID: \"fcc64ce5-28af-4a53-b5f2-6574758ae27c\") " pod="openshift-ingress/payload-processing-e2e-migrate-935e4abf-7d747dd7d7-gs66f"
Nov 02 03:07:52.382004 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:52.381977    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcc64ce5-28af-4a53-b5f2-6574758ae27c-plugins-config-volume\") pod \"payload-processing-e2e-migrate-935e4abf-7d747dd7d7-gs66f\" (UID: \"fcc64ce5-28af-4a53-b5f2-6574758ae27c\") " pod="openshift-ingress/payload-processing-e2e-migrate-935e4abf-7d747dd7d7-gs66f"
Nov 02 03:07:52.438835 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:52.438803    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqjjt\" (UniqueName: \"kubernetes.io/projected/fcc64ce5-28af-4a53-b5f2-6574758ae27c-kube-api-access-lqjjt\") pod \"payload-processing-e2e-migrate-935e4abf-7d747dd7d7-gs66f\" (UID: \"fcc64ce5-28af-4a53-b5f2-6574758ae27c\") " pod="openshift-ingress/payload-processing-e2e-migrate-935e4abf-7d747dd7d7-gs66f"
Nov 02 03:07:52.481955 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:52.481894    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-935e4abf-7d747dd7d7-gs66f"
Nov 02 03:07:52.694803 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:52.694777    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-935e4abf-57498b49f5-qkrxc"]
Nov 02 03:07:52.695567 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:07:52.695539    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7665e3a4_191c_4de6_b99e_3ae4b4e78b9b.slice/crio-244709b48a32359f21ac87f090ec8b94cbac285ae2c65c025cff8fc79172be0e WatchSource:0}: Error finding container 244709b48a32359f21ac87f090ec8b94cbac285ae2c65c025cff8fc79172be0e: Status 404 returned error can't find the container with id 244709b48a32359f21ac87f090ec8b94cbac285ae2c65c025cff8fc79172be0e
Nov 02 03:07:52.858699 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:52.858672    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-935e4abf-7d747dd7d7-gs66f"]
Nov 02 03:07:52.859595 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:07:52.859572    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcc64ce5_28af_4a53_b5f2_6574758ae27c.slice/crio-d628a4e60cd68e3f5ad3e9755644ae9e569a307937ad332d39ae43e4a4d539fd WatchSource:0}: Error finding container d628a4e60cd68e3f5ad3e9755644ae9e569a307937ad332d39ae43e4a4d539fd: Status 404 returned error can't find the container with id d628a4e60cd68e3f5ad3e9755644ae9e569a307937ad332d39ae43e4a4d539fd
Nov 02 03:07:52.943646 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:52.943608    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-935e4abf-57498b49f5-qkrxc" event={"ID":"7665e3a4-191c-4de6-b99e-3ae4b4e78b9b","Type":"ContainerStarted","Data":"4f7d9a854fab477ca8c4b8046e812c013146308e2805acb80511aa5ada5b5da9"}
Nov 02 03:07:52.943790 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:52.943713    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-935e4abf-57498b49f5-qkrxc" event={"ID":"7665e3a4-191c-4de6-b99e-3ae4b4e78b9b","Type":"ContainerStarted","Data":"244709b48a32359f21ac87f090ec8b94cbac285ae2c65c025cff8fc79172be0e"}
Nov 02 03:07:52.943790 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:52.943753    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-935e4abf-57498b49f5-qkrxc"
Nov 02 03:07:52.945412 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:52.945382    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-935e4abf-7d747dd7d7-gs66f" event={"ID":"fcc64ce5-28af-4a53-b5f2-6574758ae27c","Type":"ContainerStarted","Data":"564c84dda559e33a6294188b3ba5f42d5b2c5b2581eb6e689a03c0b506ee7819"}
Nov 02 03:07:52.945505 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:52.945430    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-935e4abf-7d747dd7d7-gs66f" event={"ID":"fcc64ce5-28af-4a53-b5f2-6574758ae27c","Type":"ContainerStarted","Data":"d628a4e60cd68e3f5ad3e9755644ae9e569a307937ad332d39ae43e4a4d539fd"}
Nov 02 03:07:52.945505 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:52.945452    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-935e4abf-7d747dd7d7-gs66f"
Nov 02 03:07:53.061206 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:53.061159    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-935e4abf-57498b49f5-qkrxc" podStartSLOduration=2.061146521 podStartE2EDuration="2.061146521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:07:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:07:53.059683521 +0000 UTC m=+1213.638290903" watchObservedRunningTime="2026-11-02 03:07:53.061146521 +0000 UTC m=+1213.639753920"
Nov 02 03:07:53.179065 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:53.178932    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-935e4abf-7d747dd7d7-gs66f" podStartSLOduration=1.178912778 podStartE2EDuration="1.178912778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:07:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:07:53.175786553 +0000 UTC m=+1213.754393934" watchObservedRunningTime="2026-11-02 03:07:53.178912778 +0000 UTC m=+1213.757520159"
Nov 02 03:07:56.701361 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:56.701325    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-66a47a3b-699d7d7899-pj48x"]
Nov 02 03:07:56.705306 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:56.705291    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-66a47a3b-699d7d7899-pj48x"
Nov 02 03:07:56.759820 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:56.759351    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-66a47a3b\""
Nov 02 03:07:56.809560 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:56.809533    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-66a47a3b-dockercfg-g42zr\""
Nov 02 03:07:56.812264 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:56.812237    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-66a47a3b-699d7d7899-pj48x"]
Nov 02 03:07:56.828348 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:56.828324    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8f885d3-8814-43fc-9e10-2bfc8e614899-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-66a47a3b-699d7d7899-pj48x\" (UID: \"c8f885d3-8814-43fc-9e10-2bfc8e614899\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-66a47a3b-699d7d7899-pj48x"
Nov 02 03:07:56.828491 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:56.828471    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpppk\" (UniqueName: \"kubernetes.io/projected/c8f885d3-8814-43fc-9e10-2bfc8e614899-kube-api-access-zpppk\") pod \"payload-pre-processing-e2e-ait-66a47a3b-699d7d7899-pj48x\" (UID: \"c8f885d3-8814-43fc-9e10-2bfc8e614899\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-66a47a3b-699d7d7899-pj48x"
Nov 02 03:07:56.871878 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:56.864718    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-66a47a3b-686d7444c9-xk4nt"]
Nov 02 03:07:56.874413 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:56.874388    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-66a47a3b-686d7444c9-xk4nt"
Nov 02 03:07:56.919974 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:56.919935    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-66a47a3b-686d7444c9-xk4nt"]
Nov 02 03:07:56.930041 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:56.929998    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpppk\" (UniqueName: \"kubernetes.io/projected/c8f885d3-8814-43fc-9e10-2bfc8e614899-kube-api-access-zpppk\") pod \"payload-pre-processing-e2e-ait-66a47a3b-699d7d7899-pj48x\" (UID: \"c8f885d3-8814-43fc-9e10-2bfc8e614899\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-66a47a3b-699d7d7899-pj48x"
Nov 02 03:07:56.930180 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:56.930116    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8f885d3-8814-43fc-9e10-2bfc8e614899-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-66a47a3b-699d7d7899-pj48x\" (UID: \"c8f885d3-8814-43fc-9e10-2bfc8e614899\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-66a47a3b-699d7d7899-pj48x"
Nov 02 03:07:56.930788 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:56.930764    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8f885d3-8814-43fc-9e10-2bfc8e614899-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-66a47a3b-699d7d7899-pj48x\" (UID: \"c8f885d3-8814-43fc-9e10-2bfc8e614899\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-66a47a3b-699d7d7899-pj48x"
Nov 02 03:07:56.988236 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:56.988202    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpppk\" (UniqueName: \"kubernetes.io/projected/c8f885d3-8814-43fc-9e10-2bfc8e614899-kube-api-access-zpppk\") pod \"payload-pre-processing-e2e-ait-66a47a3b-699d7d7899-pj48x\" (UID: \"c8f885d3-8814-43fc-9e10-2bfc8e614899\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-66a47a3b-699d7d7899-pj48x"
Nov 02 03:07:57.018841 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:57.018813    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-66a47a3b-699d7d7899-pj48x"
Nov 02 03:07:57.031262 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:57.031239    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-279xw\" (UniqueName: \"kubernetes.io/projected/46896864-f510-4a9f-a225-376ebc87cdf5-kube-api-access-279xw\") pod \"payload-processing-e2e-ait-66a47a3b-686d7444c9-xk4nt\" (UID: \"46896864-f510-4a9f-a225-376ebc87cdf5\") " pod="openshift-ingress/payload-processing-e2e-ait-66a47a3b-686d7444c9-xk4nt"
Nov 02 03:07:57.031370 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:57.031299    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46896864-f510-4a9f-a225-376ebc87cdf5-plugins-config-volume\") pod \"payload-processing-e2e-ait-66a47a3b-686d7444c9-xk4nt\" (UID: \"46896864-f510-4a9f-a225-376ebc87cdf5\") " pod="openshift-ingress/payload-processing-e2e-ait-66a47a3b-686d7444c9-xk4nt"
Nov 02 03:07:57.132503 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:57.132469    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-279xw\" (UniqueName: \"kubernetes.io/projected/46896864-f510-4a9f-a225-376ebc87cdf5-kube-api-access-279xw\") pod \"payload-processing-e2e-ait-66a47a3b-686d7444c9-xk4nt\" (UID: \"46896864-f510-4a9f-a225-376ebc87cdf5\") " pod="openshift-ingress/payload-processing-e2e-ait-66a47a3b-686d7444c9-xk4nt"
Nov 02 03:07:57.132665 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:57.132557    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46896864-f510-4a9f-a225-376ebc87cdf5-plugins-config-volume\") pod \"payload-processing-e2e-ait-66a47a3b-686d7444c9-xk4nt\" (UID: \"46896864-f510-4a9f-a225-376ebc87cdf5\") " pod="openshift-ingress/payload-processing-e2e-ait-66a47a3b-686d7444c9-xk4nt"
Nov 02 03:07:57.133292 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:57.133273    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46896864-f510-4a9f-a225-376ebc87cdf5-plugins-config-volume\") pod \"payload-processing-e2e-ait-66a47a3b-686d7444c9-xk4nt\" (UID: \"46896864-f510-4a9f-a225-376ebc87cdf5\") " pod="openshift-ingress/payload-processing-e2e-ait-66a47a3b-686d7444c9-xk4nt"
Nov 02 03:07:57.190783 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:57.190753    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-279xw\" (UniqueName: \"kubernetes.io/projected/46896864-f510-4a9f-a225-376ebc87cdf5-kube-api-access-279xw\") pod \"payload-processing-e2e-ait-66a47a3b-686d7444c9-xk4nt\" (UID: \"46896864-f510-4a9f-a225-376ebc87cdf5\") " pod="openshift-ingress/payload-processing-e2e-ait-66a47a3b-686d7444c9-xk4nt"
Nov 02 03:07:57.389670 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:57.389643    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-66a47a3b-699d7d7899-pj48x"]
Nov 02 03:07:57.391882 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:07:57.391850    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8f885d3_8814_43fc_9e10_2bfc8e614899.slice/crio-d6ad5735699cd3dcdfd0092a660e34e7820f7ba7e04bfadbe063dd1ef60bd5c9 WatchSource:0}: Error finding container d6ad5735699cd3dcdfd0092a660e34e7820f7ba7e04bfadbe063dd1ef60bd5c9: Status 404 returned error can't find the container with id d6ad5735699cd3dcdfd0092a660e34e7820f7ba7e04bfadbe063dd1ef60bd5c9
Nov 02 03:07:57.487225 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:57.487198    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-66a47a3b-686d7444c9-xk4nt"
Nov 02 03:07:57.858504 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:57.858475    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-66a47a3b-686d7444c9-xk4nt"]
Nov 02 03:07:57.859391 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:07:57.859369    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46896864_f510_4a9f_a225_376ebc87cdf5.slice/crio-aef76a44298e1720175cc0f438b5e79e3d920eafeef7a3c7e500ae72c6db76a8 WatchSource:0}: Error finding container aef76a44298e1720175cc0f438b5e79e3d920eafeef7a3c7e500ae72c6db76a8: Status 404 returned error can't find the container with id aef76a44298e1720175cc0f438b5e79e3d920eafeef7a3c7e500ae72c6db76a8
Nov 02 03:07:57.978174 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:57.978133    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-66a47a3b-686d7444c9-xk4nt" event={"ID":"46896864-f510-4a9f-a225-376ebc87cdf5","Type":"ContainerStarted","Data":"94764994f6a4803565cba22fd4303114f94c4294dd28699f23eb7ae9202d3602"}
Nov 02 03:07:57.978352 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:57.978189    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-66a47a3b-686d7444c9-xk4nt" event={"ID":"46896864-f510-4a9f-a225-376ebc87cdf5","Type":"ContainerStarted","Data":"aef76a44298e1720175cc0f438b5e79e3d920eafeef7a3c7e500ae72c6db76a8"}
Nov 02 03:07:57.978422 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:57.978357    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-66a47a3b-686d7444c9-xk4nt"
Nov 02 03:07:57.979795 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:57.979771    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-66a47a3b-699d7d7899-pj48x" event={"ID":"c8f885d3-8814-43fc-9e10-2bfc8e614899","Type":"ContainerStarted","Data":"f0836346ff6499159381cdd2079dd21b082b638da60930a84d85cbafd457948d"}
Nov 02 03:07:57.979898 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:57.979811    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-66a47a3b-699d7d7899-pj48x" event={"ID":"c8f885d3-8814-43fc-9e10-2bfc8e614899","Type":"ContainerStarted","Data":"d6ad5735699cd3dcdfd0092a660e34e7820f7ba7e04bfadbe063dd1ef60bd5c9"}
Nov 02 03:07:57.979898 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:57.979834    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-66a47a3b-699d7d7899-pj48x"
Nov 02 03:07:58.093967 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:58.093918    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-66a47a3b-686d7444c9-xk4nt" podStartSLOduration=2.093903689 podStartE2EDuration="2.093903689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:07:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:07:58.092302275 +0000 UTC m=+1218.670909656" watchObservedRunningTime="2026-11-02 03:07:58.093903689 +0000 UTC m=+1218.672511112"
Nov 02 03:07:58.208649 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:58.208519    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-66a47a3b-699d7d7899-pj48x" podStartSLOduration=2.208504978 podStartE2EDuration="2.208504978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:07:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:07:58.208452912 +0000 UTC m=+1218.787060292" watchObservedRunningTime="2026-11-02 03:07:58.208504978 +0000 UTC m=+1218.787112358"
Nov 02 03:07:59.127473 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:59.127442    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d1107301-675fb7899-nsrpt"]
Nov 02 03:07:59.131393 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:59.131377    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d1107301-675fb7899-nsrpt"
Nov 02 03:07:59.185162 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:59.185128    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-d1107301\""
Nov 02 03:07:59.195190 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:59.195162    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-d1107301-dockercfg-7jpks\""
Nov 02 03:07:59.196605 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:59.196572    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d1107301-675fb7899-nsrpt"]
Nov 02 03:07:59.254484 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:59.254451    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8e4bd54-5ae6-49c0-ad5f-786322530e97-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d1107301-675fb7899-nsrpt\" (UID: \"c8e4bd54-5ae6-49c0-ad5f-786322530e97\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d1107301-675fb7899-nsrpt"
Nov 02 03:07:59.254623 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:59.254489    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwc74\" (UniqueName: \"kubernetes.io/projected/c8e4bd54-5ae6-49c0-ad5f-786322530e97-kube-api-access-hwc74\") pod \"payload-pre-processing-e2e-ait-d1107301-675fb7899-nsrpt\" (UID: \"c8e4bd54-5ae6-49c0-ad5f-786322530e97\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d1107301-675fb7899-nsrpt"
Nov 02 03:07:59.283859 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:59.283831    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d1107301-7b968d8f77-c2tv7"]
Nov 02 03:07:59.287736 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:59.287721    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d1107301-7b968d8f77-c2tv7"
Nov 02 03:07:59.348083 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:59.348047    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d1107301-7b968d8f77-c2tv7"]
Nov 02 03:07:59.356196 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:59.356154    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8e4bd54-5ae6-49c0-ad5f-786322530e97-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d1107301-675fb7899-nsrpt\" (UID: \"c8e4bd54-5ae6-49c0-ad5f-786322530e97\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d1107301-675fb7899-nsrpt"
Nov 02 03:07:59.356369 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:59.356200    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwc74\" (UniqueName: \"kubernetes.io/projected/c8e4bd54-5ae6-49c0-ad5f-786322530e97-kube-api-access-hwc74\") pod \"payload-pre-processing-e2e-ait-d1107301-675fb7899-nsrpt\" (UID: \"c8e4bd54-5ae6-49c0-ad5f-786322530e97\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d1107301-675fb7899-nsrpt"
Nov 02 03:07:59.357203 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:59.357177    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8e4bd54-5ae6-49c0-ad5f-786322530e97-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d1107301-675fb7899-nsrpt\" (UID: \"c8e4bd54-5ae6-49c0-ad5f-786322530e97\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d1107301-675fb7899-nsrpt"
Nov 02 03:07:59.415163 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:59.415092    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwc74\" (UniqueName: \"kubernetes.io/projected/c8e4bd54-5ae6-49c0-ad5f-786322530e97-kube-api-access-hwc74\") pod \"payload-pre-processing-e2e-ait-d1107301-675fb7899-nsrpt\" (UID: \"c8e4bd54-5ae6-49c0-ad5f-786322530e97\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d1107301-675fb7899-nsrpt"
Nov 02 03:07:59.445009 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:59.444983    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d1107301-675fb7899-nsrpt"
Nov 02 03:07:59.456963 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:59.456938    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14f52df5-6ec4-4647-ac27-1014e2669d66-plugins-config-volume\") pod \"payload-processing-e2e-ait-d1107301-7b968d8f77-c2tv7\" (UID: \"14f52df5-6ec4-4647-ac27-1014e2669d66\") " pod="openshift-ingress/payload-processing-e2e-ait-d1107301-7b968d8f77-c2tv7"
Nov 02 03:07:59.457115 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:59.457052    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59ntt\" (UniqueName: \"kubernetes.io/projected/14f52df5-6ec4-4647-ac27-1014e2669d66-kube-api-access-59ntt\") pod \"payload-processing-e2e-ait-d1107301-7b968d8f77-c2tv7\" (UID: \"14f52df5-6ec4-4647-ac27-1014e2669d66\") " pod="openshift-ingress/payload-processing-e2e-ait-d1107301-7b968d8f77-c2tv7"
Nov 02 03:07:59.558690 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:59.558656    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14f52df5-6ec4-4647-ac27-1014e2669d66-plugins-config-volume\") pod \"payload-processing-e2e-ait-d1107301-7b968d8f77-c2tv7\" (UID: \"14f52df5-6ec4-4647-ac27-1014e2669d66\") " pod="openshift-ingress/payload-processing-e2e-ait-d1107301-7b968d8f77-c2tv7"
Nov 02 03:07:59.558833 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:59.558715    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59ntt\" (UniqueName: \"kubernetes.io/projected/14f52df5-6ec4-4647-ac27-1014e2669d66-kube-api-access-59ntt\") pod \"payload-processing-e2e-ait-d1107301-7b968d8f77-c2tv7\" (UID: \"14f52df5-6ec4-4647-ac27-1014e2669d66\") " pod="openshift-ingress/payload-processing-e2e-ait-d1107301-7b968d8f77-c2tv7"
Nov 02 03:07:59.559382 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:59.559361    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14f52df5-6ec4-4647-ac27-1014e2669d66-plugins-config-volume\") pod \"payload-processing-e2e-ait-d1107301-7b968d8f77-c2tv7\" (UID: \"14f52df5-6ec4-4647-ac27-1014e2669d66\") " pod="openshift-ingress/payload-processing-e2e-ait-d1107301-7b968d8f77-c2tv7"
Nov 02 03:07:59.619593 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:59.619557    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59ntt\" (UniqueName: \"kubernetes.io/projected/14f52df5-6ec4-4647-ac27-1014e2669d66-kube-api-access-59ntt\") pod \"payload-processing-e2e-ait-d1107301-7b968d8f77-c2tv7\" (UID: \"14f52df5-6ec4-4647-ac27-1014e2669d66\") " pod="openshift-ingress/payload-processing-e2e-ait-d1107301-7b968d8f77-c2tv7"
Nov 02 03:07:59.819406 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:59.819376    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d1107301-675fb7899-nsrpt"]
Nov 02 03:07:59.820634 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:07:59.820608    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8e4bd54_5ae6_49c0_ad5f_786322530e97.slice/crio-137fd4936d915ce38ddb32a4dc5a35a836a68262a411f76e62a4a62e07979050 WatchSource:0}: Error finding container 137fd4936d915ce38ddb32a4dc5a35a836a68262a411f76e62a4a62e07979050: Status 404 returned error can't find the container with id 137fd4936d915ce38ddb32a4dc5a35a836a68262a411f76e62a4a62e07979050
Nov 02 03:07:59.900038 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:59.900004    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d1107301-7b968d8f77-c2tv7"
Nov 02 03:07:59.995316 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:59.995280    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d1107301-675fb7899-nsrpt" event={"ID":"c8e4bd54-5ae6-49c0-ad5f-786322530e97","Type":"ContainerStarted","Data":"3c40a5add4ae5c5f3d80068054010a2d9aa4e52918dcfe02f1664a1a580be4e6"}
Nov 02 03:07:59.995481 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:59.995340    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d1107301-675fb7899-nsrpt" event={"ID":"c8e4bd54-5ae6-49c0-ad5f-786322530e97","Type":"ContainerStarted","Data":"137fd4936d915ce38ddb32a4dc5a35a836a68262a411f76e62a4a62e07979050"}
Nov 02 03:07:59.995481 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:07:59.995428    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-d1107301-675fb7899-nsrpt"
Nov 02 03:08:00.111333 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:00.111229    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-d1107301-675fb7899-nsrpt" podStartSLOduration=1.111214709 podStartE2EDuration="1.111214709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:07:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:08:00.109753045 +0000 UTC m=+1220.688360425" watchObservedRunningTime="2026-11-02 03:08:00.111214709 +0000 UTC m=+1220.689822088"
Nov 02 03:08:00.272035 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:00.271996    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d1107301-7b968d8f77-c2tv7"]
Nov 02 03:08:00.272861 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:08:00.272839    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14f52df5_6ec4_4647_ac27_1014e2669d66.slice/crio-184143cbe2071b6f4a0dd46f5347d8b86e603f50d395b0bdd7727ff2ba511b05 WatchSource:0}: Error finding container 184143cbe2071b6f4a0dd46f5347d8b86e603f50d395b0bdd7727ff2ba511b05: Status 404 returned error can't find the container with id 184143cbe2071b6f4a0dd46f5347d8b86e603f50d395b0bdd7727ff2ba511b05
Nov 02 03:08:01.002803 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:01.002760    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d1107301-7b968d8f77-c2tv7" event={"ID":"14f52df5-6ec4-4647-ac27-1014e2669d66","Type":"ContainerStarted","Data":"ccd0e5d0b00e13b85f007d05b9cff30854948ba6348a02d5dd687e5c9be788cf"}
Nov 02 03:08:01.002992 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:01.002830    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d1107301-7b968d8f77-c2tv7" event={"ID":"14f52df5-6ec4-4647-ac27-1014e2669d66","Type":"ContainerStarted","Data":"184143cbe2071b6f4a0dd46f5347d8b86e603f50d395b0bdd7727ff2ba511b05"}
Nov 02 03:08:01.003067 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:01.003046    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-d1107301-7b968d8f77-c2tv7"
Nov 02 03:08:01.118534 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:01.118484    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-d1107301-7b968d8f77-c2tv7" podStartSLOduration=2.118469355 podStartE2EDuration="2.118469355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:07:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:08:01.11794028 +0000 UTC m=+1221.696547659" watchObservedRunningTime="2026-11-02 03:08:01.118469355 +0000 UTC m=+1221.697076735"
Nov 02 03:08:01.652730 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:01.652696    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4bbf98be-b9594874c-9fqx9"]
Nov 02 03:08:01.656556 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:01.656535    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4bbf98be-b9594874c-9fqx9"
Nov 02 03:08:01.718935 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:01.718897    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4bbf98be-b9594874c-9fqx9"]
Nov 02 03:08:01.719150 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:01.719125    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-4bbf98be\""
Nov 02 03:08:01.750067 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:01.750005    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-4bbf98be-dockercfg-xjxkv\""
Nov 02 03:08:01.782267 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:01.782238    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a879b9fd-06a4-4d5a-b3be-3a3d904683e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-4bbf98be-b9594874c-9fqx9\" (UID: \"a879b9fd-06a4-4d5a-b3be-3a3d904683e2\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4bbf98be-b9594874c-9fqx9"
Nov 02 03:08:01.782459 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:01.782387    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pw44r\" (UniqueName: \"kubernetes.io/projected/a879b9fd-06a4-4d5a-b3be-3a3d904683e2-kube-api-access-pw44r\") pod \"payload-pre-processing-e2e-ait-4bbf98be-b9594874c-9fqx9\" (UID: \"a879b9fd-06a4-4d5a-b3be-3a3d904683e2\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4bbf98be-b9594874c-9fqx9"
Nov 02 03:08:01.805415 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:01.805382    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4bbf98be-57fd77bc87-5lzs4"]
Nov 02 03:08:01.809405 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:01.809374    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4bbf98be-57fd77bc87-5lzs4"
Nov 02 03:08:01.864848 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:01.864811    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4bbf98be-57fd77bc87-5lzs4"]
Nov 02 03:08:01.883221 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:01.883190    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a879b9fd-06a4-4d5a-b3be-3a3d904683e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-4bbf98be-b9594874c-9fqx9\" (UID: \"a879b9fd-06a4-4d5a-b3be-3a3d904683e2\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4bbf98be-b9594874c-9fqx9"
Nov 02 03:08:01.883373 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:01.883312    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pw44r\" (UniqueName: \"kubernetes.io/projected/a879b9fd-06a4-4d5a-b3be-3a3d904683e2-kube-api-access-pw44r\") pod \"payload-pre-processing-e2e-ait-4bbf98be-b9594874c-9fqx9\" (UID: \"a879b9fd-06a4-4d5a-b3be-3a3d904683e2\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4bbf98be-b9594874c-9fqx9"
Nov 02 03:08:01.884184 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:01.884136    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a879b9fd-06a4-4d5a-b3be-3a3d904683e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-4bbf98be-b9594874c-9fqx9\" (UID: \"a879b9fd-06a4-4d5a-b3be-3a3d904683e2\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4bbf98be-b9594874c-9fqx9"
Nov 02 03:08:01.940927 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:01.940853    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pw44r\" (UniqueName: \"kubernetes.io/projected/a879b9fd-06a4-4d5a-b3be-3a3d904683e2-kube-api-access-pw44r\") pod \"payload-pre-processing-e2e-ait-4bbf98be-b9594874c-9fqx9\" (UID: \"a879b9fd-06a4-4d5a-b3be-3a3d904683e2\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4bbf98be-b9594874c-9fqx9"
Nov 02 03:08:01.970776 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:01.970742    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4bbf98be-b9594874c-9fqx9"
Nov 02 03:08:01.984262 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:01.984238    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff37f37f-b7ba-4461-b859-5e6a9cc02efc-plugins-config-volume\") pod \"payload-processing-e2e-ait-4bbf98be-57fd77bc87-5lzs4\" (UID: \"ff37f37f-b7ba-4461-b859-5e6a9cc02efc\") " pod="openshift-ingress/payload-processing-e2e-ait-4bbf98be-57fd77bc87-5lzs4"
Nov 02 03:08:01.984349 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:01.984270    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpj8h\" (UniqueName: \"kubernetes.io/projected/ff37f37f-b7ba-4461-b859-5e6a9cc02efc-kube-api-access-jpj8h\") pod \"payload-processing-e2e-ait-4bbf98be-57fd77bc87-5lzs4\" (UID: \"ff37f37f-b7ba-4461-b859-5e6a9cc02efc\") " pod="openshift-ingress/payload-processing-e2e-ait-4bbf98be-57fd77bc87-5lzs4"
Nov 02 03:08:02.085096 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:02.085057    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff37f37f-b7ba-4461-b859-5e6a9cc02efc-plugins-config-volume\") pod \"payload-processing-e2e-ait-4bbf98be-57fd77bc87-5lzs4\" (UID: \"ff37f37f-b7ba-4461-b859-5e6a9cc02efc\") " pod="openshift-ingress/payload-processing-e2e-ait-4bbf98be-57fd77bc87-5lzs4"
Nov 02 03:08:02.085096 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:02.085099    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpj8h\" (UniqueName: \"kubernetes.io/projected/ff37f37f-b7ba-4461-b859-5e6a9cc02efc-kube-api-access-jpj8h\") pod \"payload-processing-e2e-ait-4bbf98be-57fd77bc87-5lzs4\" (UID: \"ff37f37f-b7ba-4461-b859-5e6a9cc02efc\") " pod="openshift-ingress/payload-processing-e2e-ait-4bbf98be-57fd77bc87-5lzs4"
Nov 02 03:08:02.085857 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:02.085835    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff37f37f-b7ba-4461-b859-5e6a9cc02efc-plugins-config-volume\") pod \"payload-processing-e2e-ait-4bbf98be-57fd77bc87-5lzs4\" (UID: \"ff37f37f-b7ba-4461-b859-5e6a9cc02efc\") " pod="openshift-ingress/payload-processing-e2e-ait-4bbf98be-57fd77bc87-5lzs4"
Nov 02 03:08:02.143364 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:02.143332    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpj8h\" (UniqueName: \"kubernetes.io/projected/ff37f37f-b7ba-4461-b859-5e6a9cc02efc-kube-api-access-jpj8h\") pod \"payload-processing-e2e-ait-4bbf98be-57fd77bc87-5lzs4\" (UID: \"ff37f37f-b7ba-4461-b859-5e6a9cc02efc\") " pod="openshift-ingress/payload-processing-e2e-ait-4bbf98be-57fd77bc87-5lzs4"
Nov 02 03:08:02.347191 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:02.347166    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4bbf98be-b9594874c-9fqx9"]
Nov 02 03:08:02.349069 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:08:02.349036    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda879b9fd_06a4_4d5a_b3be_3a3d904683e2.slice/crio-efe1ab68d47ec2784f738a06338a14e4559b9bff722827dc199b643204ba94ff WatchSource:0}: Error finding container efe1ab68d47ec2784f738a06338a14e4559b9bff722827dc199b643204ba94ff: Status 404 returned error can't find the container with id efe1ab68d47ec2784f738a06338a14e4559b9bff722827dc199b643204ba94ff
Nov 02 03:08:02.422448 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:02.422421    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4bbf98be-57fd77bc87-5lzs4"
Nov 02 03:08:02.801813 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:02.801783    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4bbf98be-57fd77bc87-5lzs4"]
Nov 02 03:08:02.802746 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:08:02.802716    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff37f37f_b7ba_4461_b859_5e6a9cc02efc.slice/crio-3a52bf57da7bcceced5a364f6c9a06116a971b6bb22762bf8d1a933537665796 WatchSource:0}: Error finding container 3a52bf57da7bcceced5a364f6c9a06116a971b6bb22762bf8d1a933537665796: Status 404 returned error can't find the container with id 3a52bf57da7bcceced5a364f6c9a06116a971b6bb22762bf8d1a933537665796
Nov 02 03:08:03.017548 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:03.017510    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4bbf98be-57fd77bc87-5lzs4" event={"ID":"ff37f37f-b7ba-4461-b859-5e6a9cc02efc","Type":"ContainerStarted","Data":"83edd830c365bd266e55e3c43b8bc14f89e27e3d59a4148014236f6274614bc1"}
Nov 02 03:08:03.017739 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:03.017572    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4bbf98be-57fd77bc87-5lzs4" event={"ID":"ff37f37f-b7ba-4461-b859-5e6a9cc02efc","Type":"ContainerStarted","Data":"3a52bf57da7bcceced5a364f6c9a06116a971b6bb22762bf8d1a933537665796"}
Nov 02 03:08:03.017739 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:03.017673    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-4bbf98be-57fd77bc87-5lzs4"
Nov 02 03:08:03.019346 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:03.019313    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4bbf98be-b9594874c-9fqx9" event={"ID":"a879b9fd-06a4-4d5a-b3be-3a3d904683e2","Type":"ContainerStarted","Data":"20baabc5966ba63e6bc4e2cfff51891b7679fb7e81119ecfb2c789c94c21fd4d"}
Nov 02 03:08:03.019478 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:03.019370    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-4bbf98be-b9594874c-9fqx9"
Nov 02 03:08:03.019478 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:03.019396    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4bbf98be-b9594874c-9fqx9" event={"ID":"a879b9fd-06a4-4d5a-b3be-3a3d904683e2","Type":"ContainerStarted","Data":"efe1ab68d47ec2784f738a06338a14e4559b9bff722827dc199b643204ba94ff"}
Nov 02 03:08:03.140456 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:03.140356    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-4bbf98be-57fd77bc87-5lzs4" podStartSLOduration=2.140340714 podStartE2EDuration="2.140340714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:08:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:08:03.138957633 +0000 UTC m=+1223.717565010" watchObservedRunningTime="2026-11-02 03:08:03.140340714 +0000 UTC m=+1223.718948094"
Nov 02 03:08:03.253824 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:03.253761    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-4bbf98be-b9594874c-9fqx9" podStartSLOduration=2.253739742 podStartE2EDuration="2.253739742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:08:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:08:03.253122473 +0000 UTC m=+1223.831729856" watchObservedRunningTime="2026-11-02 03:08:03.253739742 +0000 UTC m=+1223.832347126"
Nov 02 03:08:03.529744 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:08:03.529705    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 03:08:03.958089 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:03.957983    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-935e4abf-57498b49f5-qkrxc"
Nov 02 03:08:03.958442 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:03.958338    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-935e4abf-7d747dd7d7-gs66f"
Nov 02 03:08:08.989690 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:08.989659    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-66a47a3b-699d7d7899-pj48x"
Nov 02 03:08:08.990097 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:08.990006    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-66a47a3b-686d7444c9-xk4nt"
Nov 02 03:08:10.896094 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:10.896060    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-bc460382-7759547547-vgh7q"]
Nov 02 03:08:10.901065 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:10.901042    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-bc460382-7759547547-vgh7q"
Nov 02 03:08:10.957662 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:10.957627    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-bc460382-7759547547-vgh7q"]
Nov 02 03:08:10.974891 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:10.974853    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j46dl\" (UniqueName: \"kubernetes.io/projected/7adde223-834d-401b-98fe-22b635e77a16-kube-api-access-j46dl\") pod \"payload-pre-processing-e2e-derive-bc460382-7759547547-vgh7q\" (UID: \"7adde223-834d-401b-98fe-22b635e77a16\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-bc460382-7759547547-vgh7q"
Nov 02 03:08:10.975101 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:10.974955    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7adde223-834d-401b-98fe-22b635e77a16-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-bc460382-7759547547-vgh7q\" (UID: \"7adde223-834d-401b-98fe-22b635e77a16\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-bc460382-7759547547-vgh7q"
Nov 02 03:08:11.004007 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:11.003972    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-bc460382\""
Nov 02 03:08:11.004227 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:11.004164    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-bc460382-dockercfg-9llk4\""
Nov 02 03:08:11.005876 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:11.005854    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-d1107301-675fb7899-nsrpt"
Nov 02 03:08:11.057613 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:11.057578    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-bc460382-7c7dd66f5-68kpj"]
Nov 02 03:08:11.061527 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:11.061490    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-bc460382-7c7dd66f5-68kpj"
Nov 02 03:08:11.075813 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:11.075791    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbbd8\" (UniqueName: \"kubernetes.io/projected/a22f346f-bf44-4fdf-adc4-1559afe4620d-kube-api-access-zbbd8\") pod \"payload-processing-e2e-derive-bc460382-7c7dd66f5-68kpj\" (UID: \"a22f346f-bf44-4fdf-adc4-1559afe4620d\") " pod="openshift-ingress/payload-processing-e2e-derive-bc460382-7c7dd66f5-68kpj"
Nov 02 03:08:11.075918 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:11.075827    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7adde223-834d-401b-98fe-22b635e77a16-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-bc460382-7759547547-vgh7q\" (UID: \"7adde223-834d-401b-98fe-22b635e77a16\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-bc460382-7759547547-vgh7q"
Nov 02 03:08:11.075963 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:11.075929    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a22f346f-bf44-4fdf-adc4-1559afe4620d-plugins-config-volume\") pod \"payload-processing-e2e-derive-bc460382-7c7dd66f5-68kpj\" (UID: \"a22f346f-bf44-4fdf-adc4-1559afe4620d\") " pod="openshift-ingress/payload-processing-e2e-derive-bc460382-7c7dd66f5-68kpj"
Nov 02 03:08:11.075963 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:11.075947    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j46dl\" (UniqueName: \"kubernetes.io/projected/7adde223-834d-401b-98fe-22b635e77a16-kube-api-access-j46dl\") pod \"payload-pre-processing-e2e-derive-bc460382-7759547547-vgh7q\" (UID: \"7adde223-834d-401b-98fe-22b635e77a16\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-bc460382-7759547547-vgh7q"
Nov 02 03:08:11.076556 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:11.076536    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7adde223-834d-401b-98fe-22b635e77a16-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-bc460382-7759547547-vgh7q\" (UID: \"7adde223-834d-401b-98fe-22b635e77a16\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-bc460382-7759547547-vgh7q"
Nov 02 03:08:11.109367 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:11.109346    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-bc460382-7c7dd66f5-68kpj"]
Nov 02 03:08:11.138569 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:11.138515    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j46dl\" (UniqueName: \"kubernetes.io/projected/7adde223-834d-401b-98fe-22b635e77a16-kube-api-access-j46dl\") pod \"payload-pre-processing-e2e-derive-bc460382-7759547547-vgh7q\" (UID: \"7adde223-834d-401b-98fe-22b635e77a16\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-bc460382-7759547547-vgh7q"
Nov 02 03:08:11.176418 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:11.176356    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbbd8\" (UniqueName: \"kubernetes.io/projected/a22f346f-bf44-4fdf-adc4-1559afe4620d-kube-api-access-zbbd8\") pod \"payload-processing-e2e-derive-bc460382-7c7dd66f5-68kpj\" (UID: \"a22f346f-bf44-4fdf-adc4-1559afe4620d\") " pod="openshift-ingress/payload-processing-e2e-derive-bc460382-7c7dd66f5-68kpj"
Nov 02 03:08:11.176531 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:11.176457    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a22f346f-bf44-4fdf-adc4-1559afe4620d-plugins-config-volume\") pod \"payload-processing-e2e-derive-bc460382-7c7dd66f5-68kpj\" (UID: \"a22f346f-bf44-4fdf-adc4-1559afe4620d\") " pod="openshift-ingress/payload-processing-e2e-derive-bc460382-7c7dd66f5-68kpj"
Nov 02 03:08:11.177094 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:11.177076    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a22f346f-bf44-4fdf-adc4-1559afe4620d-plugins-config-volume\") pod \"payload-processing-e2e-derive-bc460382-7c7dd66f5-68kpj\" (UID: \"a22f346f-bf44-4fdf-adc4-1559afe4620d\") " pod="openshift-ingress/payload-processing-e2e-derive-bc460382-7c7dd66f5-68kpj"
Nov 02 03:08:11.214405 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:11.214379    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-bc460382-7759547547-vgh7q"
Nov 02 03:08:11.234700 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:11.234672    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbbd8\" (UniqueName: \"kubernetes.io/projected/a22f346f-bf44-4fdf-adc4-1559afe4620d-kube-api-access-zbbd8\") pod \"payload-processing-e2e-derive-bc460382-7c7dd66f5-68kpj\" (UID: \"a22f346f-bf44-4fdf-adc4-1559afe4620d\") " pod="openshift-ingress/payload-processing-e2e-derive-bc460382-7c7dd66f5-68kpj"
Nov 02 03:08:11.374807 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:11.374774    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-bc460382-7c7dd66f5-68kpj"
Nov 02 03:08:11.589992 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:11.589971    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-bc460382-7759547547-vgh7q"]
Nov 02 03:08:11.591603 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:08:11.591579    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7adde223_834d_401b_98fe_22b635e77a16.slice/crio-77eaaed4c4c149620aa0d50514ad82b506197399db4eee86d969407acd3ed2a9 WatchSource:0}: Error finding container 77eaaed4c4c149620aa0d50514ad82b506197399db4eee86d969407acd3ed2a9: Status 404 returned error can't find the container with id 77eaaed4c4c149620aa0d50514ad82b506197399db4eee86d969407acd3ed2a9
Nov 02 03:08:11.746816 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:11.746793    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-bc460382-7c7dd66f5-68kpj"]
Nov 02 03:08:11.748391 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:08:11.748364    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda22f346f_bf44_4fdf_adc4_1559afe4620d.slice/crio-0245b4ddc804f2089e30800e2b91238d66dcf2e46c8a869798f19e3e6493fae8 WatchSource:0}: Error finding container 0245b4ddc804f2089e30800e2b91238d66dcf2e46c8a869798f19e3e6493fae8: Status 404 returned error can't find the container with id 0245b4ddc804f2089e30800e2b91238d66dcf2e46c8a869798f19e3e6493fae8
Nov 02 03:08:12.011745 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:12.011668    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-d1107301-7b968d8f77-c2tv7"
Nov 02 03:08:12.072652 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:12.072619    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-bc460382-7c7dd66f5-68kpj" event={"ID":"a22f346f-bf44-4fdf-adc4-1559afe4620d","Type":"ContainerStarted","Data":"fce27c407f322e39ab28b3a849111442778a548e5b28bb8af95a4dee2c233c88"}
Nov 02 03:08:12.072832 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:12.072672    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-bc460382-7c7dd66f5-68kpj" event={"ID":"a22f346f-bf44-4fdf-adc4-1559afe4620d","Type":"ContainerStarted","Data":"0245b4ddc804f2089e30800e2b91238d66dcf2e46c8a869798f19e3e6493fae8"}
Nov 02 03:08:12.072832 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:12.072714    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-bc460382-7c7dd66f5-68kpj"
Nov 02 03:08:12.074364 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:12.074339    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-bc460382-7759547547-vgh7q" event={"ID":"7adde223-834d-401b-98fe-22b635e77a16","Type":"ContainerStarted","Data":"355712ce7ce07fa21ec659556706f959b22cf5bf9406e90d8eb46ccdd53b8d4c"}
Nov 02 03:08:12.074470 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:12.074386    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-bc460382-7759547547-vgh7q"
Nov 02 03:08:12.074470 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:12.074401    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-bc460382-7759547547-vgh7q" event={"ID":"7adde223-834d-401b-98fe-22b635e77a16","Type":"ContainerStarted","Data":"77eaaed4c4c149620aa0d50514ad82b506197399db4eee86d969407acd3ed2a9"}
Nov 02 03:08:12.249285 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:12.249236    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-bc460382-7759547547-vgh7q" podStartSLOduration=2.24922394 podStartE2EDuration="2.24922394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:08:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:08:12.247300511 +0000 UTC m=+1232.825907885" watchObservedRunningTime="2026-11-02 03:08:12.24922394 +0000 UTC m=+1232.827831320"
Nov 02 03:08:12.364740 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:12.364620    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-bc460382-7c7dd66f5-68kpj" podStartSLOduration=1.364599601 podStartE2EDuration="1.364599601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:08:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:08:12.362523624 +0000 UTC m=+1232.941131007" watchObservedRunningTime="2026-11-02 03:08:12.364599601 +0000 UTC m=+1232.943206981"
Nov 02 03:08:14.028702 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:14.028657    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-4bbf98be-57fd77bc87-5lzs4"
Nov 02 03:08:14.029639 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:14.028782    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-4bbf98be-b9594874c-9fqx9"
Nov 02 03:08:23.082752 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:23.082720    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-bc460382-7759547547-vgh7q"
Nov 02 03:08:23.083235 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:23.082793    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-bc460382-7c7dd66f5-68kpj"
Nov 02 03:08:29.179818 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:29.179726    4496 generic.go:356] "Generic (PLEG): container finished" podID="9d8d9948-fdb9-40bb-830f-aa6fb34e9451" containerID="9be1cb2167112fdd91ceec1512a9cccb3d25e5353f8c850e55724f8c4b398c60" exitCode=0
Nov 02 03:08:29.179818 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:29.179798    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-kv6dd-hgn6k" event={"ID":"9d8d9948-fdb9-40bb-830f-aa6fb34e9451","Type":"ContainerDied","Data":"9be1cb2167112fdd91ceec1512a9cccb3d25e5353f8c850e55724f8c4b398c60"}
Nov 02 03:08:30.324773 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:30.324738    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-kv6dd-hgn6k"
Nov 02 03:08:30.363802 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:30.363778    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d8d9948-fdb9-40bb-830f-aa6fb34e9451-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9d8d9948-fdb9-40bb-830f-aa6fb34e9451-archives-path\") pod \"9d8d9948-fdb9-40bb-830f-aa6fb34e9451\" (UID: \"9d8d9948-fdb9-40bb-830f-aa6fb34e9451\") "
Nov 02 03:08:30.363943 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:30.363817    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d8d9948-fdb9-40bb-830f-aa6fb34e9451-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d8d9948-fdb9-40bb-830f-aa6fb34e9451-service-ca-bundle\") pod \"9d8d9948-fdb9-40bb-830f-aa6fb34e9451\" (UID: \"9d8d9948-fdb9-40bb-830f-aa6fb34e9451\") "
Nov 02 03:08:30.363943 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:30.363868    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d8d9948-fdb9-40bb-830f-aa6fb34e9451-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d8d9948-fdb9-40bb-830f-aa6fb34e9451-trusted-ca-bundle\") pod \"9d8d9948-fdb9-40bb-830f-aa6fb34e9451\" (UID: \"9d8d9948-fdb9-40bb-830f-aa6fb34e9451\") "
Nov 02 03:08:30.363943 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:30.363894    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d8d9948-fdb9-40bb-830f-aa6fb34e9451-kube-api-access-bvxpw\" (UniqueName: \"kubernetes.io/projected/9d8d9948-fdb9-40bb-830f-aa6fb34e9451-kube-api-access-bvxpw\") pod \"9d8d9948-fdb9-40bb-830f-aa6fb34e9451\" (UID: \"9d8d9948-fdb9-40bb-830f-aa6fb34e9451\") "
Nov 02 03:08:30.364187 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:30.364161    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d8d9948-fdb9-40bb-830f-aa6fb34e9451-archives-path" pod "9d8d9948-fdb9-40bb-830f-aa6fb34e9451" (UID: "9d8d9948-fdb9-40bb-830f-aa6fb34e9451"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:08:30.364258 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:30.364221    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d8d9948-fdb9-40bb-830f-aa6fb34e9451-service-ca-bundle" pod "9d8d9948-fdb9-40bb-830f-aa6fb34e9451" (UID: "9d8d9948-fdb9-40bb-830f-aa6fb34e9451"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:08:30.364258 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:30.364241    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d8d9948-fdb9-40bb-830f-aa6fb34e9451-trusted-ca-bundle" pod "9d8d9948-fdb9-40bb-830f-aa6fb34e9451" (UID: "9d8d9948-fdb9-40bb-830f-aa6fb34e9451"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:08:30.364353 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:30.364299    4496 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9d8d9948-fdb9-40bb-830f-aa6fb34e9451-archives-path\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:08:30.366046 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:30.366026    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d8d9948-fdb9-40bb-830f-aa6fb34e9451-kube-api-access-bvxpw" pod "9d8d9948-fdb9-40bb-830f-aa6fb34e9451" (UID: "9d8d9948-fdb9-40bb-830f-aa6fb34e9451"). InnerVolumeSpecName "kube-api-access-bvxpw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:08:30.465705 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:30.465625    4496 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d8d9948-fdb9-40bb-830f-aa6fb34e9451-service-ca-bundle\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:08:30.465705 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:30.465665    4496 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d8d9948-fdb9-40bb-830f-aa6fb34e9451-trusted-ca-bundle\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:08:30.465880 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:30.465707    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvxpw\" (UniqueName: \"kubernetes.io/projected/9d8d9948-fdb9-40bb-830f-aa6fb34e9451-kube-api-access-bvxpw\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:08:31.192708 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:31.192675    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-kv6dd-hgn6k" event={"ID":"9d8d9948-fdb9-40bb-830f-aa6fb34e9451","Type":"ContainerDied","Data":"6404f125b563287dec37d99a68bf39946b6e9027b15c97a9ccc8eab6bf51ba6d"}
Nov 02 03:08:31.192883 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:31.192720    4496 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6404f125b563287dec37d99a68bf39946b6e9027b15c97a9ccc8eab6bf51ba6d"
Nov 02 03:08:31.192883 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:31.192689    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-kv6dd-hgn6k"
Nov 02 03:08:33.526322 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:08:33.526284    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 03:08:34.629319 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:34.629285    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d1107301-vfm7v"]
Nov 02 03:08:34.629816 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:34.629802    4496 cpu_manager.go:414] "containerMap: removing container" podUID="9d8d9948-fdb9-40bb-830f-aa6fb34e9451" containerName="insights-gathering"
Nov 02 03:08:34.629861 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:34.629817    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d8d9948-fdb9-40bb-830f-aa6fb34e9451" containerName="insights-gathering"
Nov 02 03:08:34.629935 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:34.629926    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d8d9948-fdb9-40bb-830f-aa6fb34e9451" containerName="insights-gathering"
Nov 02 03:08:34.635095 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:34.635067    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d1107301-vfm7v"
Nov 02 03:08:34.691886 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:34.691857    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d1107301-vfm7v"]
Nov 02 03:08:34.705468 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:34.705440    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p59tb\" (UniqueName: \"kubernetes.io/projected/cc960cba-6ad5-499f-aabc-a0ff5f8ba51b-kube-api-access-p59tb\") pod \"maas-api-revoke-keys-e2e-ait-d1107301-vfm7v\" (UID: \"cc960cba-6ad5-499f-aabc-a0ff5f8ba51b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d1107301-vfm7v"
Nov 02 03:08:34.705622 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:34.705481    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc960cba-6ad5-499f-aabc-a0ff5f8ba51b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-d1107301-vfm7v\" (UID: \"cc960cba-6ad5-499f-aabc-a0ff5f8ba51b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d1107301-vfm7v"
Nov 02 03:08:34.705867 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:34.705842    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc960cba-6ad5-499f-aabc-a0ff5f8ba51b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-d1107301-vfm7v\" (UID: \"cc960cba-6ad5-499f-aabc-a0ff5f8ba51b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d1107301-vfm7v"
Nov 02 03:08:34.739156 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:34.739128    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 03:08:34.739328 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:34.739293    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 03:08:34.739328 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:34.739303    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 03:08:34.739439 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:34.739353    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9z5bk\""
Nov 02 03:08:34.807182 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:34.807154    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p59tb\" (UniqueName: \"kubernetes.io/projected/cc960cba-6ad5-499f-aabc-a0ff5f8ba51b-kube-api-access-p59tb\") pod \"maas-api-revoke-keys-e2e-ait-d1107301-vfm7v\" (UID: \"cc960cba-6ad5-499f-aabc-a0ff5f8ba51b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d1107301-vfm7v"
Nov 02 03:08:34.807182 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:34.807185    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc960cba-6ad5-499f-aabc-a0ff5f8ba51b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-d1107301-vfm7v\" (UID: \"cc960cba-6ad5-499f-aabc-a0ff5f8ba51b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d1107301-vfm7v"
Nov 02 03:08:34.807417 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:34.807301    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc960cba-6ad5-499f-aabc-a0ff5f8ba51b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-d1107301-vfm7v\" (UID: \"cc960cba-6ad5-499f-aabc-a0ff5f8ba51b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d1107301-vfm7v"
Nov 02 03:08:34.807913 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:34.807883    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc960cba-6ad5-499f-aabc-a0ff5f8ba51b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-d1107301-vfm7v\" (UID: \"cc960cba-6ad5-499f-aabc-a0ff5f8ba51b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d1107301-vfm7v"
Nov 02 03:08:34.809516 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:34.809493    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc960cba-6ad5-499f-aabc-a0ff5f8ba51b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-d1107301-vfm7v\" (UID: \"cc960cba-6ad5-499f-aabc-a0ff5f8ba51b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d1107301-vfm7v"
Nov 02 03:08:34.865742 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:34.865711    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p59tb\" (UniqueName: \"kubernetes.io/projected/cc960cba-6ad5-499f-aabc-a0ff5f8ba51b-kube-api-access-p59tb\") pod \"maas-api-revoke-keys-e2e-ait-d1107301-vfm7v\" (UID: \"cc960cba-6ad5-499f-aabc-a0ff5f8ba51b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d1107301-vfm7v"
Nov 02 03:08:34.948169 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:34.948084    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d1107301-vfm7v"
Nov 02 03:08:35.319868 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:35.319843    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d1107301-vfm7v"]
Nov 02 03:08:35.320975 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:08:35.320952    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc960cba_6ad5_499f_aabc_a0ff5f8ba51b.slice/crio-07c7e5defea46f24003712b80f84ccb081bdd4e314adca27d352717d17e8d5b6 WatchSource:0}: Error finding container 07c7e5defea46f24003712b80f84ccb081bdd4e314adca27d352717d17e8d5b6: Status 404 returned error can't find the container with id 07c7e5defea46f24003712b80f84ccb081bdd4e314adca27d352717d17e8d5b6
Nov 02 03:08:35.322856 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:35.322840    4496 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 03:08:36.223312 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:36.223279    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d1107301-vfm7v" event={"ID":"cc960cba-6ad5-499f-aabc-a0ff5f8ba51b","Type":"ContainerStarted","Data":"07c7e5defea46f24003712b80f84ccb081bdd4e314adca27d352717d17e8d5b6"}
Nov 02 03:08:39.304364 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:39.304322    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-935e4abf-4hmkn"]
Nov 02 03:08:39.312212 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:39.312188    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-935e4abf-4hmkn"
Nov 02 03:08:39.362630 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:39.362595    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-935e4abf-4hmkn"]
Nov 02 03:08:39.454705 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:39.454668    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nn9kb\" (UniqueName: \"kubernetes.io/projected/a6199019-fe2d-4d93-badd-0b38509b0dcd-kube-api-access-nn9kb\") pod \"maas-api-revoke-keys-e2e-migrate-935e4abf-4hmkn\" (UID: \"a6199019-fe2d-4d93-badd-0b38509b0dcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-935e4abf-4hmkn"
Nov 02 03:08:39.454893 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:39.454722    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6199019-fe2d-4d93-badd-0b38509b0dcd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-935e4abf-4hmkn\" (UID: \"a6199019-fe2d-4d93-badd-0b38509b0dcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-935e4abf-4hmkn"
Nov 02 03:08:39.454893 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:39.454871    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6199019-fe2d-4d93-badd-0b38509b0dcd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-935e4abf-4hmkn\" (UID: \"a6199019-fe2d-4d93-badd-0b38509b0dcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-935e4abf-4hmkn"
Nov 02 03:08:39.556269 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:39.556187    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6199019-fe2d-4d93-badd-0b38509b0dcd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-935e4abf-4hmkn\" (UID: \"a6199019-fe2d-4d93-badd-0b38509b0dcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-935e4abf-4hmkn"
Nov 02 03:08:39.556441 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:39.556265    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6199019-fe2d-4d93-badd-0b38509b0dcd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-935e4abf-4hmkn\" (UID: \"a6199019-fe2d-4d93-badd-0b38509b0dcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-935e4abf-4hmkn"
Nov 02 03:08:39.556441 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:39.556340    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nn9kb\" (UniqueName: \"kubernetes.io/projected/a6199019-fe2d-4d93-badd-0b38509b0dcd-kube-api-access-nn9kb\") pod \"maas-api-revoke-keys-e2e-migrate-935e4abf-4hmkn\" (UID: \"a6199019-fe2d-4d93-badd-0b38509b0dcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-935e4abf-4hmkn"
Nov 02 03:08:39.557035 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:39.556994    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6199019-fe2d-4d93-badd-0b38509b0dcd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-935e4abf-4hmkn\" (UID: \"a6199019-fe2d-4d93-badd-0b38509b0dcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-935e4abf-4hmkn"
Nov 02 03:08:39.558947 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:39.558918    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6199019-fe2d-4d93-badd-0b38509b0dcd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-935e4abf-4hmkn\" (UID: \"a6199019-fe2d-4d93-badd-0b38509b0dcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-935e4abf-4hmkn"
Nov 02 03:08:39.614891 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:39.614865    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nn9kb\" (UniqueName: \"kubernetes.io/projected/a6199019-fe2d-4d93-badd-0b38509b0dcd-kube-api-access-nn9kb\") pod \"maas-api-revoke-keys-e2e-migrate-935e4abf-4hmkn\" (UID: \"a6199019-fe2d-4d93-badd-0b38509b0dcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-935e4abf-4hmkn"
Nov 02 03:08:39.628976 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:39.628951    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-935e4abf-4hmkn"
Nov 02 03:08:40.017243 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:40.017202    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-935e4abf-4hmkn"]
Nov 02 03:08:40.019966 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:08:40.019850    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6199019_fe2d_4d93_badd_0b38509b0dcd.slice/crio-d86aa84ce13ac61c0df66ad2486878d1aeeafe81c2dec860f173cd185428608a WatchSource:0}: Error finding container d86aa84ce13ac61c0df66ad2486878d1aeeafe81c2dec860f173cd185428608a: Status 404 returned error can't find the container with id d86aa84ce13ac61c0df66ad2486878d1aeeafe81c2dec860f173cd185428608a
Nov 02 03:08:40.249027 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:40.248981    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-935e4abf-4hmkn" event={"ID":"a6199019-fe2d-4d93-badd-0b38509b0dcd","Type":"ContainerStarted","Data":"d86aa84ce13ac61c0df66ad2486878d1aeeafe81c2dec860f173cd185428608a"}
Nov 02 03:08:41.257621 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:41.257587    4496 generic.go:356] "Generic (PLEG): container finished" podID="cc960cba-6ad5-499f-aabc-a0ff5f8ba51b" containerID="bc066aa7a3a37cc3379cb4487d3db5ae6b32747d7f20041a3216da434dae43eb" exitCode=0
Nov 02 03:08:41.257997 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:41.257669    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d1107301-vfm7v" event={"ID":"cc960cba-6ad5-499f-aabc-a0ff5f8ba51b","Type":"ContainerDied","Data":"bc066aa7a3a37cc3379cb4487d3db5ae6b32747d7f20041a3216da434dae43eb"}
Nov 02 03:08:41.259192 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:41.259168    4496 generic.go:356] "Generic (PLEG): container finished" podID="a6199019-fe2d-4d93-badd-0b38509b0dcd" containerID="93916a427110f0f0a5a2db3bd7b792560c2ea4a786e9cf96f4236115e12baf77" exitCode=0
Nov 02 03:08:41.259309 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:41.259199    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-935e4abf-4hmkn" event={"ID":"a6199019-fe2d-4d93-badd-0b38509b0dcd","Type":"ContainerDied","Data":"93916a427110f0f0a5a2db3bd7b792560c2ea4a786e9cf96f4236115e12baf77"}
Nov 02 03:08:42.425002 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:42.424976    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d1107301-vfm7v"
Nov 02 03:08:42.428287 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:42.428264    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-935e4abf-4hmkn"
Nov 02 03:08:42.586354 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:42.586280    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6199019-fe2d-4d93-badd-0b38509b0dcd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6199019-fe2d-4d93-badd-0b38509b0dcd-maas-api-xks-ca\") pod \"a6199019-fe2d-4d93-badd-0b38509b0dcd\" (UID: \"a6199019-fe2d-4d93-badd-0b38509b0dcd\") "
Nov 02 03:08:42.586354 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:42.586312    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6199019-fe2d-4d93-badd-0b38509b0dcd-kube-api-access-nn9kb\" (UniqueName: \"kubernetes.io/projected/a6199019-fe2d-4d93-badd-0b38509b0dcd-kube-api-access-nn9kb\") pod \"a6199019-fe2d-4d93-badd-0b38509b0dcd\" (UID: \"a6199019-fe2d-4d93-badd-0b38509b0dcd\") "
Nov 02 03:08:42.586575 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:42.586358    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc960cba-6ad5-499f-aabc-a0ff5f8ba51b-kube-api-access-p59tb\" (UniqueName: \"kubernetes.io/projected/cc960cba-6ad5-499f-aabc-a0ff5f8ba51b-kube-api-access-p59tb\") pod \"cc960cba-6ad5-499f-aabc-a0ff5f8ba51b\" (UID: \"cc960cba-6ad5-499f-aabc-a0ff5f8ba51b\") "
Nov 02 03:08:42.586575 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:42.586519    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc960cba-6ad5-499f-aabc-a0ff5f8ba51b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc960cba-6ad5-499f-aabc-a0ff5f8ba51b-maas-api-xks-ca\") pod \"cc960cba-6ad5-499f-aabc-a0ff5f8ba51b\" (UID: \"cc960cba-6ad5-499f-aabc-a0ff5f8ba51b\") "
Nov 02 03:08:42.586686 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:42.586602    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc960cba-6ad5-499f-aabc-a0ff5f8ba51b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc960cba-6ad5-499f-aabc-a0ff5f8ba51b-maas-api-openshift-ca\") pod \"cc960cba-6ad5-499f-aabc-a0ff5f8ba51b\" (UID: \"cc960cba-6ad5-499f-aabc-a0ff5f8ba51b\") "
Nov 02 03:08:42.586780 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:42.586762    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6199019-fe2d-4d93-badd-0b38509b0dcd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6199019-fe2d-4d93-badd-0b38509b0dcd-maas-api-openshift-ca\") pod \"a6199019-fe2d-4d93-badd-0b38509b0dcd\" (UID: \"a6199019-fe2d-4d93-badd-0b38509b0dcd\") "
Nov 02 03:08:42.586986 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:42.586958    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc960cba-6ad5-499f-aabc-a0ff5f8ba51b-maas-api-openshift-ca" pod "cc960cba-6ad5-499f-aabc-a0ff5f8ba51b" (UID: "cc960cba-6ad5-499f-aabc-a0ff5f8ba51b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:08:42.588052 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:42.587682    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6199019-fe2d-4d93-badd-0b38509b0dcd-maas-api-openshift-ca" pod "a6199019-fe2d-4d93-badd-0b38509b0dcd" (UID: "a6199019-fe2d-4d93-badd-0b38509b0dcd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:08:42.588052 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:42.587967    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6199019-fe2d-4d93-badd-0b38509b0dcd-maas-api-openshift-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:08:42.589048 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:42.587989    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc960cba-6ad5-499f-aabc-a0ff5f8ba51b-maas-api-openshift-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:08:42.589048 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:42.588945    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6199019-fe2d-4d93-badd-0b38509b0dcd-maas-api-xks-ca" pod "a6199019-fe2d-4d93-badd-0b38509b0dcd" (UID: "a6199019-fe2d-4d93-badd-0b38509b0dcd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 03:08:42.589902 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:42.589381    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6199019-fe2d-4d93-badd-0b38509b0dcd-kube-api-access-nn9kb" pod "a6199019-fe2d-4d93-badd-0b38509b0dcd" (UID: "a6199019-fe2d-4d93-badd-0b38509b0dcd"). InnerVolumeSpecName "kube-api-access-nn9kb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:08:42.589902 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:42.589617    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc960cba-6ad5-499f-aabc-a0ff5f8ba51b-kube-api-access-p59tb" pod "cc960cba-6ad5-499f-aabc-a0ff5f8ba51b" (UID: "cc960cba-6ad5-499f-aabc-a0ff5f8ba51b"). InnerVolumeSpecName "kube-api-access-p59tb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:08:42.592499 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:42.590247    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc960cba-6ad5-499f-aabc-a0ff5f8ba51b-maas-api-xks-ca" pod "cc960cba-6ad5-499f-aabc-a0ff5f8ba51b" (UID: "cc960cba-6ad5-499f-aabc-a0ff5f8ba51b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 03:08:42.690290 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:42.690255    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p59tb\" (UniqueName: \"kubernetes.io/projected/cc960cba-6ad5-499f-aabc-a0ff5f8ba51b-kube-api-access-p59tb\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:08:42.690290 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:42.690291    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc960cba-6ad5-499f-aabc-a0ff5f8ba51b-maas-api-xks-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:08:42.690511 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:42.690307    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6199019-fe2d-4d93-badd-0b38509b0dcd-maas-api-xks-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:08:42.690511 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:42.690322    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nn9kb\" (UniqueName: \"kubernetes.io/projected/a6199019-fe2d-4d93-badd-0b38509b0dcd-kube-api-access-nn9kb\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:08:43.273695 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:43.273663    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d1107301-vfm7v"
Nov 02 03:08:43.273887 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:43.273695    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d1107301-vfm7v" event={"ID":"cc960cba-6ad5-499f-aabc-a0ff5f8ba51b","Type":"ContainerDied","Data":"07c7e5defea46f24003712b80f84ccb081bdd4e314adca27d352717d17e8d5b6"}
Nov 02 03:08:43.273887 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:43.273776    4496 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="07c7e5defea46f24003712b80f84ccb081bdd4e314adca27d352717d17e8d5b6"
Nov 02 03:08:43.275090 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:43.275068    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-935e4abf-4hmkn" event={"ID":"a6199019-fe2d-4d93-badd-0b38509b0dcd","Type":"ContainerDied","Data":"d86aa84ce13ac61c0df66ad2486878d1aeeafe81c2dec860f173cd185428608a"}
Nov 02 03:08:43.275174 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:43.275115    4496 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d86aa84ce13ac61c0df66ad2486878d1aeeafe81c2dec860f173cd185428608a"
Nov 02 03:08:43.275174 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:43.275075    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-935e4abf-4hmkn"
Nov 02 03:08:46.122025 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:46.121989    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d1107301-7b968d8f77-c2tv7"]
Nov 02 03:08:46.228854 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:46.228819    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d1107301-675fb7899-nsrpt"]
Nov 02 03:08:50.111906 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:50.111869    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-66a47a3b-jl2xl"]
Nov 02 03:08:50.112442 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:50.112425    4496 cpu_manager.go:414] "containerMap: removing container" podUID="cc960cba-6ad5-499f-aabc-a0ff5f8ba51b" containerName="revoke-keys"
Nov 02 03:08:50.112486 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:50.112442    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc960cba-6ad5-499f-aabc-a0ff5f8ba51b" containerName="revoke-keys"
Nov 02 03:08:50.112486 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:50.112475    4496 cpu_manager.go:414] "containerMap: removing container" podUID="a6199019-fe2d-4d93-badd-0b38509b0dcd" containerName="revoke-keys"
Nov 02 03:08:50.112486 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:50.112480    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6199019-fe2d-4d93-badd-0b38509b0dcd" containerName="revoke-keys"
Nov 02 03:08:50.112580 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:50.112563    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6199019-fe2d-4d93-badd-0b38509b0dcd" containerName="revoke-keys"
Nov 02 03:08:50.112580 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:50.112575    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc960cba-6ad5-499f-aabc-a0ff5f8ba51b" containerName="revoke-keys"
Nov 02 03:08:50.117030 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:50.116999    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-66a47a3b-jl2xl"
Nov 02 03:08:50.170524 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:50.170499    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-66a47a3b-jl2xl"]
Nov 02 03:08:50.218096 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:50.218073    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 03:08:50.218096 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:50.218089    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 03:08:50.220479 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:50.220455    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 03:08:50.220590 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:50.220540    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9z5bk\""
Nov 02 03:08:50.268546 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:50.268516    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tktth\" (UniqueName: \"kubernetes.io/projected/31830abf-3442-42e0-b1ee-bf14bd1a4534-kube-api-access-tktth\") pod \"maas-api-revoke-keys-e2e-ait-66a47a3b-jl2xl\" (UID: \"31830abf-3442-42e0-b1ee-bf14bd1a4534\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-66a47a3b-jl2xl"
Nov 02 03:08:50.268678 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:50.268579    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31830abf-3442-42e0-b1ee-bf14bd1a4534-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-66a47a3b-jl2xl\" (UID: \"31830abf-3442-42e0-b1ee-bf14bd1a4534\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-66a47a3b-jl2xl"
Nov 02 03:08:50.268678 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:50.268604    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31830abf-3442-42e0-b1ee-bf14bd1a4534-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-66a47a3b-jl2xl\" (UID: \"31830abf-3442-42e0-b1ee-bf14bd1a4534\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-66a47a3b-jl2xl"
Nov 02 03:08:50.369442 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:50.369368    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31830abf-3442-42e0-b1ee-bf14bd1a4534-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-66a47a3b-jl2xl\" (UID: \"31830abf-3442-42e0-b1ee-bf14bd1a4534\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-66a47a3b-jl2xl"
Nov 02 03:08:50.369442 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:50.369405    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31830abf-3442-42e0-b1ee-bf14bd1a4534-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-66a47a3b-jl2xl\" (UID: \"31830abf-3442-42e0-b1ee-bf14bd1a4534\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-66a47a3b-jl2xl"
Nov 02 03:08:50.369658 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:50.369481    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tktth\" (UniqueName: \"kubernetes.io/projected/31830abf-3442-42e0-b1ee-bf14bd1a4534-kube-api-access-tktth\") pod \"maas-api-revoke-keys-e2e-ait-66a47a3b-jl2xl\" (UID: \"31830abf-3442-42e0-b1ee-bf14bd1a4534\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-66a47a3b-jl2xl"
Nov 02 03:08:50.370139 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:50.370117    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31830abf-3442-42e0-b1ee-bf14bd1a4534-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-66a47a3b-jl2xl\" (UID: \"31830abf-3442-42e0-b1ee-bf14bd1a4534\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-66a47a3b-jl2xl"
Nov 02 03:08:50.371820 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:50.371802    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31830abf-3442-42e0-b1ee-bf14bd1a4534-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-66a47a3b-jl2xl\" (UID: \"31830abf-3442-42e0-b1ee-bf14bd1a4534\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-66a47a3b-jl2xl"
Nov 02 03:08:50.427210 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:50.427185    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tktth\" (UniqueName: \"kubernetes.io/projected/31830abf-3442-42e0-b1ee-bf14bd1a4534-kube-api-access-tktth\") pod \"maas-api-revoke-keys-e2e-ait-66a47a3b-jl2xl\" (UID: \"31830abf-3442-42e0-b1ee-bf14bd1a4534\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-66a47a3b-jl2xl"
Nov 02 03:08:50.429944 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:50.429928    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-66a47a3b-jl2xl"
Nov 02 03:08:50.802170 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:50.802143    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-66a47a3b-jl2xl"]
Nov 02 03:08:50.802975 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:08:50.802948    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31830abf_3442_42e0_b1ee_bf14bd1a4534.slice/crio-60a421cd7cff9d5c4b0e8423a7f297ca1d4f79a9ae31b74178413b2db7e9fe5d WatchSource:0}: Error finding container 60a421cd7cff9d5c4b0e8423a7f297ca1d4f79a9ae31b74178413b2db7e9fe5d: Status 404 returned error can't find the container with id 60a421cd7cff9d5c4b0e8423a7f297ca1d4f79a9ae31b74178413b2db7e9fe5d
Nov 02 03:08:51.122336 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:51.122287    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-d1107301-7b968d8f77-c2tv7" podUID="14f52df5-6ec4-4647-ac27-1014e2669d66" containerName="payload-processing" containerID="cri-o://ccd0e5d0b00e13b85f007d05b9cff30854948ba6348a02d5dd687e5c9be788cf" gracePeriod=25
Nov 02 03:08:51.229610 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:51.229554    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-d1107301-675fb7899-nsrpt" podUID="c8e4bd54-5ae6-49c0-ad5f-786322530e97" containerName="payload-pre-processing" containerID="cri-o://3c40a5add4ae5c5f3d80068054010a2d9aa4e52918dcfe02f1664a1a580be4e6" gracePeriod=25
Nov 02 03:08:51.327638 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:51.327570    4496 generic.go:356] "Generic (PLEG): container finished" podID="14f52df5-6ec4-4647-ac27-1014e2669d66" containerID="ccd0e5d0b00e13b85f007d05b9cff30854948ba6348a02d5dd687e5c9be788cf" exitCode=0
Nov 02 03:08:51.327778 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:51.327663    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d1107301-7b968d8f77-c2tv7" event={"ID":"14f52df5-6ec4-4647-ac27-1014e2669d66","Type":"ContainerDied","Data":"ccd0e5d0b00e13b85f007d05b9cff30854948ba6348a02d5dd687e5c9be788cf"}
Nov 02 03:08:51.329471 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:51.329446    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-66a47a3b-jl2xl" event={"ID":"31830abf-3442-42e0-b1ee-bf14bd1a4534","Type":"ContainerStarted","Data":"9f36b9bb5119eb47bb21369a13ab71640e278abb370d0c5366c43d324a931ee0"}
Nov 02 03:08:51.329609 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:51.329488    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-66a47a3b-jl2xl" event={"ID":"31830abf-3442-42e0-b1ee-bf14bd1a4534","Type":"ContainerStarted","Data":"60a421cd7cff9d5c4b0e8423a7f297ca1d4f79a9ae31b74178413b2db7e9fe5d"}
Nov 02 03:08:51.372235 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:51.372205    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d1107301-7b968d8f77-c2tv7"
Nov 02 03:08:51.445266 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:51.445210    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-66a47a3b-jl2xl" podStartSLOduration=1.445191164 podStartE2EDuration="1.445191164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:08:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:08:51.443524864 +0000 UTC m=+1272.022132246" watchObservedRunningTime="2026-11-02 03:08:51.445191164 +0000 UTC m=+1272.023798546"
Nov 02 03:08:51.481600 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:51.481575    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14f52df5-6ec4-4647-ac27-1014e2669d66-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14f52df5-6ec4-4647-ac27-1014e2669d66-plugins-config-volume\") pod \"14f52df5-6ec4-4647-ac27-1014e2669d66\" (UID: \"14f52df5-6ec4-4647-ac27-1014e2669d66\") "
Nov 02 03:08:51.481741 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:51.481659    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14f52df5-6ec4-4647-ac27-1014e2669d66-kube-api-access-59ntt\" (UniqueName: \"kubernetes.io/projected/14f52df5-6ec4-4647-ac27-1014e2669d66-kube-api-access-59ntt\") pod \"14f52df5-6ec4-4647-ac27-1014e2669d66\" (UID: \"14f52df5-6ec4-4647-ac27-1014e2669d66\") "
Nov 02 03:08:51.481992 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:51.481963    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14f52df5-6ec4-4647-ac27-1014e2669d66-plugins-config-volume" pod "14f52df5-6ec4-4647-ac27-1014e2669d66" (UID: "14f52df5-6ec4-4647-ac27-1014e2669d66"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:08:51.482144 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:51.482117    4496 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14f52df5-6ec4-4647-ac27-1014e2669d66-plugins-config-volume\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:08:51.483899 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:51.483875    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14f52df5-6ec4-4647-ac27-1014e2669d66-kube-api-access-59ntt" pod "14f52df5-6ec4-4647-ac27-1014e2669d66" (UID: "14f52df5-6ec4-4647-ac27-1014e2669d66"). InnerVolumeSpecName "kube-api-access-59ntt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:08:51.503297 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:51.503275    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d1107301-675fb7899-nsrpt"
Nov 02 03:08:51.584003 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:51.583871    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59ntt\" (UniqueName: \"kubernetes.io/projected/14f52df5-6ec4-4647-ac27-1014e2669d66-kube-api-access-59ntt\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:08:51.684296 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:51.684263    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8e4bd54-5ae6-49c0-ad5f-786322530e97-kube-api-access-hwc74\" (UniqueName: \"kubernetes.io/projected/c8e4bd54-5ae6-49c0-ad5f-786322530e97-kube-api-access-hwc74\") pod \"c8e4bd54-5ae6-49c0-ad5f-786322530e97\" (UID: \"c8e4bd54-5ae6-49c0-ad5f-786322530e97\") "
Nov 02 03:08:51.684440 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:51.684404    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8e4bd54-5ae6-49c0-ad5f-786322530e97-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8e4bd54-5ae6-49c0-ad5f-786322530e97-plugins-config-volume\") pod \"c8e4bd54-5ae6-49c0-ad5f-786322530e97\" (UID: \"c8e4bd54-5ae6-49c0-ad5f-786322530e97\") "
Nov 02 03:08:51.684821 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:51.684789    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8e4bd54-5ae6-49c0-ad5f-786322530e97-plugins-config-volume" pod "c8e4bd54-5ae6-49c0-ad5f-786322530e97" (UID: "c8e4bd54-5ae6-49c0-ad5f-786322530e97"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:08:51.686636 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:51.686607    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8e4bd54-5ae6-49c0-ad5f-786322530e97-kube-api-access-hwc74" pod "c8e4bd54-5ae6-49c0-ad5f-786322530e97" (UID: "c8e4bd54-5ae6-49c0-ad5f-786322530e97"). InnerVolumeSpecName "kube-api-access-hwc74". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:08:51.785369 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:51.785338    4496 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8e4bd54-5ae6-49c0-ad5f-786322530e97-plugins-config-volume\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:08:51.785369 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:51.785360    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwc74\" (UniqueName: \"kubernetes.io/projected/c8e4bd54-5ae6-49c0-ad5f-786322530e97-kube-api-access-hwc74\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:08:52.336799 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:52.336765    4496 generic.go:356] "Generic (PLEG): container finished" podID="31830abf-3442-42e0-b1ee-bf14bd1a4534" containerID="9f36b9bb5119eb47bb21369a13ab71640e278abb370d0c5366c43d324a931ee0" exitCode=0
Nov 02 03:08:52.337242 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:52.336814    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-66a47a3b-jl2xl" event={"ID":"31830abf-3442-42e0-b1ee-bf14bd1a4534","Type":"ContainerDied","Data":"9f36b9bb5119eb47bb21369a13ab71640e278abb370d0c5366c43d324a931ee0"}
Nov 02 03:08:52.338273 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:52.338246    4496 generic.go:356] "Generic (PLEG): container finished" podID="c8e4bd54-5ae6-49c0-ad5f-786322530e97" containerID="3c40a5add4ae5c5f3d80068054010a2d9aa4e52918dcfe02f1664a1a580be4e6" exitCode=0
Nov 02 03:08:52.338419 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:52.338285    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d1107301-675fb7899-nsrpt"
Nov 02 03:08:52.338419 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:52.338329    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d1107301-675fb7899-nsrpt" event={"ID":"c8e4bd54-5ae6-49c0-ad5f-786322530e97","Type":"ContainerDied","Data":"3c40a5add4ae5c5f3d80068054010a2d9aa4e52918dcfe02f1664a1a580be4e6"}
Nov 02 03:08:52.338419 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:52.338378    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d1107301-675fb7899-nsrpt" event={"ID":"c8e4bd54-5ae6-49c0-ad5f-786322530e97","Type":"ContainerDied","Data":"137fd4936d915ce38ddb32a4dc5a35a836a68262a411f76e62a4a62e07979050"}
Nov 02 03:08:52.338419 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:52.338408    4496 scope.go:122] "RemoveContainer" containerID="3c40a5add4ae5c5f3d80068054010a2d9aa4e52918dcfe02f1664a1a580be4e6"
Nov 02 03:08:52.339854 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:52.339838    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d1107301-7b968d8f77-c2tv7"
Nov 02 03:08:52.339971 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:52.339865    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d1107301-7b968d8f77-c2tv7" event={"ID":"14f52df5-6ec4-4647-ac27-1014e2669d66","Type":"ContainerDied","Data":"184143cbe2071b6f4a0dd46f5347d8b86e603f50d395b0bdd7727ff2ba511b05"}
Nov 02 03:08:52.348676 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:52.348657    4496 scope.go:122] "RemoveContainer" containerID="3c40a5add4ae5c5f3d80068054010a2d9aa4e52918dcfe02f1664a1a580be4e6"
Nov 02 03:08:52.348934 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:08:52.348917    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c40a5add4ae5c5f3d80068054010a2d9aa4e52918dcfe02f1664a1a580be4e6\": container with ID starting with 3c40a5add4ae5c5f3d80068054010a2d9aa4e52918dcfe02f1664a1a580be4e6 not found: ID does not exist" containerID="3c40a5add4ae5c5f3d80068054010a2d9aa4e52918dcfe02f1664a1a580be4e6"
Nov 02 03:08:52.348972 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:52.348943    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c40a5add4ae5c5f3d80068054010a2d9aa4e52918dcfe02f1664a1a580be4e6"} err="failed to get container status \"3c40a5add4ae5c5f3d80068054010a2d9aa4e52918dcfe02f1664a1a580be4e6\": rpc error: code = NotFound desc = could not find container \"3c40a5add4ae5c5f3d80068054010a2d9aa4e52918dcfe02f1664a1a580be4e6\": container with ID starting with 3c40a5add4ae5c5f3d80068054010a2d9aa4e52918dcfe02f1664a1a580be4e6 not found: ID does not exist"
Nov 02 03:08:52.348972 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:52.348960    4496 scope.go:122] "RemoveContainer" containerID="ccd0e5d0b00e13b85f007d05b9cff30854948ba6348a02d5dd687e5c9be788cf"
Nov 02 03:08:52.568156 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:52.568129    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d1107301-7b968d8f77-c2tv7"]
Nov 02 03:08:52.572713 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:52.572686    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d1107301-7b968d8f77-c2tv7"]
Nov 02 03:08:52.683258 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:52.683186    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d1107301-675fb7899-nsrpt"]
Nov 02 03:08:52.689477 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:52.689452    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d1107301-675fb7899-nsrpt"]
Nov 02 03:08:53.482959 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:53.482936    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-66a47a3b-jl2xl"
Nov 02 03:08:53.499821 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:53.499799    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31830abf-3442-42e0-b1ee-bf14bd1a4534-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31830abf-3442-42e0-b1ee-bf14bd1a4534-maas-api-openshift-ca\") pod \"31830abf-3442-42e0-b1ee-bf14bd1a4534\" (UID: \"31830abf-3442-42e0-b1ee-bf14bd1a4534\") "
Nov 02 03:08:53.499934 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:53.499851    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31830abf-3442-42e0-b1ee-bf14bd1a4534-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31830abf-3442-42e0-b1ee-bf14bd1a4534-maas-api-xks-ca\") pod \"31830abf-3442-42e0-b1ee-bf14bd1a4534\" (UID: \"31830abf-3442-42e0-b1ee-bf14bd1a4534\") "
Nov 02 03:08:53.499988 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:53.499937    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31830abf-3442-42e0-b1ee-bf14bd1a4534-kube-api-access-tktth\" (UniqueName: \"kubernetes.io/projected/31830abf-3442-42e0-b1ee-bf14bd1a4534-kube-api-access-tktth\") pod \"31830abf-3442-42e0-b1ee-bf14bd1a4534\" (UID: \"31830abf-3442-42e0-b1ee-bf14bd1a4534\") "
Nov 02 03:08:53.500120 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:53.500097    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31830abf-3442-42e0-b1ee-bf14bd1a4534-maas-api-openshift-ca" pod "31830abf-3442-42e0-b1ee-bf14bd1a4534" (UID: "31830abf-3442-42e0-b1ee-bf14bd1a4534"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:08:53.500292 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:53.500279    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31830abf-3442-42e0-b1ee-bf14bd1a4534-maas-api-openshift-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:08:53.501857 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:53.501833    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31830abf-3442-42e0-b1ee-bf14bd1a4534-maas-api-xks-ca" pod "31830abf-3442-42e0-b1ee-bf14bd1a4534" (UID: "31830abf-3442-42e0-b1ee-bf14bd1a4534"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 03:08:53.501940 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:53.501905    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31830abf-3442-42e0-b1ee-bf14bd1a4534-kube-api-access-tktth" pod "31830abf-3442-42e0-b1ee-bf14bd1a4534" (UID: "31830abf-3442-42e0-b1ee-bf14bd1a4534"). InnerVolumeSpecName "kube-api-access-tktth". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:08:53.573275 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:53.573238    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14f52df5-6ec4-4647-ac27-1014e2669d66" path="/var/lib/kubelet/pods/14f52df5-6ec4-4647-ac27-1014e2669d66/volumes"
Nov 02 03:08:53.573784 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:53.573765    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8e4bd54-5ae6-49c0-ad5f-786322530e97" path="/var/lib/kubelet/pods/c8e4bd54-5ae6-49c0-ad5f-786322530e97/volumes"
Nov 02 03:08:53.601361 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:53.601306    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tktth\" (UniqueName: \"kubernetes.io/projected/31830abf-3442-42e0-b1ee-bf14bd1a4534-kube-api-access-tktth\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:08:53.601361 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:53.601329    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31830abf-3442-42e0-b1ee-bf14bd1a4534-maas-api-xks-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:08:54.354820 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:54.354790    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-66a47a3b-jl2xl"
Nov 02 03:08:54.354995 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:54.354822    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-66a47a3b-jl2xl" event={"ID":"31830abf-3442-42e0-b1ee-bf14bd1a4534","Type":"ContainerDied","Data":"60a421cd7cff9d5c4b0e8423a7f297ca1d4f79a9ae31b74178413b2db7e9fe5d"}
Nov 02 03:08:54.354995 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:54.354880    4496 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="60a421cd7cff9d5c4b0e8423a7f297ca1d4f79a9ae31b74178413b2db7e9fe5d"
Nov 02 03:08:55.440965 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:55.440930    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bc460382-gjgs8"]
Nov 02 03:08:55.441741 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:55.441720    4496 cpu_manager.go:414] "containerMap: removing container" podUID="c8e4bd54-5ae6-49c0-ad5f-786322530e97" containerName="payload-pre-processing"
Nov 02 03:08:55.441783 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:55.441743    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8e4bd54-5ae6-49c0-ad5f-786322530e97" containerName="payload-pre-processing"
Nov 02 03:08:55.441783 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:55.441761    4496 cpu_manager.go:414] "containerMap: removing container" podUID="14f52df5-6ec4-4647-ac27-1014e2669d66" containerName="payload-processing"
Nov 02 03:08:55.441783 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:55.441769    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14f52df5-6ec4-4647-ac27-1014e2669d66" containerName="payload-processing"
Nov 02 03:08:55.441880 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:55.441796    4496 cpu_manager.go:414] "containerMap: removing container" podUID="31830abf-3442-42e0-b1ee-bf14bd1a4534" containerName="revoke-keys"
Nov 02 03:08:55.441880 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:55.441804    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31830abf-3442-42e0-b1ee-bf14bd1a4534" containerName="revoke-keys"
Nov 02 03:08:55.441937 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:55.441910    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="31830abf-3442-42e0-b1ee-bf14bd1a4534" containerName="revoke-keys"
Nov 02 03:08:55.441937 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:55.441928    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8e4bd54-5ae6-49c0-ad5f-786322530e97" containerName="payload-pre-processing"
Nov 02 03:08:55.441996 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:55.441938    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="14f52df5-6ec4-4647-ac27-1014e2669d66" containerName="payload-processing"
Nov 02 03:08:55.446872 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:55.446854    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bc460382-gjgs8"
Nov 02 03:08:55.500152 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:55.500120    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 03:08:55.515831 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:55.515796    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkjbv\" (UniqueName: \"kubernetes.io/projected/5eb1a9f0-146a-4734-8b94-87f373fe9d50-kube-api-access-xkjbv\") pod \"maas-api-revoke-keys-e2e-derive-bc460382-gjgs8\" (UID: \"5eb1a9f0-146a-4734-8b94-87f373fe9d50\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bc460382-gjgs8"
Nov 02 03:08:55.515989 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:55.515839    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5eb1a9f0-146a-4734-8b94-87f373fe9d50-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-bc460382-gjgs8\" (UID: \"5eb1a9f0-146a-4734-8b94-87f373fe9d50\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bc460382-gjgs8"
Nov 02 03:08:55.516094 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:55.516036    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5eb1a9f0-146a-4734-8b94-87f373fe9d50-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-bc460382-gjgs8\" (UID: \"5eb1a9f0-146a-4734-8b94-87f373fe9d50\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bc460382-gjgs8"
Nov 02 03:08:55.516456 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:55.516436    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 03:08:55.516540 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:55.516495    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 03:08:55.550980 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:55.550951    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9z5bk\""
Nov 02 03:08:55.554260 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:55.554233    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bc460382-gjgs8"]
Nov 02 03:08:55.616879 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:55.616848    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5eb1a9f0-146a-4734-8b94-87f373fe9d50-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-bc460382-gjgs8\" (UID: \"5eb1a9f0-146a-4734-8b94-87f373fe9d50\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bc460382-gjgs8"
Nov 02 03:08:55.617097 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:55.616991    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkjbv\" (UniqueName: \"kubernetes.io/projected/5eb1a9f0-146a-4734-8b94-87f373fe9d50-kube-api-access-xkjbv\") pod \"maas-api-revoke-keys-e2e-derive-bc460382-gjgs8\" (UID: \"5eb1a9f0-146a-4734-8b94-87f373fe9d50\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bc460382-gjgs8"
Nov 02 03:08:55.617097 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:55.617053    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5eb1a9f0-146a-4734-8b94-87f373fe9d50-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-bc460382-gjgs8\" (UID: \"5eb1a9f0-146a-4734-8b94-87f373fe9d50\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bc460382-gjgs8"
Nov 02 03:08:55.617652 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:55.617632    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5eb1a9f0-146a-4734-8b94-87f373fe9d50-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-bc460382-gjgs8\" (UID: \"5eb1a9f0-146a-4734-8b94-87f373fe9d50\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bc460382-gjgs8"
Nov 02 03:08:55.619259 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:55.619239    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5eb1a9f0-146a-4734-8b94-87f373fe9d50-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-bc460382-gjgs8\" (UID: \"5eb1a9f0-146a-4734-8b94-87f373fe9d50\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bc460382-gjgs8"
Nov 02 03:08:55.674211 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:55.674180    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkjbv\" (UniqueName: \"kubernetes.io/projected/5eb1a9f0-146a-4734-8b94-87f373fe9d50-kube-api-access-xkjbv\") pod \"maas-api-revoke-keys-e2e-derive-bc460382-gjgs8\" (UID: \"5eb1a9f0-146a-4734-8b94-87f373fe9d50\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bc460382-gjgs8"
Nov 02 03:08:55.763126 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:55.763095    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bc460382-gjgs8"
Nov 02 03:08:55.875268 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:55.875234    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-935e4abf-7d747dd7d7-gs66f"]
Nov 02 03:08:55.930429 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:55.930401    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-935e4abf-57498b49f5-qkrxc"]
Nov 02 03:08:56.152848 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:56.152814    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bc460382-gjgs8"]
Nov 02 03:08:56.154149 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:08:56.154123    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5eb1a9f0_146a_4734_8b94_87f373fe9d50.slice/crio-ebd72db624bed1562a2338cf036383ba0ab155da0adadfe117343f3a72e12783 WatchSource:0}: Error finding container ebd72db624bed1562a2338cf036383ba0ab155da0adadfe117343f3a72e12783: Status 404 returned error can't find the container with id ebd72db624bed1562a2338cf036383ba0ab155da0adadfe117343f3a72e12783
Nov 02 03:08:56.370551 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:56.370519    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bc460382-gjgs8" event={"ID":"5eb1a9f0-146a-4734-8b94-87f373fe9d50","Type":"ContainerStarted","Data":"ebd72db624bed1562a2338cf036383ba0ab155da0adadfe117343f3a72e12783"}
Nov 02 03:08:56.594809 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:56.594779    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d1107301-vfm7v"]
Nov 02 03:08:56.597242 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:56.597222    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d1107301-vfm7v"]
Nov 02 03:08:57.066361 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:57.066329    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4bbf98be-c5wlc"]
Nov 02 03:08:57.070119 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:57.070103    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4bbf98be-c5wlc"
Nov 02 03:08:57.127444 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:57.127402    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4bbf98be-c5wlc"]
Nov 02 03:08:57.137690 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:57.137665    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6494ffa1-e21e-415a-b29e-9c3e517de1c1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-4bbf98be-c5wlc\" (UID: \"6494ffa1-e21e-415a-b29e-9c3e517de1c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4bbf98be-c5wlc"
Nov 02 03:08:57.137839 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:57.137778    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4d9w\" (UniqueName: \"kubernetes.io/projected/6494ffa1-e21e-415a-b29e-9c3e517de1c1-kube-api-access-r4d9w\") pod \"maas-api-revoke-keys-e2e-ait-4bbf98be-c5wlc\" (UID: \"6494ffa1-e21e-415a-b29e-9c3e517de1c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4bbf98be-c5wlc"
Nov 02 03:08:57.137839 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:57.137810    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6494ffa1-e21e-415a-b29e-9c3e517de1c1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-4bbf98be-c5wlc\" (UID: \"6494ffa1-e21e-415a-b29e-9c3e517de1c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4bbf98be-c5wlc"
Nov 02 03:08:57.238619 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:57.238580    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6494ffa1-e21e-415a-b29e-9c3e517de1c1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-4bbf98be-c5wlc\" (UID: \"6494ffa1-e21e-415a-b29e-9c3e517de1c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4bbf98be-c5wlc"
Nov 02 03:08:57.238777 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:57.238713    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4d9w\" (UniqueName: \"kubernetes.io/projected/6494ffa1-e21e-415a-b29e-9c3e517de1c1-kube-api-access-r4d9w\") pod \"maas-api-revoke-keys-e2e-ait-4bbf98be-c5wlc\" (UID: \"6494ffa1-e21e-415a-b29e-9c3e517de1c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4bbf98be-c5wlc"
Nov 02 03:08:57.238777 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:57.238742    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6494ffa1-e21e-415a-b29e-9c3e517de1c1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-4bbf98be-c5wlc\" (UID: \"6494ffa1-e21e-415a-b29e-9c3e517de1c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4bbf98be-c5wlc"
Nov 02 03:08:57.239531 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:57.239505    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6494ffa1-e21e-415a-b29e-9c3e517de1c1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-4bbf98be-c5wlc\" (UID: \"6494ffa1-e21e-415a-b29e-9c3e517de1c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4bbf98be-c5wlc"
Nov 02 03:08:57.240989 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:57.240959    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6494ffa1-e21e-415a-b29e-9c3e517de1c1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-4bbf98be-c5wlc\" (UID: \"6494ffa1-e21e-415a-b29e-9c3e517de1c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4bbf98be-c5wlc"
Nov 02 03:08:57.296128 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:57.296103    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4d9w\" (UniqueName: \"kubernetes.io/projected/6494ffa1-e21e-415a-b29e-9c3e517de1c1-kube-api-access-r4d9w\") pod \"maas-api-revoke-keys-e2e-ait-4bbf98be-c5wlc\" (UID: \"6494ffa1-e21e-415a-b29e-9c3e517de1c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4bbf98be-c5wlc"
Nov 02 03:08:57.380877 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:57.380845    4496 generic.go:356] "Generic (PLEG): container finished" podID="5eb1a9f0-146a-4734-8b94-87f373fe9d50" containerID="bee095c3a6f485380270328d114cd455f861554ecfe5f44670887fc7ae54bb62" exitCode=0
Nov 02 03:08:57.381044 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:57.380926    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bc460382-gjgs8" event={"ID":"5eb1a9f0-146a-4734-8b94-87f373fe9d50","Type":"ContainerDied","Data":"bee095c3a6f485380270328d114cd455f861554ecfe5f44670887fc7ae54bb62"}
Nov 02 03:08:57.384192 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:57.384172    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4bbf98be-c5wlc"
Nov 02 03:08:57.573333 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:57.573255    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc960cba-6ad5-499f-aabc-a0ff5f8ba51b" path="/var/lib/kubelet/pods/cc960cba-6ad5-499f-aabc-a0ff5f8ba51b/volumes"
Nov 02 03:08:57.758548 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:57.758516    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4bbf98be-c5wlc"]
Nov 02 03:08:57.759174 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:08:57.759148    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6494ffa1_e21e_415a_b29e_9c3e517de1c1.slice/crio-0cf29936f5805441d0d3e73adbd18b1f3a0c959a1dc12d0a31d29ff1f35794a7 WatchSource:0}: Error finding container 0cf29936f5805441d0d3e73adbd18b1f3a0c959a1dc12d0a31d29ff1f35794a7: Status 404 returned error can't find the container with id 0cf29936f5805441d0d3e73adbd18b1f3a0c959a1dc12d0a31d29ff1f35794a7
Nov 02 03:08:58.388325 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:58.388230    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4bbf98be-c5wlc" event={"ID":"6494ffa1-e21e-415a-b29e-9c3e517de1c1","Type":"ContainerStarted","Data":"160ed1f6c3554e1ee4f2718e30785c7beb547a10a313e472f63fca38b33563ed"}
Nov 02 03:08:58.388325 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:58.388294    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4bbf98be-c5wlc" event={"ID":"6494ffa1-e21e-415a-b29e-9c3e517de1c1","Type":"ContainerStarted","Data":"0cf29936f5805441d0d3e73adbd18b1f3a0c959a1dc12d0a31d29ff1f35794a7"}
Nov 02 03:08:58.505394 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:58.505340    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4bbf98be-c5wlc" podStartSLOduration=1.505324855 podStartE2EDuration="1.505324855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:08:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:08:58.503755499 +0000 UTC m=+1279.082362881" watchObservedRunningTime="2026-11-02 03:08:58.505324855 +0000 UTC m=+1279.083932234"
Nov 02 03:08:58.530312 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:58.530291    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bc460382-gjgs8"
Nov 02 03:08:58.552471 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:58.552448    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5eb1a9f0-146a-4734-8b94-87f373fe9d50-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5eb1a9f0-146a-4734-8b94-87f373fe9d50-maas-api-xks-ca\") pod \"5eb1a9f0-146a-4734-8b94-87f373fe9d50\" (UID: \"5eb1a9f0-146a-4734-8b94-87f373fe9d50\") "
Nov 02 03:08:58.552600 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:58.552528    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5eb1a9f0-146a-4734-8b94-87f373fe9d50-kube-api-access-xkjbv\" (UniqueName: \"kubernetes.io/projected/5eb1a9f0-146a-4734-8b94-87f373fe9d50-kube-api-access-xkjbv\") pod \"5eb1a9f0-146a-4734-8b94-87f373fe9d50\" (UID: \"5eb1a9f0-146a-4734-8b94-87f373fe9d50\") "
Nov 02 03:08:58.552600 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:58.552584    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5eb1a9f0-146a-4734-8b94-87f373fe9d50-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5eb1a9f0-146a-4734-8b94-87f373fe9d50-maas-api-openshift-ca\") pod \"5eb1a9f0-146a-4734-8b94-87f373fe9d50\" (UID: \"5eb1a9f0-146a-4734-8b94-87f373fe9d50\") "
Nov 02 03:08:58.552943 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:58.552919    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5eb1a9f0-146a-4734-8b94-87f373fe9d50-maas-api-openshift-ca" pod "5eb1a9f0-146a-4734-8b94-87f373fe9d50" (UID: "5eb1a9f0-146a-4734-8b94-87f373fe9d50"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:08:58.554420 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:58.554399    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5eb1a9f0-146a-4734-8b94-87f373fe9d50-maas-api-xks-ca" pod "5eb1a9f0-146a-4734-8b94-87f373fe9d50" (UID: "5eb1a9f0-146a-4734-8b94-87f373fe9d50"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 03:08:58.554518 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:58.554498    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5eb1a9f0-146a-4734-8b94-87f373fe9d50-kube-api-access-xkjbv" pod "5eb1a9f0-146a-4734-8b94-87f373fe9d50" (UID: "5eb1a9f0-146a-4734-8b94-87f373fe9d50"). InnerVolumeSpecName "kube-api-access-xkjbv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:08:58.654288 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:58.654218    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5eb1a9f0-146a-4734-8b94-87f373fe9d50-maas-api-xks-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:08:58.654465 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:58.654440    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkjbv\" (UniqueName: \"kubernetes.io/projected/5eb1a9f0-146a-4734-8b94-87f373fe9d50-kube-api-access-xkjbv\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:08:58.654563 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:58.654468    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5eb1a9f0-146a-4734-8b94-87f373fe9d50-maas-api-openshift-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:08:59.395039 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:59.394989    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bc460382-gjgs8"
Nov 02 03:08:59.395039 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:59.395029    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bc460382-gjgs8" event={"ID":"5eb1a9f0-146a-4734-8b94-87f373fe9d50","Type":"ContainerDied","Data":"ebd72db624bed1562a2338cf036383ba0ab155da0adadfe117343f3a72e12783"}
Nov 02 03:08:59.395543 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:59.395084    4496 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ebd72db624bed1562a2338cf036383ba0ab155da0adadfe117343f3a72e12783"
Nov 02 03:08:59.396643 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:59.396617    4496 generic.go:356] "Generic (PLEG): container finished" podID="6494ffa1-e21e-415a-b29e-9c3e517de1c1" containerID="160ed1f6c3554e1ee4f2718e30785c7beb547a10a313e472f63fca38b33563ed" exitCode=0
Nov 02 03:08:59.396770 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:59.396695    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4bbf98be-c5wlc" event={"ID":"6494ffa1-e21e-415a-b29e-9c3e517de1c1","Type":"ContainerDied","Data":"160ed1f6c3554e1ee4f2718e30785c7beb547a10a313e472f63fca38b33563ed"}
Nov 02 03:08:59.602545 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:59.602512    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-935e4abf-4hmkn"]
Nov 02 03:08:59.607454 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:08:59.607428    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-935e4abf-4hmkn"]
Nov 02 03:09:00.562664 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:00.562633    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4bbf98be-c5wlc"
Nov 02 03:09:00.569608 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:00.569582    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6494ffa1-e21e-415a-b29e-9c3e517de1c1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6494ffa1-e21e-415a-b29e-9c3e517de1c1-maas-api-xks-ca\") pod \"6494ffa1-e21e-415a-b29e-9c3e517de1c1\" (UID: \"6494ffa1-e21e-415a-b29e-9c3e517de1c1\") "
Nov 02 03:09:00.569726 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:00.569687    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6494ffa1-e21e-415a-b29e-9c3e517de1c1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6494ffa1-e21e-415a-b29e-9c3e517de1c1-maas-api-openshift-ca\") pod \"6494ffa1-e21e-415a-b29e-9c3e517de1c1\" (UID: \"6494ffa1-e21e-415a-b29e-9c3e517de1c1\") "
Nov 02 03:09:00.569767 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:00.569726    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6494ffa1-e21e-415a-b29e-9c3e517de1c1-kube-api-access-r4d9w\" (UniqueName: \"kubernetes.io/projected/6494ffa1-e21e-415a-b29e-9c3e517de1c1-kube-api-access-r4d9w\") pod \"6494ffa1-e21e-415a-b29e-9c3e517de1c1\" (UID: \"6494ffa1-e21e-415a-b29e-9c3e517de1c1\") "
Nov 02 03:09:00.570135 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:00.570097    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6494ffa1-e21e-415a-b29e-9c3e517de1c1-maas-api-openshift-ca" pod "6494ffa1-e21e-415a-b29e-9c3e517de1c1" (UID: "6494ffa1-e21e-415a-b29e-9c3e517de1c1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:09:00.571930 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:00.571906    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6494ffa1-e21e-415a-b29e-9c3e517de1c1-kube-api-access-r4d9w" pod "6494ffa1-e21e-415a-b29e-9c3e517de1c1" (UID: "6494ffa1-e21e-415a-b29e-9c3e517de1c1"). InnerVolumeSpecName "kube-api-access-r4d9w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:09:00.572093 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:00.572075    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6494ffa1-e21e-415a-b29e-9c3e517de1c1-maas-api-xks-ca" pod "6494ffa1-e21e-415a-b29e-9c3e517de1c1" (UID: "6494ffa1-e21e-415a-b29e-9c3e517de1c1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 03:09:00.671092 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:00.670977    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6494ffa1-e21e-415a-b29e-9c3e517de1c1-maas-api-xks-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:09:00.671092 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:00.671046    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6494ffa1-e21e-415a-b29e-9c3e517de1c1-maas-api-openshift-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:09:00.671092 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:00.671067    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4d9w\" (UniqueName: \"kubernetes.io/projected/6494ffa1-e21e-415a-b29e-9c3e517de1c1-kube-api-access-r4d9w\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:09:00.876323 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:00.876259    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-935e4abf-7d747dd7d7-gs66f" podUID="fcc64ce5-28af-4a53-b5f2-6574758ae27c" containerName="payload-processing" containerID="cri-o://564c84dda559e33a6294188b3ba5f42d5b2c5b2581eb6e689a03c0b506ee7819" gracePeriod=25
Nov 02 03:09:00.930892 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:00.930815    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-935e4abf-57498b49f5-qkrxc" podUID="7665e3a4-191c-4de6-b99e-3ae4b4e78b9b" containerName="payload-pre-processing" containerID="cri-o://4f7d9a854fab477ca8c4b8046e812c013146308e2805acb80511aa5ada5b5da9" gracePeriod=25
Nov 02 03:09:01.098060 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:01.097715    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-66a47a3b-686d7444c9-xk4nt"]
Nov 02 03:09:01.192266 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:01.192247    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-935e4abf-57498b49f5-qkrxc"
Nov 02 03:09:01.199512 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:01.199489    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-935e4abf-7d747dd7d7-gs66f"
Nov 02 03:09:01.200550 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:01.200530    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-66a47a3b-699d7d7899-pj48x"]
Nov 02 03:09:01.384787 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:01.384755    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7665e3a4-191c-4de6-b99e-3ae4b4e78b9b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7665e3a4-191c-4de6-b99e-3ae4b4e78b9b-plugins-config-volume\") pod \"7665e3a4-191c-4de6-b99e-3ae4b4e78b9b\" (UID: \"7665e3a4-191c-4de6-b99e-3ae4b4e78b9b\") "
Nov 02 03:09:01.384968 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:01.384799    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcc64ce5-28af-4a53-b5f2-6574758ae27c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcc64ce5-28af-4a53-b5f2-6574758ae27c-plugins-config-volume\") pod \"fcc64ce5-28af-4a53-b5f2-6574758ae27c\" (UID: \"fcc64ce5-28af-4a53-b5f2-6574758ae27c\") "
Nov 02 03:09:01.384968 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:01.384883    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7665e3a4-191c-4de6-b99e-3ae4b4e78b9b-kube-api-access-m6nqp\" (UniqueName: \"kubernetes.io/projected/7665e3a4-191c-4de6-b99e-3ae4b4e78b9b-kube-api-access-m6nqp\") pod \"7665e3a4-191c-4de6-b99e-3ae4b4e78b9b\" (UID: \"7665e3a4-191c-4de6-b99e-3ae4b4e78b9b\") "
Nov 02 03:09:01.384968 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:01.384919    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcc64ce5-28af-4a53-b5f2-6574758ae27c-kube-api-access-lqjjt\" (UniqueName: \"kubernetes.io/projected/fcc64ce5-28af-4a53-b5f2-6574758ae27c-kube-api-access-lqjjt\") pod \"fcc64ce5-28af-4a53-b5f2-6574758ae27c\" (UID: \"fcc64ce5-28af-4a53-b5f2-6574758ae27c\") "
Nov 02 03:09:01.385235 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:01.385210    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7665e3a4-191c-4de6-b99e-3ae4b4e78b9b-plugins-config-volume" pod "7665e3a4-191c-4de6-b99e-3ae4b4e78b9b" (UID: "7665e3a4-191c-4de6-b99e-3ae4b4e78b9b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:09:01.385286 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:01.385213    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcc64ce5-28af-4a53-b5f2-6574758ae27c-plugins-config-volume" pod "fcc64ce5-28af-4a53-b5f2-6574758ae27c" (UID: "fcc64ce5-28af-4a53-b5f2-6574758ae27c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:09:01.387044 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:01.387003    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcc64ce5-28af-4a53-b5f2-6574758ae27c-kube-api-access-lqjjt" pod "fcc64ce5-28af-4a53-b5f2-6574758ae27c" (UID: "fcc64ce5-28af-4a53-b5f2-6574758ae27c"). InnerVolumeSpecName "kube-api-access-lqjjt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:09:01.387135 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:01.387038    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7665e3a4-191c-4de6-b99e-3ae4b4e78b9b-kube-api-access-m6nqp" pod "7665e3a4-191c-4de6-b99e-3ae4b4e78b9b" (UID: "7665e3a4-191c-4de6-b99e-3ae4b4e78b9b"). InnerVolumeSpecName "kube-api-access-m6nqp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:09:01.415354 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:01.415327    4496 generic.go:356] "Generic (PLEG): container finished" podID="7665e3a4-191c-4de6-b99e-3ae4b4e78b9b" containerID="4f7d9a854fab477ca8c4b8046e812c013146308e2805acb80511aa5ada5b5da9" exitCode=0
Nov 02 03:09:01.415499 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:01.415471    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-935e4abf-57498b49f5-qkrxc"
Nov 02 03:09:01.415845 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:01.415823    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-935e4abf-57498b49f5-qkrxc" event={"ID":"7665e3a4-191c-4de6-b99e-3ae4b4e78b9b","Type":"ContainerDied","Data":"4f7d9a854fab477ca8c4b8046e812c013146308e2805acb80511aa5ada5b5da9"}
Nov 02 03:09:01.415920 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:01.415881    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-935e4abf-57498b49f5-qkrxc" event={"ID":"7665e3a4-191c-4de6-b99e-3ae4b4e78b9b","Type":"ContainerDied","Data":"244709b48a32359f21ac87f090ec8b94cbac285ae2c65c025cff8fc79172be0e"}
Nov 02 03:09:01.415920 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:01.415912    4496 scope.go:122] "RemoveContainer" containerID="4f7d9a854fab477ca8c4b8046e812c013146308e2805acb80511aa5ada5b5da9"
Nov 02 03:09:01.421177 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:01.421151    4496 generic.go:356] "Generic (PLEG): container finished" podID="fcc64ce5-28af-4a53-b5f2-6574758ae27c" containerID="564c84dda559e33a6294188b3ba5f42d5b2c5b2581eb6e689a03c0b506ee7819" exitCode=0
Nov 02 03:09:01.421285 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:01.421273    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-935e4abf-7d747dd7d7-gs66f" event={"ID":"fcc64ce5-28af-4a53-b5f2-6574758ae27c","Type":"ContainerDied","Data":"564c84dda559e33a6294188b3ba5f42d5b2c5b2581eb6e689a03c0b506ee7819"}
Nov 02 03:09:01.421336 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:01.421319    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-935e4abf-7d747dd7d7-gs66f" event={"ID":"fcc64ce5-28af-4a53-b5f2-6574758ae27c","Type":"ContainerDied","Data":"d628a4e60cd68e3f5ad3e9755644ae9e569a307937ad332d39ae43e4a4d539fd"}
Nov 02 03:09:01.421416 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:01.421401    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-935e4abf-7d747dd7d7-gs66f"
Nov 02 03:09:01.424723 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:01.424702    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4bbf98be-c5wlc"
Nov 02 03:09:01.424723 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:01.424709    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4bbf98be-c5wlc" event={"ID":"6494ffa1-e21e-415a-b29e-9c3e517de1c1","Type":"ContainerDied","Data":"0cf29936f5805441d0d3e73adbd18b1f3a0c959a1dc12d0a31d29ff1f35794a7"}
Nov 02 03:09:01.424901 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:01.424747    4496 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0cf29936f5805441d0d3e73adbd18b1f3a0c959a1dc12d0a31d29ff1f35794a7"
Nov 02 03:09:01.429854 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:01.429832    4496 scope.go:122] "RemoveContainer" containerID="4f7d9a854fab477ca8c4b8046e812c013146308e2805acb80511aa5ada5b5da9"
Nov 02 03:09:01.430169 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:09:01.430149    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f7d9a854fab477ca8c4b8046e812c013146308e2805acb80511aa5ada5b5da9\": container with ID starting with 4f7d9a854fab477ca8c4b8046e812c013146308e2805acb80511aa5ada5b5da9 not found: ID does not exist" containerID="4f7d9a854fab477ca8c4b8046e812c013146308e2805acb80511aa5ada5b5da9"
Nov 02 03:09:01.430228 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:01.430175    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f7d9a854fab477ca8c4b8046e812c013146308e2805acb80511aa5ada5b5da9"} err="failed to get container status \"4f7d9a854fab477ca8c4b8046e812c013146308e2805acb80511aa5ada5b5da9\": rpc error: code = NotFound desc = could not find container \"4f7d9a854fab477ca8c4b8046e812c013146308e2805acb80511aa5ada5b5da9\": container with ID starting with 4f7d9a854fab477ca8c4b8046e812c013146308e2805acb80511aa5ada5b5da9 not found: ID does not exist"
Nov 02 03:09:01.430228 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:01.430193    4496 scope.go:122] "RemoveContainer" containerID="564c84dda559e33a6294188b3ba5f42d5b2c5b2581eb6e689a03c0b506ee7819"
Nov 02 03:09:01.439195 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:01.439174    4496 scope.go:122] "RemoveContainer" containerID="564c84dda559e33a6294188b3ba5f42d5b2c5b2581eb6e689a03c0b506ee7819"
Nov 02 03:09:01.439509 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:09:01.439491    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"564c84dda559e33a6294188b3ba5f42d5b2c5b2581eb6e689a03c0b506ee7819\": container with ID starting with 564c84dda559e33a6294188b3ba5f42d5b2c5b2581eb6e689a03c0b506ee7819 not found: ID does not exist" containerID="564c84dda559e33a6294188b3ba5f42d5b2c5b2581eb6e689a03c0b506ee7819"
Nov 02 03:09:01.439582 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:01.439514    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"564c84dda559e33a6294188b3ba5f42d5b2c5b2581eb6e689a03c0b506ee7819"} err="failed to get container status \"564c84dda559e33a6294188b3ba5f42d5b2c5b2581eb6e689a03c0b506ee7819\": rpc error: code = NotFound desc = could not find container \"564c84dda559e33a6294188b3ba5f42d5b2c5b2581eb6e689a03c0b506ee7819\": container with ID starting with 564c84dda559e33a6294188b3ba5f42d5b2c5b2581eb6e689a03c0b506ee7819 not found: ID does not exist"
Nov 02 03:09:01.485887 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:01.485864    4496 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcc64ce5-28af-4a53-b5f2-6574758ae27c-plugins-config-volume\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:09:01.485887 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:01.485886    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6nqp\" (UniqueName: \"kubernetes.io/projected/7665e3a4-191c-4de6-b99e-3ae4b4e78b9b-kube-api-access-m6nqp\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:09:01.486025 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:01.485896    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqjjt\" (UniqueName: \"kubernetes.io/projected/fcc64ce5-28af-4a53-b5f2-6574758ae27c-kube-api-access-lqjjt\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:09:01.486025 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:01.485905    4496 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7665e3a4-191c-4de6-b99e-3ae4b4e78b9b-plugins-config-volume\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:09:01.575663 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:01.575620    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6199019-fe2d-4d93-badd-0b38509b0dcd" path="/var/lib/kubelet/pods/a6199019-fe2d-4d93-badd-0b38509b0dcd/volumes"
Nov 02 03:09:01.677223 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:01.677189    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-935e4abf-57498b49f5-qkrxc"]
Nov 02 03:09:01.682427 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:01.682397    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-935e4abf-57498b49f5-qkrxc"]
Nov 02 03:09:01.794854 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:01.794807    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-935e4abf-7d747dd7d7-gs66f"]
Nov 02 03:09:01.800718 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:01.800681    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-935e4abf-7d747dd7d7-gs66f"]
Nov 02 03:09:03.526448 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:09:03.526416    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 03:09:03.579111 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:03.579077    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7665e3a4-191c-4de6-b99e-3ae4b4e78b9b" path="/var/lib/kubelet/pods/7665e3a4-191c-4de6-b99e-3ae4b4e78b9b/volumes"
Nov 02 03:09:03.579500 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:03.579486    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fcc64ce5-28af-4a53-b5f2-6574758ae27c" path="/var/lib/kubelet/pods/fcc64ce5-28af-4a53-b5f2-6574758ae27c/volumes"
Nov 02 03:09:06.098915 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:06.098870    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-66a47a3b-686d7444c9-xk4nt" podUID="46896864-f510-4a9f-a225-376ebc87cdf5" containerName="payload-processing" containerID="cri-o://94764994f6a4803565cba22fd4303114f94c4294dd28699f23eb7ae9202d3602" gracePeriod=25
Nov 02 03:09:06.201602 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:06.201564    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-66a47a3b-699d7d7899-pj48x" podUID="c8f885d3-8814-43fc-9e10-2bfc8e614899" containerName="payload-pre-processing" containerID="cri-o://f0836346ff6499159381cdd2079dd21b082b638da60930a84d85cbafd457948d" gracePeriod=25
Nov 02 03:09:06.348707 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:06.348680    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-66a47a3b-686d7444c9-xk4nt"
Nov 02 03:09:06.433074 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:06.433045    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46896864-f510-4a9f-a225-376ebc87cdf5-kube-api-access-279xw\" (UniqueName: \"kubernetes.io/projected/46896864-f510-4a9f-a225-376ebc87cdf5-kube-api-access-279xw\") pod \"46896864-f510-4a9f-a225-376ebc87cdf5\" (UID: \"46896864-f510-4a9f-a225-376ebc87cdf5\") "
Nov 02 03:09:06.433247 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:06.433170    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46896864-f510-4a9f-a225-376ebc87cdf5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46896864-f510-4a9f-a225-376ebc87cdf5-plugins-config-volume\") pod \"46896864-f510-4a9f-a225-376ebc87cdf5\" (UID: \"46896864-f510-4a9f-a225-376ebc87cdf5\") "
Nov 02 03:09:06.433531 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:06.433502    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46896864-f510-4a9f-a225-376ebc87cdf5-plugins-config-volume" pod "46896864-f510-4a9f-a225-376ebc87cdf5" (UID: "46896864-f510-4a9f-a225-376ebc87cdf5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:09:06.435049 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:06.435026    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46896864-f510-4a9f-a225-376ebc87cdf5-kube-api-access-279xw" pod "46896864-f510-4a9f-a225-376ebc87cdf5" (UID: "46896864-f510-4a9f-a225-376ebc87cdf5"). InnerVolumeSpecName "kube-api-access-279xw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:09:06.451920 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:06.451898    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-66a47a3b-699d7d7899-pj48x"
Nov 02 03:09:06.458714 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:06.458688    4496 generic.go:356] "Generic (PLEG): container finished" podID="46896864-f510-4a9f-a225-376ebc87cdf5" containerID="94764994f6a4803565cba22fd4303114f94c4294dd28699f23eb7ae9202d3602" exitCode=0
Nov 02 03:09:06.458816 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:06.458736    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-66a47a3b-686d7444c9-xk4nt"
Nov 02 03:09:06.458816 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:06.458773    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-66a47a3b-686d7444c9-xk4nt" event={"ID":"46896864-f510-4a9f-a225-376ebc87cdf5","Type":"ContainerDied","Data":"94764994f6a4803565cba22fd4303114f94c4294dd28699f23eb7ae9202d3602"}
Nov 02 03:09:06.458913 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:06.458820    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-66a47a3b-686d7444c9-xk4nt" event={"ID":"46896864-f510-4a9f-a225-376ebc87cdf5","Type":"ContainerDied","Data":"aef76a44298e1720175cc0f438b5e79e3d920eafeef7a3c7e500ae72c6db76a8"}
Nov 02 03:09:06.458913 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:06.458844    4496 scope.go:122] "RemoveContainer" containerID="94764994f6a4803565cba22fd4303114f94c4294dd28699f23eb7ae9202d3602"
Nov 02 03:09:06.460434 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:06.460409    4496 generic.go:356] "Generic (PLEG): container finished" podID="c8f885d3-8814-43fc-9e10-2bfc8e614899" containerID="f0836346ff6499159381cdd2079dd21b082b638da60930a84d85cbafd457948d" exitCode=0
Nov 02 03:09:06.460527 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:06.460469    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-66a47a3b-699d7d7899-pj48x"
Nov 02 03:09:06.460527 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:06.460494    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-66a47a3b-699d7d7899-pj48x" event={"ID":"c8f885d3-8814-43fc-9e10-2bfc8e614899","Type":"ContainerDied","Data":"f0836346ff6499159381cdd2079dd21b082b638da60930a84d85cbafd457948d"}
Nov 02 03:09:06.460641 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:06.460554    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-66a47a3b-699d7d7899-pj48x" event={"ID":"c8f885d3-8814-43fc-9e10-2bfc8e614899","Type":"ContainerDied","Data":"d6ad5735699cd3dcdfd0092a660e34e7820f7ba7e04bfadbe063dd1ef60bd5c9"}
Nov 02 03:09:06.473612 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:06.473594    4496 scope.go:122] "RemoveContainer" containerID="94764994f6a4803565cba22fd4303114f94c4294dd28699f23eb7ae9202d3602"
Nov 02 03:09:06.473896 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:09:06.473878    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94764994f6a4803565cba22fd4303114f94c4294dd28699f23eb7ae9202d3602\": container with ID starting with 94764994f6a4803565cba22fd4303114f94c4294dd28699f23eb7ae9202d3602 not found: ID does not exist" containerID="94764994f6a4803565cba22fd4303114f94c4294dd28699f23eb7ae9202d3602"
Nov 02 03:09:06.473961 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:06.473906    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94764994f6a4803565cba22fd4303114f94c4294dd28699f23eb7ae9202d3602"} err="failed to get container status \"94764994f6a4803565cba22fd4303114f94c4294dd28699f23eb7ae9202d3602\": rpc error: code = NotFound desc = could not find container \"94764994f6a4803565cba22fd4303114f94c4294dd28699f23eb7ae9202d3602\": container with ID starting with 94764994f6a4803565cba22fd4303114f94c4294dd28699f23eb7ae9202d3602 not found: ID does not exist"
Nov 02 03:09:06.473961 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:06.473923    4496 scope.go:122] "RemoveContainer" containerID="f0836346ff6499159381cdd2079dd21b082b638da60930a84d85cbafd457948d"
Nov 02 03:09:06.484457 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:06.484400    4496 scope.go:122] "RemoveContainer" containerID="f0836346ff6499159381cdd2079dd21b082b638da60930a84d85cbafd457948d"
Nov 02 03:09:06.484735 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:09:06.484713    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0836346ff6499159381cdd2079dd21b082b638da60930a84d85cbafd457948d\": container with ID starting with f0836346ff6499159381cdd2079dd21b082b638da60930a84d85cbafd457948d not found: ID does not exist" containerID="f0836346ff6499159381cdd2079dd21b082b638da60930a84d85cbafd457948d"
Nov 02 03:09:06.484830 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:06.484740    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0836346ff6499159381cdd2079dd21b082b638da60930a84d85cbafd457948d"} err="failed to get container status \"f0836346ff6499159381cdd2079dd21b082b638da60930a84d85cbafd457948d\": rpc error: code = NotFound desc = could not find container \"f0836346ff6499159381cdd2079dd21b082b638da60930a84d85cbafd457948d\": container with ID starting with f0836346ff6499159381cdd2079dd21b082b638da60930a84d85cbafd457948d not found: ID does not exist"
Nov 02 03:09:06.534066 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:06.534036    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8f885d3-8814-43fc-9e10-2bfc8e614899-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8f885d3-8814-43fc-9e10-2bfc8e614899-plugins-config-volume\") pod \"c8f885d3-8814-43fc-9e10-2bfc8e614899\" (UID: \"c8f885d3-8814-43fc-9e10-2bfc8e614899\") "
Nov 02 03:09:06.534188 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:06.534085    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8f885d3-8814-43fc-9e10-2bfc8e614899-kube-api-access-zpppk\" (UniqueName: \"kubernetes.io/projected/c8f885d3-8814-43fc-9e10-2bfc8e614899-kube-api-access-zpppk\") pod \"c8f885d3-8814-43fc-9e10-2bfc8e614899\" (UID: \"c8f885d3-8814-43fc-9e10-2bfc8e614899\") "
Nov 02 03:09:06.534338 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:06.534320    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8f885d3-8814-43fc-9e10-2bfc8e614899-plugins-config-volume" pod "c8f885d3-8814-43fc-9e10-2bfc8e614899" (UID: "c8f885d3-8814-43fc-9e10-2bfc8e614899"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:09:06.534490 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:06.534471    4496 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46896864-f510-4a9f-a225-376ebc87cdf5-plugins-config-volume\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:09:06.534529 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:06.534493    4496 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8f885d3-8814-43fc-9e10-2bfc8e614899-plugins-config-volume\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:09:06.534529 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:06.534502    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-279xw\" (UniqueName: \"kubernetes.io/projected/46896864-f510-4a9f-a225-376ebc87cdf5-kube-api-access-279xw\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:09:06.536001 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:06.535985    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8f885d3-8814-43fc-9e10-2bfc8e614899-kube-api-access-zpppk" pod "c8f885d3-8814-43fc-9e10-2bfc8e614899" (UID: "c8f885d3-8814-43fc-9e10-2bfc8e614899"). InnerVolumeSpecName "kube-api-access-zpppk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:09:06.635188 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:06.635110    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpppk\" (UniqueName: \"kubernetes.io/projected/c8f885d3-8814-43fc-9e10-2bfc8e614899-kube-api-access-zpppk\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:09:06.751303 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:06.751271    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-66a47a3b-686d7444c9-xk4nt"]
Nov 02 03:09:06.755843 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:06.755815    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-66a47a3b-686d7444c9-xk4nt"]
Nov 02 03:09:06.786842 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:06.786812    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-66a47a3b-jl2xl"]
Nov 02 03:09:06.802256 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:06.802227    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-66a47a3b-jl2xl"]
Nov 02 03:09:06.895320 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:06.895238    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-66a47a3b-699d7d7899-pj48x"]
Nov 02 03:09:06.901091 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:06.901061    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-66a47a3b-699d7d7899-pj48x"]
Nov 02 03:09:07.065919 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:07.065882    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-ca6546-76c6c456c9-vxsgp"]
Nov 02 03:09:07.066429 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:07.066412    4496 cpu_manager.go:414] "containerMap: removing container" podUID="6494ffa1-e21e-415a-b29e-9c3e517de1c1" containerName="revoke-keys"
Nov 02 03:09:07.066537 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:07.066430    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6494ffa1-e21e-415a-b29e-9c3e517de1c1" containerName="revoke-keys"
Nov 02 03:09:07.066537 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:07.066449    4496 cpu_manager.go:414] "containerMap: removing container" podUID="fcc64ce5-28af-4a53-b5f2-6574758ae27c" containerName="payload-processing"
Nov 02 03:09:07.066537 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:07.066456    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcc64ce5-28af-4a53-b5f2-6574758ae27c" containerName="payload-processing"
Nov 02 03:09:07.066537 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:07.066473    4496 cpu_manager.go:414] "containerMap: removing container" podUID="7665e3a4-191c-4de6-b99e-3ae4b4e78b9b" containerName="payload-pre-processing"
Nov 02 03:09:07.066537 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:07.066480    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7665e3a4-191c-4de6-b99e-3ae4b4e78b9b" containerName="payload-pre-processing"
Nov 02 03:09:07.066537 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:07.066500    4496 cpu_manager.go:414] "containerMap: removing container" podUID="c8f885d3-8814-43fc-9e10-2bfc8e614899" containerName="payload-pre-processing"
Nov 02 03:09:07.066537 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:07.066508    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8f885d3-8814-43fc-9e10-2bfc8e614899" containerName="payload-pre-processing"
Nov 02 03:09:07.066537 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:07.066517    4496 cpu_manager.go:414] "containerMap: removing container" podUID="5eb1a9f0-146a-4734-8b94-87f373fe9d50" containerName="revoke-keys"
Nov 02 03:09:07.066537 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:07.066524    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5eb1a9f0-146a-4734-8b94-87f373fe9d50" containerName="revoke-keys"
Nov 02 03:09:07.066955 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:07.066551    4496 cpu_manager.go:414] "containerMap: removing container" podUID="46896864-f510-4a9f-a225-376ebc87cdf5" containerName="payload-processing"
Nov 02 03:09:07.066955 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:07.066560    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46896864-f510-4a9f-a225-376ebc87cdf5" containerName="payload-processing"
Nov 02 03:09:07.066955 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:07.066668    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="46896864-f510-4a9f-a225-376ebc87cdf5" containerName="payload-processing"
Nov 02 03:09:07.066955 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:07.066686    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="6494ffa1-e21e-415a-b29e-9c3e517de1c1" containerName="revoke-keys"
Nov 02 03:09:07.066955 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:07.066695    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcc64ce5-28af-4a53-b5f2-6574758ae27c" containerName="payload-processing"
Nov 02 03:09:07.066955 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:07.066706    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="7665e3a4-191c-4de6-b99e-3ae4b4e78b9b" containerName="payload-pre-processing"
Nov 02 03:09:07.066955 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:07.066717    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8f885d3-8814-43fc-9e10-2bfc8e614899" containerName="payload-pre-processing"
Nov 02 03:09:07.066955 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:07.066727    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="5eb1a9f0-146a-4734-8b94-87f373fe9d50" containerName="revoke-keys"
Nov 02 03:09:07.071596 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:07.071574    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ca6546-76c6c456c9-vxsgp"
Nov 02 03:09:07.125736 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:07.125003    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-ca6546\""
Nov 02 03:09:07.125736 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:07.125700    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-ca6546-dockercfg-fwlhx\""
Nov 02 03:09:07.127686 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:07.127650    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-ca6546-76c6c456c9-vxsgp"]
Nov 02 03:09:07.153733 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:07.152744    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57fc7cac-886f-4b70-86ee-6f97a711531a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-ca6546-76c6c456c9-vxsgp\" (UID: \"57fc7cac-886f-4b70-86ee-6f97a711531a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ca6546-76c6c456c9-vxsgp"
Nov 02 03:09:07.153733 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:07.152904    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzrgb\" (UniqueName: \"kubernetes.io/projected/57fc7cac-886f-4b70-86ee-6f97a711531a-kube-api-access-kzrgb\") pod \"payload-pre-processing-e2e-worker-w2-ca6546-76c6c456c9-vxsgp\" (UID: \"57fc7cac-886f-4b70-86ee-6f97a711531a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ca6546-76c6c456c9-vxsgp"
Nov 02 03:09:07.227537 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:07.227505    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-ca6546-8d886cbbf-qx4dh"]
Nov 02 03:09:07.232255 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:07.232232    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-ca6546-8d886cbbf-qx4dh"
Nov 02 03:09:07.254033 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:07.253994    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzrgb\" (UniqueName: \"kubernetes.io/projected/57fc7cac-886f-4b70-86ee-6f97a711531a-kube-api-access-kzrgb\") pod \"payload-pre-processing-e2e-worker-w2-ca6546-76c6c456c9-vxsgp\" (UID: \"57fc7cac-886f-4b70-86ee-6f97a711531a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ca6546-76c6c456c9-vxsgp"
Nov 02 03:09:07.254160 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:07.254094    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57fc7cac-886f-4b70-86ee-6f97a711531a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-ca6546-76c6c456c9-vxsgp\" (UID: \"57fc7cac-886f-4b70-86ee-6f97a711531a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ca6546-76c6c456c9-vxsgp"
Nov 02 03:09:07.261120 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:07.255320    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57fc7cac-886f-4b70-86ee-6f97a711531a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-ca6546-76c6c456c9-vxsgp\" (UID: \"57fc7cac-886f-4b70-86ee-6f97a711531a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ca6546-76c6c456c9-vxsgp"
Nov 02 03:09:07.294471 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:07.294277    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-ca6546-8d886cbbf-qx4dh"]
Nov 02 03:09:07.312804 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:07.312765    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzrgb\" (UniqueName: \"kubernetes.io/projected/57fc7cac-886f-4b70-86ee-6f97a711531a-kube-api-access-kzrgb\") pod \"payload-pre-processing-e2e-worker-w2-ca6546-76c6c456c9-vxsgp\" (UID: \"57fc7cac-886f-4b70-86ee-6f97a711531a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ca6546-76c6c456c9-vxsgp"
Nov 02 03:09:07.355591 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:07.355550    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8kdw\" (UniqueName: \"kubernetes.io/projected/0fabbfdc-768a-49d7-a62d-2bf8680a5cc8-kube-api-access-q8kdw\") pod \"payload-processing-e2e-worker-w2-ca6546-8d886cbbf-qx4dh\" (UID: \"0fabbfdc-768a-49d7-a62d-2bf8680a5cc8\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-ca6546-8d886cbbf-qx4dh"
Nov 02 03:09:07.355742 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:07.355606    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fabbfdc-768a-49d7-a62d-2bf8680a5cc8-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-ca6546-8d886cbbf-qx4dh\" (UID: \"0fabbfdc-768a-49d7-a62d-2bf8680a5cc8\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-ca6546-8d886cbbf-qx4dh"
Nov 02 03:09:07.387004 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:07.386976    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ca6546-76c6c456c9-vxsgp"
Nov 02 03:09:07.467208 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:07.466609    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fabbfdc-768a-49d7-a62d-2bf8680a5cc8-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-ca6546-8d886cbbf-qx4dh\" (UID: \"0fabbfdc-768a-49d7-a62d-2bf8680a5cc8\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-ca6546-8d886cbbf-qx4dh"
Nov 02 03:09:07.467553 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:07.467526    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fabbfdc-768a-49d7-a62d-2bf8680a5cc8-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-ca6546-8d886cbbf-qx4dh\" (UID: \"0fabbfdc-768a-49d7-a62d-2bf8680a5cc8\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-ca6546-8d886cbbf-qx4dh"
Nov 02 03:09:07.467740 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:07.467716    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8kdw\" (UniqueName: \"kubernetes.io/projected/0fabbfdc-768a-49d7-a62d-2bf8680a5cc8-kube-api-access-q8kdw\") pod \"payload-processing-e2e-worker-w2-ca6546-8d886cbbf-qx4dh\" (UID: \"0fabbfdc-768a-49d7-a62d-2bf8680a5cc8\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-ca6546-8d886cbbf-qx4dh"
Nov 02 03:09:07.526306 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:07.526274    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8kdw\" (UniqueName: \"kubernetes.io/projected/0fabbfdc-768a-49d7-a62d-2bf8680a5cc8-kube-api-access-q8kdw\") pod \"payload-processing-e2e-worker-w2-ca6546-8d886cbbf-qx4dh\" (UID: \"0fabbfdc-768a-49d7-a62d-2bf8680a5cc8\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-ca6546-8d886cbbf-qx4dh"
Nov 02 03:09:07.545851 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:07.545826    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-ca6546-8d886cbbf-qx4dh"
Nov 02 03:09:07.573769 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:07.573740    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31830abf-3442-42e0-b1ee-bf14bd1a4534" path="/var/lib/kubelet/pods/31830abf-3442-42e0-b1ee-bf14bd1a4534/volumes"
Nov 02 03:09:07.574208 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:07.574189    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46896864-f510-4a9f-a225-376ebc87cdf5" path="/var/lib/kubelet/pods/46896864-f510-4a9f-a225-376ebc87cdf5/volumes"
Nov 02 03:09:07.574587 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:07.574569    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8f885d3-8814-43fc-9e10-2bfc8e614899" path="/var/lib/kubelet/pods/c8f885d3-8814-43fc-9e10-2bfc8e614899/volumes"
Nov 02 03:09:07.762540 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:07.762512    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-ca6546-76c6c456c9-vxsgp"]
Nov 02 03:09:07.763272 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:09:07.763236    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57fc7cac_886f_4b70_86ee_6f97a711531a.slice/crio-355642174b8498c4b63bfe08b5611c64ba1bfe6e8a84318d97eace49b90eda4b WatchSource:0}: Error finding container 355642174b8498c4b63bfe08b5611c64ba1bfe6e8a84318d97eace49b90eda4b: Status 404 returned error can't find the container with id 355642174b8498c4b63bfe08b5611c64ba1bfe6e8a84318d97eace49b90eda4b
Nov 02 03:09:07.925099 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:09:07.925069    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fabbfdc_768a_49d7_a62d_2bf8680a5cc8.slice/crio-a2b82b13393601331dbf1bba50e8af82c80b6526f2ca90a3ad556cf949b46190 WatchSource:0}: Error finding container a2b82b13393601331dbf1bba50e8af82c80b6526f2ca90a3ad556cf949b46190: Status 404 returned error can't find the container with id a2b82b13393601331dbf1bba50e8af82c80b6526f2ca90a3ad556cf949b46190
Nov 02 03:09:07.925599 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:07.925575    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-ca6546-8d886cbbf-qx4dh"]
Nov 02 03:09:08.483668 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:08.483630    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-ca6546-8d886cbbf-qx4dh" event={"ID":"0fabbfdc-768a-49d7-a62d-2bf8680a5cc8","Type":"ContainerStarted","Data":"541e7fab7b4167b8ccec28cd967e41b7bb3ae7c1e42a7e04d4b97d385f1b8fa4"}
Nov 02 03:09:08.484141 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:08.483690    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-ca6546-8d886cbbf-qx4dh" event={"ID":"0fabbfdc-768a-49d7-a62d-2bf8680a5cc8","Type":"ContainerStarted","Data":"a2b82b13393601331dbf1bba50e8af82c80b6526f2ca90a3ad556cf949b46190"}
Nov 02 03:09:08.484141 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:08.483850    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-ca6546-8d886cbbf-qx4dh"
Nov 02 03:09:08.485391 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:08.485370    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ca6546-76c6c456c9-vxsgp" event={"ID":"57fc7cac-886f-4b70-86ee-6f97a711531a","Type":"ContainerStarted","Data":"8bb2eb3c9f6d6d1bd9c14597c4210682e67a1cf29c335b66ca3fede9c3561db2"}
Nov 02 03:09:08.485455 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:08.485410    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ca6546-76c6c456c9-vxsgp" event={"ID":"57fc7cac-886f-4b70-86ee-6f97a711531a","Type":"ContainerStarted","Data":"355642174b8498c4b63bfe08b5611c64ba1bfe6e8a84318d97eace49b90eda4b"}
Nov 02 03:09:08.485511 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:08.485454    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ca6546-76c6c456c9-vxsgp"
Nov 02 03:09:08.602363 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:08.602313    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-ca6546-8d886cbbf-qx4dh" podStartSLOduration=1.60230272 podStartE2EDuration="1.60230272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:09:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:09:08.599883722 +0000 UTC m=+1289.178491103" watchObservedRunningTime="2026-11-02 03:09:08.60230272 +0000 UTC m=+1289.180910100"
Nov 02 03:09:08.718279 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:08.718197    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ca6546-76c6c456c9-vxsgp" podStartSLOduration=1.718177375 podStartE2EDuration="1.718177375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:09:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:09:08.71585348 +0000 UTC m=+1289.294460860" watchObservedRunningTime="2026-11-02 03:09:08.718177375 +0000 UTC m=+1289.296784755"
Nov 02 03:09:10.058213 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:10.058180    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-bc460382-7c7dd66f5-68kpj"]
Nov 02 03:09:10.169265 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:10.169233    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-bc460382-7759547547-vgh7q"]
Nov 02 03:09:12.482239 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:12.482204    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4bbf98be-57fd77bc87-5lzs4"]
Nov 02 03:09:12.586724 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:12.586696    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4bbf98be-b9594874c-9fqx9"]
Nov 02 03:09:15.059497 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:15.059449    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-bc460382-7c7dd66f5-68kpj" podUID="a22f346f-bf44-4fdf-adc4-1559afe4620d" containerName="payload-processing" containerID="cri-o://fce27c407f322e39ab28b3a849111442778a548e5b28bb8af95a4dee2c233c88" gracePeriod=25
Nov 02 03:09:15.169676 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:15.169624    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-bc460382-7759547547-vgh7q" podUID="7adde223-834d-401b-98fe-22b635e77a16" containerName="payload-pre-processing" containerID="cri-o://355712ce7ce07fa21ec659556706f959b22cf5bf9406e90d8eb46ccdd53b8d4c" gracePeriod=25
Nov 02 03:09:15.314742 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:15.314681    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-bc460382-7c7dd66f5-68kpj"
Nov 02 03:09:15.437242 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:15.437215    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-bc460382-7759547547-vgh7q"
Nov 02 03:09:15.452383 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:15.452347    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a22f346f-bf44-4fdf-adc4-1559afe4620d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a22f346f-bf44-4fdf-adc4-1559afe4620d-plugins-config-volume\") pod \"a22f346f-bf44-4fdf-adc4-1559afe4620d\" (UID: \"a22f346f-bf44-4fdf-adc4-1559afe4620d\") "
Nov 02 03:09:15.452541 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:15.452437    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a22f346f-bf44-4fdf-adc4-1559afe4620d-kube-api-access-zbbd8\" (UniqueName: \"kubernetes.io/projected/a22f346f-bf44-4fdf-adc4-1559afe4620d-kube-api-access-zbbd8\") pod \"a22f346f-bf44-4fdf-adc4-1559afe4620d\" (UID: \"a22f346f-bf44-4fdf-adc4-1559afe4620d\") "
Nov 02 03:09:15.452735 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:15.452712    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a22f346f-bf44-4fdf-adc4-1559afe4620d-plugins-config-volume" pod "a22f346f-bf44-4fdf-adc4-1559afe4620d" (UID: "a22f346f-bf44-4fdf-adc4-1559afe4620d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:09:15.452848 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:15.452832    4496 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a22f346f-bf44-4fdf-adc4-1559afe4620d-plugins-config-volume\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:09:15.454541 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:15.454512    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a22f346f-bf44-4fdf-adc4-1559afe4620d-kube-api-access-zbbd8" pod "a22f346f-bf44-4fdf-adc4-1559afe4620d" (UID: "a22f346f-bf44-4fdf-adc4-1559afe4620d"). InnerVolumeSpecName "kube-api-access-zbbd8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:09:15.533756 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:15.533724    4496 generic.go:356] "Generic (PLEG): container finished" podID="a22f346f-bf44-4fdf-adc4-1559afe4620d" containerID="fce27c407f322e39ab28b3a849111442778a548e5b28bb8af95a4dee2c233c88" exitCode=0
Nov 02 03:09:15.533926 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:15.533793    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-bc460382-7c7dd66f5-68kpj"
Nov 02 03:09:15.533926 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:15.533815    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-bc460382-7c7dd66f5-68kpj" event={"ID":"a22f346f-bf44-4fdf-adc4-1559afe4620d","Type":"ContainerDied","Data":"fce27c407f322e39ab28b3a849111442778a548e5b28bb8af95a4dee2c233c88"}
Nov 02 03:09:15.533926 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:15.533878    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-bc460382-7c7dd66f5-68kpj" event={"ID":"a22f346f-bf44-4fdf-adc4-1559afe4620d","Type":"ContainerDied","Data":"0245b4ddc804f2089e30800e2b91238d66dcf2e46c8a869798f19e3e6493fae8"}
Nov 02 03:09:15.533926 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:15.533907    4496 scope.go:122] "RemoveContainer" containerID="fce27c407f322e39ab28b3a849111442778a548e5b28bb8af95a4dee2c233c88"
Nov 02 03:09:15.535400 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:15.535379    4496 generic.go:356] "Generic (PLEG): container finished" podID="7adde223-834d-401b-98fe-22b635e77a16" containerID="355712ce7ce07fa21ec659556706f959b22cf5bf9406e90d8eb46ccdd53b8d4c" exitCode=0
Nov 02 03:09:15.535535 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:15.535428    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-bc460382-7759547547-vgh7q"
Nov 02 03:09:15.535535 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:15.535453    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-bc460382-7759547547-vgh7q" event={"ID":"7adde223-834d-401b-98fe-22b635e77a16","Type":"ContainerDied","Data":"355712ce7ce07fa21ec659556706f959b22cf5bf9406e90d8eb46ccdd53b8d4c"}
Nov 02 03:09:15.535635 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:15.535536    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-bc460382-7759547547-vgh7q" event={"ID":"7adde223-834d-401b-98fe-22b635e77a16","Type":"ContainerDied","Data":"77eaaed4c4c149620aa0d50514ad82b506197399db4eee86d969407acd3ed2a9"}
Nov 02 03:09:15.544243 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:15.544148    4496 scope.go:122] "RemoveContainer" containerID="fce27c407f322e39ab28b3a849111442778a548e5b28bb8af95a4dee2c233c88"
Nov 02 03:09:15.544458 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:09:15.544429    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fce27c407f322e39ab28b3a849111442778a548e5b28bb8af95a4dee2c233c88\": container with ID starting with fce27c407f322e39ab28b3a849111442778a548e5b28bb8af95a4dee2c233c88 not found: ID does not exist" containerID="fce27c407f322e39ab28b3a849111442778a548e5b28bb8af95a4dee2c233c88"
Nov 02 03:09:15.544569 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:15.544468    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fce27c407f322e39ab28b3a849111442778a548e5b28bb8af95a4dee2c233c88"} err="failed to get container status \"fce27c407f322e39ab28b3a849111442778a548e5b28bb8af95a4dee2c233c88\": rpc error: code = NotFound desc = could not find container \"fce27c407f322e39ab28b3a849111442778a548e5b28bb8af95a4dee2c233c88\": container with ID starting with fce27c407f322e39ab28b3a849111442778a548e5b28bb8af95a4dee2c233c88 not found: ID does not exist"
Nov 02 03:09:15.544569 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:15.544487    4496 scope.go:122] "RemoveContainer" containerID="355712ce7ce07fa21ec659556706f959b22cf5bf9406e90d8eb46ccdd53b8d4c"
Nov 02 03:09:15.553280 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:15.553255    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7adde223-834d-401b-98fe-22b635e77a16-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7adde223-834d-401b-98fe-22b635e77a16-plugins-config-volume\") pod \"7adde223-834d-401b-98fe-22b635e77a16\" (UID: \"7adde223-834d-401b-98fe-22b635e77a16\") "
Nov 02 03:09:15.553396 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:15.553328    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7adde223-834d-401b-98fe-22b635e77a16-kube-api-access-j46dl\" (UniqueName: \"kubernetes.io/projected/7adde223-834d-401b-98fe-22b635e77a16-kube-api-access-j46dl\") pod \"7adde223-834d-401b-98fe-22b635e77a16\" (UID: \"7adde223-834d-401b-98fe-22b635e77a16\") "
Nov 02 03:09:15.553841 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:15.553638    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7adde223-834d-401b-98fe-22b635e77a16-plugins-config-volume" pod "7adde223-834d-401b-98fe-22b635e77a16" (UID: "7adde223-834d-401b-98fe-22b635e77a16"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:09:15.553841 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:15.553693    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbbd8\" (UniqueName: \"kubernetes.io/projected/a22f346f-bf44-4fdf-adc4-1559afe4620d-kube-api-access-zbbd8\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:09:15.555253 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:15.555196    4496 scope.go:122] "RemoveContainer" containerID="355712ce7ce07fa21ec659556706f959b22cf5bf9406e90d8eb46ccdd53b8d4c"
Nov 02 03:09:15.556170 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:15.555904    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7adde223-834d-401b-98fe-22b635e77a16-kube-api-access-j46dl" pod "7adde223-834d-401b-98fe-22b635e77a16" (UID: "7adde223-834d-401b-98fe-22b635e77a16"). InnerVolumeSpecName "kube-api-access-j46dl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:09:15.556170 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:09:15.556127    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"355712ce7ce07fa21ec659556706f959b22cf5bf9406e90d8eb46ccdd53b8d4c\": container with ID starting with 355712ce7ce07fa21ec659556706f959b22cf5bf9406e90d8eb46ccdd53b8d4c not found: ID does not exist" containerID="355712ce7ce07fa21ec659556706f959b22cf5bf9406e90d8eb46ccdd53b8d4c"
Nov 02 03:09:15.556170 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:15.556159    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"355712ce7ce07fa21ec659556706f959b22cf5bf9406e90d8eb46ccdd53b8d4c"} err="failed to get container status \"355712ce7ce07fa21ec659556706f959b22cf5bf9406e90d8eb46ccdd53b8d4c\": rpc error: code = NotFound desc = could not find container \"355712ce7ce07fa21ec659556706f959b22cf5bf9406e90d8eb46ccdd53b8d4c\": container with ID starting with 355712ce7ce07fa21ec659556706f959b22cf5bf9406e90d8eb46ccdd53b8d4c not found: ID does not exist"
Nov 02 03:09:15.654730 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:15.654697    4496 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7adde223-834d-401b-98fe-22b635e77a16-plugins-config-volume\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:09:15.654730 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:15.654725    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j46dl\" (UniqueName: \"kubernetes.io/projected/7adde223-834d-401b-98fe-22b635e77a16-kube-api-access-j46dl\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:09:15.718174 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:15.718143    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-bc460382-7c7dd66f5-68kpj"]
Nov 02 03:09:15.722565 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:15.722541    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-bc460382-7c7dd66f5-68kpj"]
Nov 02 03:09:15.952189 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:15.952102    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-bc460382-7759547547-vgh7q"]
Nov 02 03:09:15.954559 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:15.954532    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-bc460382-7759547547-vgh7q"]
Nov 02 03:09:16.689349 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:16.689317    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-858236-5c5788679f-7r9bv"]
Nov 02 03:09:16.689980 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:16.689958    4496 cpu_manager.go:414] "containerMap: removing container" podUID="7adde223-834d-401b-98fe-22b635e77a16" containerName="payload-pre-processing"
Nov 02 03:09:16.690032 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:16.689984    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7adde223-834d-401b-98fe-22b635e77a16" containerName="payload-pre-processing"
Nov 02 03:09:16.690087 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:16.690060    4496 cpu_manager.go:414] "containerMap: removing container" podUID="a22f346f-bf44-4fdf-adc4-1559afe4620d" containerName="payload-processing"
Nov 02 03:09:16.690087 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:16.690070    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a22f346f-bf44-4fdf-adc4-1559afe4620d" containerName="payload-processing"
Nov 02 03:09:16.690213 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:16.690199    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="a22f346f-bf44-4fdf-adc4-1559afe4620d" containerName="payload-processing"
Nov 02 03:09:16.690248 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:16.690223    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="7adde223-834d-401b-98fe-22b635e77a16" containerName="payload-pre-processing"
Nov 02 03:09:16.694842 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:16.694820    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-858236-5c5788679f-7r9bv"
Nov 02 03:09:16.749054 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:16.748989    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-858236-dockercfg-77p5v\""
Nov 02 03:09:16.750386 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:16.750338    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-858236\""
Nov 02 03:09:16.772615 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:16.763647    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-858236-5c5788679f-7r9bv"]
Nov 02 03:09:16.773591 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:16.773534    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ea5b3ef-2d8a-49a5-be96-3a9b3dc766be-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-858236-5c5788679f-7r9bv\" (UID: \"4ea5b3ef-2d8a-49a5-be96-3a9b3dc766be\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-858236-5c5788679f-7r9bv"
Nov 02 03:09:16.773906 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:16.773876    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ng9qj\" (UniqueName: \"kubernetes.io/projected/4ea5b3ef-2d8a-49a5-be96-3a9b3dc766be-kube-api-access-ng9qj\") pod \"payload-pre-processing-e2e-worker-w3-858236-5c5788679f-7r9bv\" (UID: \"4ea5b3ef-2d8a-49a5-be96-3a9b3dc766be\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-858236-5c5788679f-7r9bv"
Nov 02 03:09:16.845866 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:16.845829    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-858236-646789f8d5-hzm6p"]
Nov 02 03:09:16.849665 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:16.849646    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-858236-646789f8d5-hzm6p"
Nov 02 03:09:16.875444 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:16.875419    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ea5b3ef-2d8a-49a5-be96-3a9b3dc766be-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-858236-5c5788679f-7r9bv\" (UID: \"4ea5b3ef-2d8a-49a5-be96-3a9b3dc766be\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-858236-5c5788679f-7r9bv"
Nov 02 03:09:16.875568 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:16.875469    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ng9qj\" (UniqueName: \"kubernetes.io/projected/4ea5b3ef-2d8a-49a5-be96-3a9b3dc766be-kube-api-access-ng9qj\") pod \"payload-pre-processing-e2e-worker-w3-858236-5c5788679f-7r9bv\" (UID: \"4ea5b3ef-2d8a-49a5-be96-3a9b3dc766be\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-858236-5c5788679f-7r9bv"
Nov 02 03:09:16.876372 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:16.876340    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ea5b3ef-2d8a-49a5-be96-3a9b3dc766be-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-858236-5c5788679f-7r9bv\" (UID: \"4ea5b3ef-2d8a-49a5-be96-3a9b3dc766be\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-858236-5c5788679f-7r9bv"
Nov 02 03:09:16.904212 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:16.904189    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-858236-646789f8d5-hzm6p"]
Nov 02 03:09:16.933251 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:16.933223    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ng9qj\" (UniqueName: \"kubernetes.io/projected/4ea5b3ef-2d8a-49a5-be96-3a9b3dc766be-kube-api-access-ng9qj\") pod \"payload-pre-processing-e2e-worker-w3-858236-5c5788679f-7r9bv\" (UID: \"4ea5b3ef-2d8a-49a5-be96-3a9b3dc766be\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-858236-5c5788679f-7r9bv"
Nov 02 03:09:16.976411 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:16.976377    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09a61fb0-97bf-4daa-8bb0-d1d07bc858fd-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-858236-646789f8d5-hzm6p\" (UID: \"09a61fb0-97bf-4daa-8bb0-d1d07bc858fd\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-858236-646789f8d5-hzm6p"
Nov 02 03:09:16.976563 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:16.976546    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcm29\" (UniqueName: \"kubernetes.io/projected/09a61fb0-97bf-4daa-8bb0-d1d07bc858fd-kube-api-access-tcm29\") pod \"payload-processing-e2e-worker-w3-858236-646789f8d5-hzm6p\" (UID: \"09a61fb0-97bf-4daa-8bb0-d1d07bc858fd\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-858236-646789f8d5-hzm6p"
Nov 02 03:09:17.008385 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:17.008356    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-858236-5c5788679f-7r9bv"
Nov 02 03:09:17.077931 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:17.077895    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcm29\" (UniqueName: \"kubernetes.io/projected/09a61fb0-97bf-4daa-8bb0-d1d07bc858fd-kube-api-access-tcm29\") pod \"payload-processing-e2e-worker-w3-858236-646789f8d5-hzm6p\" (UID: \"09a61fb0-97bf-4daa-8bb0-d1d07bc858fd\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-858236-646789f8d5-hzm6p"
Nov 02 03:09:17.078075 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:17.077992    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09a61fb0-97bf-4daa-8bb0-d1d07bc858fd-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-858236-646789f8d5-hzm6p\" (UID: \"09a61fb0-97bf-4daa-8bb0-d1d07bc858fd\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-858236-646789f8d5-hzm6p"
Nov 02 03:09:17.078810 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:17.078709    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09a61fb0-97bf-4daa-8bb0-d1d07bc858fd-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-858236-646789f8d5-hzm6p\" (UID: \"09a61fb0-97bf-4daa-8bb0-d1d07bc858fd\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-858236-646789f8d5-hzm6p"
Nov 02 03:09:17.144429 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:17.144398    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcm29\" (UniqueName: \"kubernetes.io/projected/09a61fb0-97bf-4daa-8bb0-d1d07bc858fd-kube-api-access-tcm29\") pod \"payload-processing-e2e-worker-w3-858236-646789f8d5-hzm6p\" (UID: \"09a61fb0-97bf-4daa-8bb0-d1d07bc858fd\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-858236-646789f8d5-hzm6p"
Nov 02 03:09:17.163205 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:17.163177    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-858236-646789f8d5-hzm6p"
Nov 02 03:09:17.386714 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:17.386679    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-858236-5c5788679f-7r9bv"]
Nov 02 03:09:17.388026 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:09:17.387987    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ea5b3ef_2d8a_49a5_be96_3a9b3dc766be.slice/crio-b9de93ba64afadeca6c42aabe8c44aac6fe1f8b4b726cf4984b49065f11f60e8 WatchSource:0}: Error finding container b9de93ba64afadeca6c42aabe8c44aac6fe1f8b4b726cf4984b49065f11f60e8: Status 404 returned error can't find the container with id b9de93ba64afadeca6c42aabe8c44aac6fe1f8b4b726cf4984b49065f11f60e8
Nov 02 03:09:17.482500 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:17.482463    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-4bbf98be-57fd77bc87-5lzs4" podUID="ff37f37f-b7ba-4461-b859-5e6a9cc02efc" containerName="payload-processing" containerID="cri-o://83edd830c365bd266e55e3c43b8bc14f89e27e3d59a4148014236f6274614bc1" gracePeriod=25
Nov 02 03:09:17.550649 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:17.550616    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-858236-646789f8d5-hzm6p"]
Nov 02 03:09:17.551855 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:09:17.551834    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09a61fb0_97bf_4daa_8bb0_d1d07bc858fd.slice/crio-66f2043bafc891cc17c45d53254c65d12b63612108f8af0dee4537077995fe6c WatchSource:0}: Error finding container 66f2043bafc891cc17c45d53254c65d12b63612108f8af0dee4537077995fe6c: Status 404 returned error can't find the container with id 66f2043bafc891cc17c45d53254c65d12b63612108f8af0dee4537077995fe6c
Nov 02 03:09:17.554646 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:17.554621    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-858236-5c5788679f-7r9bv" event={"ID":"4ea5b3ef-2d8a-49a5-be96-3a9b3dc766be","Type":"ContainerStarted","Data":"ad12f9d3f2e8c81ea9b3d40c001151ff6a4cce7393b642cb0132494d121ca366"}
Nov 02 03:09:17.554745 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:17.554663    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-858236-5c5788679f-7r9bv" event={"ID":"4ea5b3ef-2d8a-49a5-be96-3a9b3dc766be","Type":"ContainerStarted","Data":"b9de93ba64afadeca6c42aabe8c44aac6fe1f8b4b726cf4984b49065f11f60e8"}
Nov 02 03:09:17.554745 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:17.554731    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-858236-5c5788679f-7r9bv"
Nov 02 03:09:17.572748 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:17.572714    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7adde223-834d-401b-98fe-22b635e77a16" path="/var/lib/kubelet/pods/7adde223-834d-401b-98fe-22b635e77a16/volumes"
Nov 02 03:09:17.573181 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:17.573163    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a22f346f-bf44-4fdf-adc4-1559afe4620d" path="/var/lib/kubelet/pods/a22f346f-bf44-4fdf-adc4-1559afe4620d/volumes"
Nov 02 03:09:17.587245 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:17.587200    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-4bbf98be-b9594874c-9fqx9" podUID="a879b9fd-06a4-4d5a-b3be-3a3d904683e2" containerName="payload-pre-processing" containerID="cri-o://20baabc5966ba63e6bc4e2cfff51891b7679fb7e81119ecfb2c789c94c21fd4d" gracePeriod=25
Nov 02 03:09:17.701347 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:17.701300    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-858236-5c5788679f-7r9bv" podStartSLOduration=1.701284946 podStartE2EDuration="1.701284946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:09:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:09:17.701231483 +0000 UTC m=+1298.279838862" watchObservedRunningTime="2026-11-02 03:09:17.701284946 +0000 UTC m=+1298.279892341"
Nov 02 03:09:17.719538 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:17.719515    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4bbf98be-57fd77bc87-5lzs4"
Nov 02 03:09:17.857124 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:17.857099    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4bbf98be-b9594874c-9fqx9"
Nov 02 03:09:17.886987 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:17.886961    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff37f37f-b7ba-4461-b859-5e6a9cc02efc-kube-api-access-jpj8h\" (UniqueName: \"kubernetes.io/projected/ff37f37f-b7ba-4461-b859-5e6a9cc02efc-kube-api-access-jpj8h\") pod \"ff37f37f-b7ba-4461-b859-5e6a9cc02efc\" (UID: \"ff37f37f-b7ba-4461-b859-5e6a9cc02efc\") "
Nov 02 03:09:17.887150 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:17.887111    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff37f37f-b7ba-4461-b859-5e6a9cc02efc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff37f37f-b7ba-4461-b859-5e6a9cc02efc-plugins-config-volume\") pod \"ff37f37f-b7ba-4461-b859-5e6a9cc02efc\" (UID: \"ff37f37f-b7ba-4461-b859-5e6a9cc02efc\") "
Nov 02 03:09:17.887452 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:17.887428    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff37f37f-b7ba-4461-b859-5e6a9cc02efc-plugins-config-volume" pod "ff37f37f-b7ba-4461-b859-5e6a9cc02efc" (UID: "ff37f37f-b7ba-4461-b859-5e6a9cc02efc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:09:17.888999 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:17.888977    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff37f37f-b7ba-4461-b859-5e6a9cc02efc-kube-api-access-jpj8h" pod "ff37f37f-b7ba-4461-b859-5e6a9cc02efc" (UID: "ff37f37f-b7ba-4461-b859-5e6a9cc02efc"). InnerVolumeSpecName "kube-api-access-jpj8h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:09:17.987928 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:17.987893    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a879b9fd-06a4-4d5a-b3be-3a3d904683e2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a879b9fd-06a4-4d5a-b3be-3a3d904683e2-plugins-config-volume\") pod \"a879b9fd-06a4-4d5a-b3be-3a3d904683e2\" (UID: \"a879b9fd-06a4-4d5a-b3be-3a3d904683e2\") "
Nov 02 03:09:17.988105 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:17.988007    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a879b9fd-06a4-4d5a-b3be-3a3d904683e2-kube-api-access-pw44r\" (UniqueName: \"kubernetes.io/projected/a879b9fd-06a4-4d5a-b3be-3a3d904683e2-kube-api-access-pw44r\") pod \"a879b9fd-06a4-4d5a-b3be-3a3d904683e2\" (UID: \"a879b9fd-06a4-4d5a-b3be-3a3d904683e2\") "
Nov 02 03:09:17.988259 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:17.988238    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a879b9fd-06a4-4d5a-b3be-3a3d904683e2-plugins-config-volume" pod "a879b9fd-06a4-4d5a-b3be-3a3d904683e2" (UID: "a879b9fd-06a4-4d5a-b3be-3a3d904683e2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:09:17.988370 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:17.988356    4496 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a879b9fd-06a4-4d5a-b3be-3a3d904683e2-plugins-config-volume\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:09:17.988412 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:17.988373    4496 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff37f37f-b7ba-4461-b859-5e6a9cc02efc-plugins-config-volume\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:09:17.988412 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:17.988381    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpj8h\" (UniqueName: \"kubernetes.io/projected/ff37f37f-b7ba-4461-b859-5e6a9cc02efc-kube-api-access-jpj8h\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:09:17.989975 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:17.989944    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a879b9fd-06a4-4d5a-b3be-3a3d904683e2-kube-api-access-pw44r" pod "a879b9fd-06a4-4d5a-b3be-3a3d904683e2" (UID: "a879b9fd-06a4-4d5a-b3be-3a3d904683e2"). InnerVolumeSpecName "kube-api-access-pw44r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:09:18.088884 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:18.088845    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pw44r\" (UniqueName: \"kubernetes.io/projected/a879b9fd-06a4-4d5a-b3be-3a3d904683e2-kube-api-access-pw44r\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:09:18.562179 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:18.562135    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-858236-646789f8d5-hzm6p" event={"ID":"09a61fb0-97bf-4daa-8bb0-d1d07bc858fd","Type":"ContainerStarted","Data":"0bf2270fc4c4642fbff347fc13b7015364e2c92d06807d7f19943e35bb55d5e7"}
Nov 02 03:09:18.562373 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:18.562294    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-858236-646789f8d5-hzm6p"
Nov 02 03:09:18.562373 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:18.562341    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-858236-646789f8d5-hzm6p" event={"ID":"09a61fb0-97bf-4daa-8bb0-d1d07bc858fd","Type":"ContainerStarted","Data":"66f2043bafc891cc17c45d53254c65d12b63612108f8af0dee4537077995fe6c"}
Nov 02 03:09:18.563602 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:18.563581    4496 generic.go:356] "Generic (PLEG): container finished" podID="ff37f37f-b7ba-4461-b859-5e6a9cc02efc" containerID="83edd830c365bd266e55e3c43b8bc14f89e27e3d59a4148014236f6274614bc1" exitCode=0
Nov 02 03:09:18.563734 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:18.563632    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4bbf98be-57fd77bc87-5lzs4"
Nov 02 03:09:18.563734 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:18.563661    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4bbf98be-57fd77bc87-5lzs4" event={"ID":"ff37f37f-b7ba-4461-b859-5e6a9cc02efc","Type":"ContainerDied","Data":"83edd830c365bd266e55e3c43b8bc14f89e27e3d59a4148014236f6274614bc1"}
Nov 02 03:09:18.563734 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:18.563702    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4bbf98be-57fd77bc87-5lzs4" event={"ID":"ff37f37f-b7ba-4461-b859-5e6a9cc02efc","Type":"ContainerDied","Data":"3a52bf57da7bcceced5a364f6c9a06116a971b6bb22762bf8d1a933537665796"}
Nov 02 03:09:18.563734 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:18.563733    4496 scope.go:122] "RemoveContainer" containerID="83edd830c365bd266e55e3c43b8bc14f89e27e3d59a4148014236f6274614bc1"
Nov 02 03:09:18.565005 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:18.564950    4496 generic.go:356] "Generic (PLEG): container finished" podID="a879b9fd-06a4-4d5a-b3be-3a3d904683e2" containerID="20baabc5966ba63e6bc4e2cfff51891b7679fb7e81119ecfb2c789c94c21fd4d" exitCode=0
Nov 02 03:09:18.565005 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:18.564988    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4bbf98be-b9594874c-9fqx9"
Nov 02 03:09:18.565121 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:18.565034    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4bbf98be-b9594874c-9fqx9" event={"ID":"a879b9fd-06a4-4d5a-b3be-3a3d904683e2","Type":"ContainerDied","Data":"20baabc5966ba63e6bc4e2cfff51891b7679fb7e81119ecfb2c789c94c21fd4d"}
Nov 02 03:09:18.565121 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:18.565080    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4bbf98be-b9594874c-9fqx9" event={"ID":"a879b9fd-06a4-4d5a-b3be-3a3d904683e2","Type":"ContainerDied","Data":"efe1ab68d47ec2784f738a06338a14e4559b9bff722827dc199b643204ba94ff"}
Nov 02 03:09:18.574753 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:18.574735    4496 scope.go:122] "RemoveContainer" containerID="83edd830c365bd266e55e3c43b8bc14f89e27e3d59a4148014236f6274614bc1"
Nov 02 03:09:18.575257 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:09:18.575234    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83edd830c365bd266e55e3c43b8bc14f89e27e3d59a4148014236f6274614bc1\": container with ID starting with 83edd830c365bd266e55e3c43b8bc14f89e27e3d59a4148014236f6274614bc1 not found: ID does not exist" containerID="83edd830c365bd266e55e3c43b8bc14f89e27e3d59a4148014236f6274614bc1"
Nov 02 03:09:18.575364 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:18.575265    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83edd830c365bd266e55e3c43b8bc14f89e27e3d59a4148014236f6274614bc1"} err="failed to get container status \"83edd830c365bd266e55e3c43b8bc14f89e27e3d59a4148014236f6274614bc1\": rpc error: code = NotFound desc = could not find container \"83edd830c365bd266e55e3c43b8bc14f89e27e3d59a4148014236f6274614bc1\": container with ID starting with 83edd830c365bd266e55e3c43b8bc14f89e27e3d59a4148014236f6274614bc1 not found: ID does not exist"
Nov 02 03:09:18.575364 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:18.575286    4496 scope.go:122] "RemoveContainer" containerID="20baabc5966ba63e6bc4e2cfff51891b7679fb7e81119ecfb2c789c94c21fd4d"
Nov 02 03:09:18.585248 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:18.585231    4496 scope.go:122] "RemoveContainer" containerID="20baabc5966ba63e6bc4e2cfff51891b7679fb7e81119ecfb2c789c94c21fd4d"
Nov 02 03:09:18.585495 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:09:18.585476    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20baabc5966ba63e6bc4e2cfff51891b7679fb7e81119ecfb2c789c94c21fd4d\": container with ID starting with 20baabc5966ba63e6bc4e2cfff51891b7679fb7e81119ecfb2c789c94c21fd4d not found: ID does not exist" containerID="20baabc5966ba63e6bc4e2cfff51891b7679fb7e81119ecfb2c789c94c21fd4d"
Nov 02 03:09:18.585554 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:18.585502    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20baabc5966ba63e6bc4e2cfff51891b7679fb7e81119ecfb2c789c94c21fd4d"} err="failed to get container status \"20baabc5966ba63e6bc4e2cfff51891b7679fb7e81119ecfb2c789c94c21fd4d\": rpc error: code = NotFound desc = could not find container \"20baabc5966ba63e6bc4e2cfff51891b7679fb7e81119ecfb2c789c94c21fd4d\": container with ID starting with 20baabc5966ba63e6bc4e2cfff51891b7679fb7e81119ecfb2c789c94c21fd4d not found: ID does not exist"
Nov 02 03:09:18.683031 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:18.682978    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-858236-646789f8d5-hzm6p" podStartSLOduration=2.682965264 podStartE2EDuration="2.682965264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:09:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:09:18.681933946 +0000 UTC m=+1299.260541326" watchObservedRunningTime="2026-11-02 03:09:18.682965264 +0000 UTC m=+1299.261572681"
Nov 02 03:09:18.807186 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:18.807153    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4bbf98be-b9594874c-9fqx9"]
Nov 02 03:09:18.813872 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:18.813798    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4bbf98be-b9594874c-9fqx9"]
Nov 02 03:09:18.924969 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:18.924935    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4bbf98be-57fd77bc87-5lzs4"]
Nov 02 03:09:18.956217 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:18.956162    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4bbf98be-57fd77bc87-5lzs4"]
Nov 02 03:09:19.270028 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:19.269987    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-398828-75678b79b7-bbqqw"]
Nov 02 03:09:19.270509 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:19.270495    4496 cpu_manager.go:414] "containerMap: removing container" podUID="a879b9fd-06a4-4d5a-b3be-3a3d904683e2" containerName="payload-pre-processing"
Nov 02 03:09:19.270554 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:19.270511    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a879b9fd-06a4-4d5a-b3be-3a3d904683e2" containerName="payload-pre-processing"
Nov 02 03:09:19.270554 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:19.270530    4496 cpu_manager.go:414] "containerMap: removing container" podUID="ff37f37f-b7ba-4461-b859-5e6a9cc02efc" containerName="payload-processing"
Nov 02 03:09:19.270554 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:19.270535    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff37f37f-b7ba-4461-b859-5e6a9cc02efc" containerName="payload-processing"
Nov 02 03:09:19.270644 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:19.270607    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="a879b9fd-06a4-4d5a-b3be-3a3d904683e2" containerName="payload-pre-processing"
Nov 02 03:09:19.270644 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:19.270618    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff37f37f-b7ba-4461-b859-5e6a9cc02efc" containerName="payload-processing"
Nov 02 03:09:19.275284 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:19.275228    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-398828-75678b79b7-bbqqw"
Nov 02 03:09:19.330913 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:19.330880    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-398828-dockercfg-666x9\""
Nov 02 03:09:19.341078 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:19.341045    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-398828\""
Nov 02 03:09:19.346470 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:19.346435    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-398828-75678b79b7-bbqqw"]
Nov 02 03:09:19.402146 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:19.402112    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce91e365-c823-4791-900a-c3bb3ccd4d36-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-398828-75678b79b7-bbqqw\" (UID: \"ce91e365-c823-4791-900a-c3bb3ccd4d36\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-398828-75678b79b7-bbqqw"
Nov 02 03:09:19.402319 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:19.402181    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nr4kq\" (UniqueName: \"kubernetes.io/projected/ce91e365-c823-4791-900a-c3bb3ccd4d36-kube-api-access-nr4kq\") pod \"payload-pre-processing-e2e-worker-w4-398828-75678b79b7-bbqqw\" (UID: \"ce91e365-c823-4791-900a-c3bb3ccd4d36\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-398828-75678b79b7-bbqqw"
Nov 02 03:09:19.405884 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:19.405857    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-398828-54955d6b75-czhwj"]
Nov 02 03:09:19.411194 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:19.411173    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-398828-54955d6b75-czhwj"
Nov 02 03:09:19.469314 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:19.469281    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-398828-54955d6b75-czhwj"]
Nov 02 03:09:19.495846 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:19.495816    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ca6546-76c6c456c9-vxsgp"
Nov 02 03:09:19.496002 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:19.495976    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-ca6546-8d886cbbf-qx4dh"
Nov 02 03:09:19.502859 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:19.502833    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce91e365-c823-4791-900a-c3bb3ccd4d36-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-398828-75678b79b7-bbqqw\" (UID: \"ce91e365-c823-4791-900a-c3bb3ccd4d36\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-398828-75678b79b7-bbqqw"
Nov 02 03:09:19.503076 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:19.502879    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba5a7a35-1ea6-48e9-8817-dd7a7f6f08fc-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-398828-54955d6b75-czhwj\" (UID: \"ba5a7a35-1ea6-48e9-8817-dd7a7f6f08fc\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-398828-54955d6b75-czhwj"
Nov 02 03:09:19.503076 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:19.502939    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nr4kq\" (UniqueName: \"kubernetes.io/projected/ce91e365-c823-4791-900a-c3bb3ccd4d36-kube-api-access-nr4kq\") pod \"payload-pre-processing-e2e-worker-w4-398828-75678b79b7-bbqqw\" (UID: \"ce91e365-c823-4791-900a-c3bb3ccd4d36\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-398828-75678b79b7-bbqqw"
Nov 02 03:09:19.503076 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:19.503070    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsz5g\" (UniqueName: \"kubernetes.io/projected/ba5a7a35-1ea6-48e9-8817-dd7a7f6f08fc-kube-api-access-gsz5g\") pod \"payload-processing-e2e-worker-w4-398828-54955d6b75-czhwj\" (UID: \"ba5a7a35-1ea6-48e9-8817-dd7a7f6f08fc\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-398828-54955d6b75-czhwj"
Nov 02 03:09:19.503512 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:19.503494    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce91e365-c823-4791-900a-c3bb3ccd4d36-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-398828-75678b79b7-bbqqw\" (UID: \"ce91e365-c823-4791-900a-c3bb3ccd4d36\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-398828-75678b79b7-bbqqw"
Nov 02 03:09:19.561394 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:19.561326    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nr4kq\" (UniqueName: \"kubernetes.io/projected/ce91e365-c823-4791-900a-c3bb3ccd4d36-kube-api-access-nr4kq\") pod \"payload-pre-processing-e2e-worker-w4-398828-75678b79b7-bbqqw\" (UID: \"ce91e365-c823-4791-900a-c3bb3ccd4d36\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-398828-75678b79b7-bbqqw"
Nov 02 03:09:19.573584 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:19.573558    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a879b9fd-06a4-4d5a-b3be-3a3d904683e2" path="/var/lib/kubelet/pods/a879b9fd-06a4-4d5a-b3be-3a3d904683e2/volumes"
Nov 02 03:09:19.574119 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:19.574088    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff37f37f-b7ba-4461-b859-5e6a9cc02efc" path="/var/lib/kubelet/pods/ff37f37f-b7ba-4461-b859-5e6a9cc02efc/volumes"
Nov 02 03:09:19.588575 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:19.588555    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-398828-75678b79b7-bbqqw"
Nov 02 03:09:19.603898 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:19.603872    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsz5g\" (UniqueName: \"kubernetes.io/projected/ba5a7a35-1ea6-48e9-8817-dd7a7f6f08fc-kube-api-access-gsz5g\") pod \"payload-processing-e2e-worker-w4-398828-54955d6b75-czhwj\" (UID: \"ba5a7a35-1ea6-48e9-8817-dd7a7f6f08fc\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-398828-54955d6b75-czhwj"
Nov 02 03:09:19.604055 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:19.604007    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba5a7a35-1ea6-48e9-8817-dd7a7f6f08fc-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-398828-54955d6b75-czhwj\" (UID: \"ba5a7a35-1ea6-48e9-8817-dd7a7f6f08fc\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-398828-54955d6b75-czhwj"
Nov 02 03:09:19.604814 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:19.604775    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba5a7a35-1ea6-48e9-8817-dd7a7f6f08fc-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-398828-54955d6b75-czhwj\" (UID: \"ba5a7a35-1ea6-48e9-8817-dd7a7f6f08fc\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-398828-54955d6b75-czhwj"
Nov 02 03:09:19.664002 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:19.663966    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsz5g\" (UniqueName: \"kubernetes.io/projected/ba5a7a35-1ea6-48e9-8817-dd7a7f6f08fc-kube-api-access-gsz5g\") pod \"payload-processing-e2e-worker-w4-398828-54955d6b75-czhwj\" (UID: \"ba5a7a35-1ea6-48e9-8817-dd7a7f6f08fc\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-398828-54955d6b75-czhwj"
Nov 02 03:09:19.727001 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:19.726972    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-398828-54955d6b75-czhwj"
Nov 02 03:09:19.992802 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:19.992755    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-398828-75678b79b7-bbqqw"]
Nov 02 03:09:19.998234 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:09:19.995603    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce91e365_c823_4791_900a_c3bb3ccd4d36.slice/crio-9386d0c28da999050e0fd491eedb424cd79950cf250ba3e0566630023360aeee WatchSource:0}: Error finding container 9386d0c28da999050e0fd491eedb424cd79950cf250ba3e0566630023360aeee: Status 404 returned error can't find the container with id 9386d0c28da999050e0fd491eedb424cd79950cf250ba3e0566630023360aeee
Nov 02 03:09:20.120384 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:20.120355    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-398828-54955d6b75-czhwj"]
Nov 02 03:09:20.121676 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:09:20.121648    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba5a7a35_1ea6_48e9_8817_dd7a7f6f08fc.slice/crio-61724c17c9ec05ffa3493f241c9790b38aacbeba527300eb01ca1c9c696e03b5 WatchSource:0}: Error finding container 61724c17c9ec05ffa3493f241c9790b38aacbeba527300eb01ca1c9c696e03b5: Status 404 returned error can't find the container with id 61724c17c9ec05ffa3493f241c9790b38aacbeba527300eb01ca1c9c696e03b5
Nov 02 03:09:20.582567 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:20.582534    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-398828-54955d6b75-czhwj" event={"ID":"ba5a7a35-1ea6-48e9-8817-dd7a7f6f08fc","Type":"ContainerStarted","Data":"2cb2c85fa4ee35903735c777ca2c32e2a41edb03c856637dcc5b1398897639ba"}
Nov 02 03:09:20.582774 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:20.582587    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-398828-54955d6b75-czhwj" event={"ID":"ba5a7a35-1ea6-48e9-8817-dd7a7f6f08fc","Type":"ContainerStarted","Data":"61724c17c9ec05ffa3493f241c9790b38aacbeba527300eb01ca1c9c696e03b5"}
Nov 02 03:09:20.582774 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:20.582626    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-398828-54955d6b75-czhwj"
Nov 02 03:09:20.584328 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:20.584305    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-398828-75678b79b7-bbqqw" event={"ID":"ce91e365-c823-4791-900a-c3bb3ccd4d36","Type":"ContainerStarted","Data":"cfbc8cee8ca607ed8ec1ba68b6374a79b6831788b1e73d2fff895294fa0cbfc5"}
Nov 02 03:09:20.584451 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:20.584347    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-398828-75678b79b7-bbqqw" event={"ID":"ce91e365-c823-4791-900a-c3bb3ccd4d36","Type":"ContainerStarted","Data":"9386d0c28da999050e0fd491eedb424cd79950cf250ba3e0566630023360aeee"}
Nov 02 03:09:20.584451 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:20.584387    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-398828-75678b79b7-bbqqw"
Nov 02 03:09:20.699511 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:20.699461    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-398828-54955d6b75-czhwj" podStartSLOduration=1.699446817 podStartE2EDuration="1.699446817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:09:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:09:20.697042377 +0000 UTC m=+1301.275649755" watchObservedRunningTime="2026-11-02 03:09:20.699446817 +0000 UTC m=+1301.278054194"
Nov 02 03:09:20.813909 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:20.813844    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-398828-75678b79b7-bbqqw" podStartSLOduration=1.813825428 podStartE2EDuration="1.813825428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:09:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:09:20.810858325 +0000 UTC m=+1301.389465706" watchObservedRunningTime="2026-11-02 03:09:20.813825428 +0000 UTC m=+1301.392432810"
Nov 02 03:09:28.568146 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:28.568075    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-858236-5c5788679f-7r9bv"
Nov 02 03:09:29.578215 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:29.578187    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-858236-646789f8d5-hzm6p"
Nov 02 03:09:30.716761 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:30.716725    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bc460382-gjgs8"]
Nov 02 03:09:30.768777 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:30.768742    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bc460382-gjgs8"]
Nov 02 03:09:31.573586 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:31.573544    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5eb1a9f0-146a-4734-8b94-87f373fe9d50" path="/var/lib/kubelet/pods/5eb1a9f0-146a-4734-8b94-87f373fe9d50/volumes"
Nov 02 03:09:31.593481 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:31.593459    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-398828-75678b79b7-bbqqw"
Nov 02 03:09:31.593626 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:31.593522    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-398828-54955d6b75-czhwj"
Nov 02 03:09:31.997038 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:31.996103    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4bbf98be-c5wlc"]
Nov 02 03:09:32.016035 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:32.015924    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4bbf98be-c5wlc"]
Nov 02 03:09:33.529574 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:09:33.529528    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 03:09:33.572929 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:33.572897    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6494ffa1-e21e-415a-b29e-9c3e517de1c1" path="/var/lib/kubelet/pods/6494ffa1-e21e-415a-b29e-9c3e517de1c1/volumes"
Nov 02 03:09:41.199444 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:41.199408    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-5ac93f-6578c5846c-lxsjl"]
Nov 02 03:09:41.205323 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:41.205301    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5ac93f-6578c5846c-lxsjl"
Nov 02 03:09:41.259588 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:41.259555    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-5ac93f-6578c5846c-lxsjl"]
Nov 02 03:09:41.307100 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:41.307065    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-5ac93f\""
Nov 02 03:09:41.307100 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:41.307102    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-5ac93f-dockercfg-qhzpr\""
Nov 02 03:09:41.344845 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:41.344792    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/deeeaa8e-ec07-4416-8f1c-c032ffe0e55e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-5ac93f-6578c5846c-lxsjl\" (UID: \"deeeaa8e-ec07-4416-8f1c-c032ffe0e55e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5ac93f-6578c5846c-lxsjl"
Nov 02 03:09:41.345425 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:41.345394    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plghg\" (UniqueName: \"kubernetes.io/projected/deeeaa8e-ec07-4416-8f1c-c032ffe0e55e-kube-api-access-plghg\") pod \"payload-pre-processing-e2e-worker-w1-5ac93f-6578c5846c-lxsjl\" (UID: \"deeeaa8e-ec07-4416-8f1c-c032ffe0e55e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5ac93f-6578c5846c-lxsjl"
Nov 02 03:09:41.360809 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:41.360781    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-5ac93f-6f67ddfdc5-bwhr4"]
Nov 02 03:09:41.364508 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:41.364489    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-5ac93f-6f67ddfdc5-bwhr4"
Nov 02 03:09:41.423889 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:41.423858    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-5ac93f-6f67ddfdc5-bwhr4"]
Nov 02 03:09:41.446931 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:41.446895    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/deeeaa8e-ec07-4416-8f1c-c032ffe0e55e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-5ac93f-6578c5846c-lxsjl\" (UID: \"deeeaa8e-ec07-4416-8f1c-c032ffe0e55e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5ac93f-6578c5846c-lxsjl"
Nov 02 03:09:41.447097 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:41.447031    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23af24b5-7173-4169-9169-ea0c14916a53-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-5ac93f-6f67ddfdc5-bwhr4\" (UID: \"23af24b5-7173-4169-9169-ea0c14916a53\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-5ac93f-6f67ddfdc5-bwhr4"
Nov 02 03:09:41.447097 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:41.447068    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plghg\" (UniqueName: \"kubernetes.io/projected/deeeaa8e-ec07-4416-8f1c-c032ffe0e55e-kube-api-access-plghg\") pod \"payload-pre-processing-e2e-worker-w1-5ac93f-6578c5846c-lxsjl\" (UID: \"deeeaa8e-ec07-4416-8f1c-c032ffe0e55e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5ac93f-6578c5846c-lxsjl"
Nov 02 03:09:41.447246 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:41.447110    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6sfw\" (UniqueName: \"kubernetes.io/projected/23af24b5-7173-4169-9169-ea0c14916a53-kube-api-access-l6sfw\") pod \"payload-processing-e2e-worker-w1-5ac93f-6f67ddfdc5-bwhr4\" (UID: \"23af24b5-7173-4169-9169-ea0c14916a53\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-5ac93f-6f67ddfdc5-bwhr4"
Nov 02 03:09:41.447703 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:41.447685    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/deeeaa8e-ec07-4416-8f1c-c032ffe0e55e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-5ac93f-6578c5846c-lxsjl\" (UID: \"deeeaa8e-ec07-4416-8f1c-c032ffe0e55e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5ac93f-6578c5846c-lxsjl"
Nov 02 03:09:41.505666 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:41.505631    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plghg\" (UniqueName: \"kubernetes.io/projected/deeeaa8e-ec07-4416-8f1c-c032ffe0e55e-kube-api-access-plghg\") pod \"payload-pre-processing-e2e-worker-w1-5ac93f-6578c5846c-lxsjl\" (UID: \"deeeaa8e-ec07-4416-8f1c-c032ffe0e55e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5ac93f-6578c5846c-lxsjl"
Nov 02 03:09:41.519592 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:41.519567    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5ac93f-6578c5846c-lxsjl"
Nov 02 03:09:41.549089 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:41.548199    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6sfw\" (UniqueName: \"kubernetes.io/projected/23af24b5-7173-4169-9169-ea0c14916a53-kube-api-access-l6sfw\") pod \"payload-processing-e2e-worker-w1-5ac93f-6f67ddfdc5-bwhr4\" (UID: \"23af24b5-7173-4169-9169-ea0c14916a53\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-5ac93f-6f67ddfdc5-bwhr4"
Nov 02 03:09:41.549089 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:41.548370    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23af24b5-7173-4169-9169-ea0c14916a53-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-5ac93f-6f67ddfdc5-bwhr4\" (UID: \"23af24b5-7173-4169-9169-ea0c14916a53\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-5ac93f-6f67ddfdc5-bwhr4"
Nov 02 03:09:41.549556 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:41.549530    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23af24b5-7173-4169-9169-ea0c14916a53-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-5ac93f-6f67ddfdc5-bwhr4\" (UID: \"23af24b5-7173-4169-9169-ea0c14916a53\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-5ac93f-6f67ddfdc5-bwhr4"
Nov 02 03:09:41.607241 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:41.607211    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6sfw\" (UniqueName: \"kubernetes.io/projected/23af24b5-7173-4169-9169-ea0c14916a53-kube-api-access-l6sfw\") pod \"payload-processing-e2e-worker-w1-5ac93f-6f67ddfdc5-bwhr4\" (UID: \"23af24b5-7173-4169-9169-ea0c14916a53\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-5ac93f-6f67ddfdc5-bwhr4"
Nov 02 03:09:41.677924 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:41.677898    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-5ac93f-6f67ddfdc5-bwhr4"
Nov 02 03:09:41.899643 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:41.899617    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-5ac93f-6578c5846c-lxsjl"]
Nov 02 03:09:41.901788 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:09:41.901755    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddeeeaa8e_ec07_4416_8f1c_c032ffe0e55e.slice/crio-029c865b7b1557d0aae99a23803127373a70291ebf0ae76905c86ff209bb65bd WatchSource:0}: Error finding container 029c865b7b1557d0aae99a23803127373a70291ebf0ae76905c86ff209bb65bd: Status 404 returned error can't find the container with id 029c865b7b1557d0aae99a23803127373a70291ebf0ae76905c86ff209bb65bd
Nov 02 03:09:42.054957 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:42.054927    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-5ac93f-6f67ddfdc5-bwhr4"]
Nov 02 03:09:42.055518 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:09:42.055494    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23af24b5_7173_4169_9169_ea0c14916a53.slice/crio-1ac49e0a51903955094983a5189b5a9a387b7a6f2c40166a77f850991c3de47a WatchSource:0}: Error finding container 1ac49e0a51903955094983a5189b5a9a387b7a6f2c40166a77f850991c3de47a: Status 404 returned error can't find the container with id 1ac49e0a51903955094983a5189b5a9a387b7a6f2c40166a77f850991c3de47a
Nov 02 03:09:42.726268 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:42.726225    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5ac93f-6578c5846c-lxsjl" event={"ID":"deeeaa8e-ec07-4416-8f1c-c032ffe0e55e","Type":"ContainerStarted","Data":"397524e61b69d74734d85c3de412784475eb5cebeb98767cf3543f18707240a3"}
Nov 02 03:09:42.726740 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:42.726290    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5ac93f-6578c5846c-lxsjl" event={"ID":"deeeaa8e-ec07-4416-8f1c-c032ffe0e55e","Type":"ContainerStarted","Data":"029c865b7b1557d0aae99a23803127373a70291ebf0ae76905c86ff209bb65bd"}
Nov 02 03:09:42.726740 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:42.726426    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5ac93f-6578c5846c-lxsjl"
Nov 02 03:09:42.727914 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:42.727883    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-5ac93f-6f67ddfdc5-bwhr4" event={"ID":"23af24b5-7173-4169-9169-ea0c14916a53","Type":"ContainerStarted","Data":"e7459e9ca917861b5598286d1018fedc02b0f2c4f3ecfc704b0250722453072a"}
Nov 02 03:09:42.728038 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:42.727930    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-5ac93f-6f67ddfdc5-bwhr4" event={"ID":"23af24b5-7173-4169-9169-ea0c14916a53","Type":"ContainerStarted","Data":"1ac49e0a51903955094983a5189b5a9a387b7a6f2c40166a77f850991c3de47a"}
Nov 02 03:09:42.728038 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:42.727958    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-5ac93f-6f67ddfdc5-bwhr4"
Nov 02 03:09:42.843209 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:42.843153    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5ac93f-6578c5846c-lxsjl" podStartSLOduration=1.843139327 podStartE2EDuration="1.843139327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:09:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:09:42.841794917 +0000 UTC m=+1323.420402296" watchObservedRunningTime="2026-11-02 03:09:42.843139327 +0000 UTC m=+1323.421746707"
Nov 02 03:09:42.956991 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:42.956940    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-5ac93f-6f67ddfdc5-bwhr4" podStartSLOduration=1.95692663 podStartE2EDuration="1.95692663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:09:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:09:42.955794722 +0000 UTC m=+1323.534402104" watchObservedRunningTime="2026-11-02 03:09:42.95692663 +0000 UTC m=+1323.535534010"
Nov 02 03:09:45.973460 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:45.973418    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-1f6e39-6d697b7999-7nsr8"]
Nov 02 03:09:45.979806 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:45.979772    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1f6e39-6d697b7999-7nsr8"
Nov 02 03:09:46.054799 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:46.054758    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-1f6e39-6d697b7999-7nsr8"]
Nov 02 03:09:46.093945 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:46.093906    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dv2mw\" (UniqueName: \"kubernetes.io/projected/725d05a7-caa3-4e07-be47-837ea9534075-kube-api-access-dv2mw\") pod \"payload-pre-processing-e2e-worker-w0-1f6e39-6d697b7999-7nsr8\" (UID: \"725d05a7-caa3-4e07-be47-837ea9534075\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1f6e39-6d697b7999-7nsr8"
Nov 02 03:09:46.094158 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:46.094071    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/725d05a7-caa3-4e07-be47-837ea9534075-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-1f6e39-6d697b7999-7nsr8\" (UID: \"725d05a7-caa3-4e07-be47-837ea9534075\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1f6e39-6d697b7999-7nsr8"
Nov 02 03:09:46.100309 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:46.100283    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-1f6e39-dockercfg-dc8pn\""
Nov 02 03:09:46.100309 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:46.100283    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-1f6e39\""
Nov 02 03:09:46.111320 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:46.111296    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-1f6e39-69f9854f95-b4c6w"]
Nov 02 03:09:46.115123 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:46.115104    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-1f6e39-69f9854f95-b4c6w"
Nov 02 03:09:46.171837 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:46.171805    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-1f6e39-69f9854f95-b4c6w"]
Nov 02 03:09:46.195249 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:46.195204    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bp8w8\" (UniqueName: \"kubernetes.io/projected/ccb123ce-4b29-4f58-8201-34433a8bd0b0-kube-api-access-bp8w8\") pod \"payload-processing-e2e-worker-w0-1f6e39-69f9854f95-b4c6w\" (UID: \"ccb123ce-4b29-4f58-8201-34433a8bd0b0\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-1f6e39-69f9854f95-b4c6w"
Nov 02 03:09:46.195249 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:46.195257    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/725d05a7-caa3-4e07-be47-837ea9534075-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-1f6e39-6d697b7999-7nsr8\" (UID: \"725d05a7-caa3-4e07-be47-837ea9534075\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1f6e39-6d697b7999-7nsr8"
Nov 02 03:09:46.195466 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:46.195339    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccb123ce-4b29-4f58-8201-34433a8bd0b0-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-1f6e39-69f9854f95-b4c6w\" (UID: \"ccb123ce-4b29-4f58-8201-34433a8bd0b0\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-1f6e39-69f9854f95-b4c6w"
Nov 02 03:09:46.195540 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:46.195517    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dv2mw\" (UniqueName: \"kubernetes.io/projected/725d05a7-caa3-4e07-be47-837ea9534075-kube-api-access-dv2mw\") pod \"payload-pre-processing-e2e-worker-w0-1f6e39-6d697b7999-7nsr8\" (UID: \"725d05a7-caa3-4e07-be47-837ea9534075\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1f6e39-6d697b7999-7nsr8"
Nov 02 03:09:46.195990 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:46.195974    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/725d05a7-caa3-4e07-be47-837ea9534075-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-1f6e39-6d697b7999-7nsr8\" (UID: \"725d05a7-caa3-4e07-be47-837ea9534075\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1f6e39-6d697b7999-7nsr8"
Nov 02 03:09:46.253919 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:46.253888    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dv2mw\" (UniqueName: \"kubernetes.io/projected/725d05a7-caa3-4e07-be47-837ea9534075-kube-api-access-dv2mw\") pod \"payload-pre-processing-e2e-worker-w0-1f6e39-6d697b7999-7nsr8\" (UID: \"725d05a7-caa3-4e07-be47-837ea9534075\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1f6e39-6d697b7999-7nsr8"
Nov 02 03:09:46.296497 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:46.296470    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccb123ce-4b29-4f58-8201-34433a8bd0b0-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-1f6e39-69f9854f95-b4c6w\" (UID: \"ccb123ce-4b29-4f58-8201-34433a8bd0b0\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-1f6e39-69f9854f95-b4c6w"
Nov 02 03:09:46.296620 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:46.296580    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bp8w8\" (UniqueName: \"kubernetes.io/projected/ccb123ce-4b29-4f58-8201-34433a8bd0b0-kube-api-access-bp8w8\") pod \"payload-processing-e2e-worker-w0-1f6e39-69f9854f95-b4c6w\" (UID: \"ccb123ce-4b29-4f58-8201-34433a8bd0b0\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-1f6e39-69f9854f95-b4c6w"
Nov 02 03:09:46.297218 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:46.297192    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccb123ce-4b29-4f58-8201-34433a8bd0b0-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-1f6e39-69f9854f95-b4c6w\" (UID: \"ccb123ce-4b29-4f58-8201-34433a8bd0b0\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-1f6e39-69f9854f95-b4c6w"
Nov 02 03:09:46.297338 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:46.297270    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1f6e39-6d697b7999-7nsr8"
Nov 02 03:09:46.358269 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:46.358181    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bp8w8\" (UniqueName: \"kubernetes.io/projected/ccb123ce-4b29-4f58-8201-34433a8bd0b0-kube-api-access-bp8w8\") pod \"payload-processing-e2e-worker-w0-1f6e39-69f9854f95-b4c6w\" (UID: \"ccb123ce-4b29-4f58-8201-34433a8bd0b0\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-1f6e39-69f9854f95-b4c6w"
Nov 02 03:09:46.428409 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:46.428379    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-1f6e39-69f9854f95-b4c6w"
Nov 02 03:09:46.676180 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:46.676153    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-1f6e39-6d697b7999-7nsr8"]
Nov 02 03:09:46.676536 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:09:46.676512    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod725d05a7_caa3_4e07_be47_837ea9534075.slice/crio-6cffc3bdd69f67f21c70a3e02d7e01b0b2b82726851bc44d8cce42aa2e7d23b2 WatchSource:0}: Error finding container 6cffc3bdd69f67f21c70a3e02d7e01b0b2b82726851bc44d8cce42aa2e7d23b2: Status 404 returned error can't find the container with id 6cffc3bdd69f67f21c70a3e02d7e01b0b2b82726851bc44d8cce42aa2e7d23b2
Nov 02 03:09:46.754009 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:46.753966    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1f6e39-6d697b7999-7nsr8" event={"ID":"725d05a7-caa3-4e07-be47-837ea9534075","Type":"ContainerStarted","Data":"c27812b8c5bc2b0f391976b47c5841944ffbc08fb9024f09969f744391103a38"}
Nov 02 03:09:46.754146 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:46.754045    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1f6e39-6d697b7999-7nsr8" event={"ID":"725d05a7-caa3-4e07-be47-837ea9534075","Type":"ContainerStarted","Data":"6cffc3bdd69f67f21c70a3e02d7e01b0b2b82726851bc44d8cce42aa2e7d23b2"}
Nov 02 03:09:46.754193 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:46.754165    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1f6e39-6d697b7999-7nsr8"
Nov 02 03:09:46.805960 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:46.805936    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-1f6e39-69f9854f95-b4c6w"]
Nov 02 03:09:46.807775 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:09:46.807750    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccb123ce_4b29_4f58_8201_34433a8bd0b0.slice/crio-98d2e67a0250c09086032a4640b6e181d52c6d78ffd4bdd5d754af7dfaa96d71 WatchSource:0}: Error finding container 98d2e67a0250c09086032a4640b6e181d52c6d78ffd4bdd5d754af7dfaa96d71: Status 404 returned error can't find the container with id 98d2e67a0250c09086032a4640b6e181d52c6d78ffd4bdd5d754af7dfaa96d71
Nov 02 03:09:46.870987 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:46.870941    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1f6e39-6d697b7999-7nsr8" podStartSLOduration=1.870928087 podStartE2EDuration="1.870928087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:09:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:09:46.868668757 +0000 UTC m=+1327.447276137" watchObservedRunningTime="2026-11-02 03:09:46.870928087 +0000 UTC m=+1327.449535466"
Nov 02 03:09:47.761701 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:47.761662    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-1f6e39-69f9854f95-b4c6w" event={"ID":"ccb123ce-4b29-4f58-8201-34433a8bd0b0","Type":"ContainerStarted","Data":"d470216e044b74b66708cf920cfe1d9849b71fdacdaa3f78dc421377e557a3d7"}
Nov 02 03:09:47.762132 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:47.761724    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-1f6e39-69f9854f95-b4c6w" event={"ID":"ccb123ce-4b29-4f58-8201-34433a8bd0b0","Type":"ContainerStarted","Data":"98d2e67a0250c09086032a4640b6e181d52c6d78ffd4bdd5d754af7dfaa96d71"}
Nov 02 03:09:47.762132 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:47.761939    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-1f6e39-69f9854f95-b4c6w"
Nov 02 03:09:47.878636 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:47.878586    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-1f6e39-69f9854f95-b4c6w" podStartSLOduration=1.8785739289999999 podStartE2EDuration="1.878573929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:09:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:09:47.876855073 +0000 UTC m=+1328.455462454" watchObservedRunningTime="2026-11-02 03:09:47.878573929 +0000 UTC m=+1328.457181308"
Nov 02 03:09:53.737728 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:53.737695    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5ac93f-6578c5846c-lxsjl"
Nov 02 03:09:53.738337 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:53.737923    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-5ac93f-6f67ddfdc5-bwhr4"
Nov 02 03:09:57.436542 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:57.436509    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-ca6546/simulator-w2-ca6546-kserve-5959bb649b-5x6tf"]
Nov 02 03:09:57.440235 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:57.440217    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-ca6546/simulator-w2-ca6546-kserve-5959bb649b-5x6tf"
Nov 02 03:09:57.496476 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:57.496447    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-ca6546/simulator-w2-ca6546-kserve-5959bb649b-5x6tf"]
Nov 02 03:09:57.502977 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:57.502942    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fd6a8f7-0b10-4009-a6db-de0a01aca2b8-tmp-dir\") pod \"simulator-w2-ca6546-kserve-5959bb649b-5x6tf\" (UID: \"4fd6a8f7-0b10-4009-a6db-de0a01aca2b8\") " pod="e2e-models-e2e-worker-w2-ca6546/simulator-w2-ca6546-kserve-5959bb649b-5x6tf"
Nov 02 03:09:57.503157 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:57.502998    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fd6a8f7-0b10-4009-a6db-de0a01aca2b8-tls-certs\") pod \"simulator-w2-ca6546-kserve-5959bb649b-5x6tf\" (UID: \"4fd6a8f7-0b10-4009-a6db-de0a01aca2b8\") " pod="e2e-models-e2e-worker-w2-ca6546/simulator-w2-ca6546-kserve-5959bb649b-5x6tf"
Nov 02 03:09:57.503224 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:57.503182    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fd6a8f7-0b10-4009-a6db-de0a01aca2b8-dshm\") pod \"simulator-w2-ca6546-kserve-5959bb649b-5x6tf\" (UID: \"4fd6a8f7-0b10-4009-a6db-de0a01aca2b8\") " pod="e2e-models-e2e-worker-w2-ca6546/simulator-w2-ca6546-kserve-5959bb649b-5x6tf"
Nov 02 03:09:57.503272 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:57.503234    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fd6a8f7-0b10-4009-a6db-de0a01aca2b8-home\") pod \"simulator-w2-ca6546-kserve-5959bb649b-5x6tf\" (UID: \"4fd6a8f7-0b10-4009-a6db-de0a01aca2b8\") " pod="e2e-models-e2e-worker-w2-ca6546/simulator-w2-ca6546-kserve-5959bb649b-5x6tf"
Nov 02 03:09:57.503341 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:57.503321    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7zxp\" (UniqueName: \"kubernetes.io/projected/4fd6a8f7-0b10-4009-a6db-de0a01aca2b8-kube-api-access-n7zxp\") pod \"simulator-w2-ca6546-kserve-5959bb649b-5x6tf\" (UID: \"4fd6a8f7-0b10-4009-a6db-de0a01aca2b8\") " pod="e2e-models-e2e-worker-w2-ca6546/simulator-w2-ca6546-kserve-5959bb649b-5x6tf"
Nov 02 03:09:57.503377 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:57.503362    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fd6a8f7-0b10-4009-a6db-de0a01aca2b8-model-cache\") pod \"simulator-w2-ca6546-kserve-5959bb649b-5x6tf\" (UID: \"4fd6a8f7-0b10-4009-a6db-de0a01aca2b8\") " pod="e2e-models-e2e-worker-w2-ca6546/simulator-w2-ca6546-kserve-5959bb649b-5x6tf"
Nov 02 03:09:57.544290 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:57.544256    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-ca6546\"/\"kube-root-ca.crt\""
Nov 02 03:09:57.544453 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:57.544343    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-ca6546\"/\"default-dockercfg-cm7t6\""
Nov 02 03:09:57.544453 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:57.544445    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-ca6546\"/\"simulator-w2-ca6546-kserve-self-signed-certs\""
Nov 02 03:09:57.544564 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:57.544470    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-ca6546\"/\"openshift-service-ca.crt\""
Nov 02 03:09:57.604257 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:57.604210    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fd6a8f7-0b10-4009-a6db-de0a01aca2b8-dshm\") pod \"simulator-w2-ca6546-kserve-5959bb649b-5x6tf\" (UID: \"4fd6a8f7-0b10-4009-a6db-de0a01aca2b8\") " pod="e2e-models-e2e-worker-w2-ca6546/simulator-w2-ca6546-kserve-5959bb649b-5x6tf"
Nov 02 03:09:57.604450 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:57.604279    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fd6a8f7-0b10-4009-a6db-de0a01aca2b8-home\") pod \"simulator-w2-ca6546-kserve-5959bb649b-5x6tf\" (UID: \"4fd6a8f7-0b10-4009-a6db-de0a01aca2b8\") " pod="e2e-models-e2e-worker-w2-ca6546/simulator-w2-ca6546-kserve-5959bb649b-5x6tf"
Nov 02 03:09:57.604450 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:57.604319    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7zxp\" (UniqueName: \"kubernetes.io/projected/4fd6a8f7-0b10-4009-a6db-de0a01aca2b8-kube-api-access-n7zxp\") pod \"simulator-w2-ca6546-kserve-5959bb649b-5x6tf\" (UID: \"4fd6a8f7-0b10-4009-a6db-de0a01aca2b8\") " pod="e2e-models-e2e-worker-w2-ca6546/simulator-w2-ca6546-kserve-5959bb649b-5x6tf"
Nov 02 03:09:57.604450 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:57.604350    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fd6a8f7-0b10-4009-a6db-de0a01aca2b8-model-cache\") pod \"simulator-w2-ca6546-kserve-5959bb649b-5x6tf\" (UID: \"4fd6a8f7-0b10-4009-a6db-de0a01aca2b8\") " pod="e2e-models-e2e-worker-w2-ca6546/simulator-w2-ca6546-kserve-5959bb649b-5x6tf"
Nov 02 03:09:57.604450 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:57.604419    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fd6a8f7-0b10-4009-a6db-de0a01aca2b8-tmp-dir\") pod \"simulator-w2-ca6546-kserve-5959bb649b-5x6tf\" (UID: \"4fd6a8f7-0b10-4009-a6db-de0a01aca2b8\") " pod="e2e-models-e2e-worker-w2-ca6546/simulator-w2-ca6546-kserve-5959bb649b-5x6tf"
Nov 02 03:09:57.604683 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:57.604455    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fd6a8f7-0b10-4009-a6db-de0a01aca2b8-tls-certs\") pod \"simulator-w2-ca6546-kserve-5959bb649b-5x6tf\" (UID: \"4fd6a8f7-0b10-4009-a6db-de0a01aca2b8\") " pod="e2e-models-e2e-worker-w2-ca6546/simulator-w2-ca6546-kserve-5959bb649b-5x6tf"
Nov 02 03:09:57.604844 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:57.604817    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fd6a8f7-0b10-4009-a6db-de0a01aca2b8-model-cache\") pod \"simulator-w2-ca6546-kserve-5959bb649b-5x6tf\" (UID: \"4fd6a8f7-0b10-4009-a6db-de0a01aca2b8\") " pod="e2e-models-e2e-worker-w2-ca6546/simulator-w2-ca6546-kserve-5959bb649b-5x6tf"
Nov 02 03:09:57.605045 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:57.605011    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fd6a8f7-0b10-4009-a6db-de0a01aca2b8-tmp-dir\") pod \"simulator-w2-ca6546-kserve-5959bb649b-5x6tf\" (UID: \"4fd6a8f7-0b10-4009-a6db-de0a01aca2b8\") " pod="e2e-models-e2e-worker-w2-ca6546/simulator-w2-ca6546-kserve-5959bb649b-5x6tf"
Nov 02 03:09:57.605165 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:57.605151    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fd6a8f7-0b10-4009-a6db-de0a01aca2b8-home\") pod \"simulator-w2-ca6546-kserve-5959bb649b-5x6tf\" (UID: \"4fd6a8f7-0b10-4009-a6db-de0a01aca2b8\") " pod="e2e-models-e2e-worker-w2-ca6546/simulator-w2-ca6546-kserve-5959bb649b-5x6tf"
Nov 02 03:09:57.606695 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:57.606670    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fd6a8f7-0b10-4009-a6db-de0a01aca2b8-dshm\") pod \"simulator-w2-ca6546-kserve-5959bb649b-5x6tf\" (UID: \"4fd6a8f7-0b10-4009-a6db-de0a01aca2b8\") " pod="e2e-models-e2e-worker-w2-ca6546/simulator-w2-ca6546-kserve-5959bb649b-5x6tf"
Nov 02 03:09:57.607485 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:57.607464    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fd6a8f7-0b10-4009-a6db-de0a01aca2b8-tls-certs\") pod \"simulator-w2-ca6546-kserve-5959bb649b-5x6tf\" (UID: \"4fd6a8f7-0b10-4009-a6db-de0a01aca2b8\") " pod="e2e-models-e2e-worker-w2-ca6546/simulator-w2-ca6546-kserve-5959bb649b-5x6tf"
Nov 02 03:09:57.661438 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:57.661408    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7zxp\" (UniqueName: \"kubernetes.io/projected/4fd6a8f7-0b10-4009-a6db-de0a01aca2b8-kube-api-access-n7zxp\") pod \"simulator-w2-ca6546-kserve-5959bb649b-5x6tf\" (UID: \"4fd6a8f7-0b10-4009-a6db-de0a01aca2b8\") " pod="e2e-models-e2e-worker-w2-ca6546/simulator-w2-ca6546-kserve-5959bb649b-5x6tf"
Nov 02 03:09:57.752774 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:57.752590    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-ca6546/simulator-w2-ca6546-kserve-5959bb649b-5x6tf"
Nov 02 03:09:57.765252 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:57.765228    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1f6e39-6d697b7999-7nsr8"
Nov 02 03:09:58.129213 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:58.129183    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-ca6546/simulator-w2-ca6546-kserve-5959bb649b-5x6tf"]
Nov 02 03:09:58.129600 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:09:58.129576    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4fd6a8f7_0b10_4009_a6db_de0a01aca2b8.slice/crio-95bf30c0c02bd5edbfa86047913f2a33dbf65d31775826e5b77ffdba3a035500 WatchSource:0}: Error finding container 95bf30c0c02bd5edbfa86047913f2a33dbf65d31775826e5b77ffdba3a035500: Status 404 returned error can't find the container with id 95bf30c0c02bd5edbfa86047913f2a33dbf65d31775826e5b77ffdba3a035500
Nov 02 03:09:58.771354 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:58.771267    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-1f6e39-69f9854f95-b4c6w"
Nov 02 03:09:58.839004 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:58.838966    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-ca6546/simulator-w2-ca6546-kserve-5959bb649b-5x6tf" event={"ID":"4fd6a8f7-0b10-4009-a6db-de0a01aca2b8","Type":"ContainerStarted","Data":"b0946945efb6dcfac7ea268630e2fdc0154b942c5f158bb1a50e0810520623b2"}
Nov 02 03:09:58.839197 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:58.839028    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-ca6546/simulator-w2-ca6546-kserve-5959bb649b-5x6tf" event={"ID":"4fd6a8f7-0b10-4009-a6db-de0a01aca2b8","Type":"ContainerStarted","Data":"95bf30c0c02bd5edbfa86047913f2a33dbf65d31775826e5b77ffdba3a035500"}
Nov 02 03:09:59.003001 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:59.002941    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-ca6546/simulator-w2-ca6546-kserve-5959bb649b-5x6tf" podStartSLOduration=2.002926542 podStartE2EDuration="2.002926542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:09:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:09:59.000786685 +0000 UTC m=+1339.579394085" watchObservedRunningTime="2026-11-02 03:09:59.002926542 +0000 UTC m=+1339.581533924"
Nov 02 03:09:59.752928 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:09:59.752897    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-ca6546/simulator-w2-ca6546-kserve-5959bb649b-5x6tf"
Nov 02 03:10:03.528572 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:10:03.528536    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 03:10:07.753567 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:10:07.753532    4496 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-ca6546/simulator-w2-ca6546-kserve-5959bb649b-5x6tf"
Nov 02 03:10:07.767153 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:10:07.767123    4496 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-ca6546/simulator-w2-ca6546-kserve-5959bb649b-5x6tf"
Nov 02 03:10:07.904237 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:10:07.904199    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-ca6546/simulator-w2-ca6546-kserve-5959bb649b-5x6tf"
Nov 02 03:10:28.886081 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:10:28.886037    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-398828/simulator-w4-398828-kserve-6d5bd49c6f-9tw2c"]
Nov 02 03:10:28.892153 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:10:28.892118    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-398828/simulator-w4-398828-kserve-6d5bd49c6f-9tw2c"
Nov 02 03:10:28.948039 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:10:28.947359    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-398828/simulator-w4-398828-kserve-6d5bd49c6f-9tw2c"]
Nov 02 03:10:28.993473 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:10:28.993439    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-398828\"/\"kube-root-ca.crt\""
Nov 02 03:10:28.993473 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:10:28.993441    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-398828\"/\"default-dockercfg-n7lmt\""
Nov 02 03:10:29.002103 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:10:29.002076    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-398828\"/\"openshift-service-ca.crt\""
Nov 02 03:10:29.002228 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:10:29.002104    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-398828\"/\"simulator-w4-398828-kserve-self-signed-certs\""
Nov 02 03:10:29.047456 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:10:29.047412    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b6aecfa-3e3b-4fb3-a80c-35b77de60ce2-model-cache\") pod \"simulator-w4-398828-kserve-6d5bd49c6f-9tw2c\" (UID: \"5b6aecfa-3e3b-4fb3-a80c-35b77de60ce2\") " pod="e2e-models-e2e-worker-w4-398828/simulator-w4-398828-kserve-6d5bd49c6f-9tw2c"
Nov 02 03:10:29.047665 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:10:29.047489    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b6aecfa-3e3b-4fb3-a80c-35b77de60ce2-dshm\") pod \"simulator-w4-398828-kserve-6d5bd49c6f-9tw2c\" (UID: \"5b6aecfa-3e3b-4fb3-a80c-35b77de60ce2\") " pod="e2e-models-e2e-worker-w4-398828/simulator-w4-398828-kserve-6d5bd49c6f-9tw2c"
Nov 02 03:10:29.048057 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:10:29.048027    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rh2f2\" (UniqueName: \"kubernetes.io/projected/5b6aecfa-3e3b-4fb3-a80c-35b77de60ce2-kube-api-access-rh2f2\") pod \"simulator-w4-398828-kserve-6d5bd49c6f-9tw2c\" (UID: \"5b6aecfa-3e3b-4fb3-a80c-35b77de60ce2\") " pod="e2e-models-e2e-worker-w4-398828/simulator-w4-398828-kserve-6d5bd49c6f-9tw2c"
Nov 02 03:10:29.048185 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:10:29.048167    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b6aecfa-3e3b-4fb3-a80c-35b77de60ce2-home\") pod \"simulator-w4-398828-kserve-6d5bd49c6f-9tw2c\" (UID: \"5b6aecfa-3e3b-4fb3-a80c-35b77de60ce2\") " pod="e2e-models-e2e-worker-w4-398828/simulator-w4-398828-kserve-6d5bd49c6f-9tw2c"
Nov 02 03:10:29.048334 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:10:29.048298    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b6aecfa-3e3b-4fb3-a80c-35b77de60ce2-tmp-dir\") pod \"simulator-w4-398828-kserve-6d5bd49c6f-9tw2c\" (UID: \"5b6aecfa-3e3b-4fb3-a80c-35b77de60ce2\") " pod="e2e-models-e2e-worker-w4-398828/simulator-w4-398828-kserve-6d5bd49c6f-9tw2c"
Nov 02 03:10:29.048497 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:10:29.048345    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b6aecfa-3e3b-4fb3-a80c-35b77de60ce2-tls-certs\") pod \"simulator-w4-398828-kserve-6d5bd49c6f-9tw2c\" (UID: \"5b6aecfa-3e3b-4fb3-a80c-35b77de60ce2\") " pod="e2e-models-e2e-worker-w4-398828/simulator-w4-398828-kserve-6d5bd49c6f-9tw2c"
Nov 02 03:10:29.149780 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:10:29.149692    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b6aecfa-3e3b-4fb3-a80c-35b77de60ce2-tmp-dir\") pod \"simulator-w4-398828-kserve-6d5bd49c6f-9tw2c\" (UID: \"5b6aecfa-3e3b-4fb3-a80c-35b77de60ce2\") " pod="e2e-models-e2e-worker-w4-398828/simulator-w4-398828-kserve-6d5bd49c6f-9tw2c"
Nov 02 03:10:29.149780 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:10:29.149734    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b6aecfa-3e3b-4fb3-a80c-35b77de60ce2-tls-certs\") pod \"simulator-w4-398828-kserve-6d5bd49c6f-9tw2c\" (UID: \"5b6aecfa-3e3b-4fb3-a80c-35b77de60ce2\") " pod="e2e-models-e2e-worker-w4-398828/simulator-w4-398828-kserve-6d5bd49c6f-9tw2c"
Nov 02 03:10:29.149780 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:10:29.149769    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b6aecfa-3e3b-4fb3-a80c-35b77de60ce2-model-cache\") pod \"simulator-w4-398828-kserve-6d5bd49c6f-9tw2c\" (UID: \"5b6aecfa-3e3b-4fb3-a80c-35b77de60ce2\") " pod="e2e-models-e2e-worker-w4-398828/simulator-w4-398828-kserve-6d5bd49c6f-9tw2c"
Nov 02 03:10:29.150085 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:10:29.149874    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b6aecfa-3e3b-4fb3-a80c-35b77de60ce2-dshm\") pod \"simulator-w4-398828-kserve-6d5bd49c6f-9tw2c\" (UID: \"5b6aecfa-3e3b-4fb3-a80c-35b77de60ce2\") " pod="e2e-models-e2e-worker-w4-398828/simulator-w4-398828-kserve-6d5bd49c6f-9tw2c"
Nov 02 03:10:29.150085 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:10:29.149926    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rh2f2\" (UniqueName: \"kubernetes.io/projected/5b6aecfa-3e3b-4fb3-a80c-35b77de60ce2-kube-api-access-rh2f2\") pod \"simulator-w4-398828-kserve-6d5bd49c6f-9tw2c\" (UID: \"5b6aecfa-3e3b-4fb3-a80c-35b77de60ce2\") " pod="e2e-models-e2e-worker-w4-398828/simulator-w4-398828-kserve-6d5bd49c6f-9tw2c"
Nov 02 03:10:29.150085 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:10:29.150044    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b6aecfa-3e3b-4fb3-a80c-35b77de60ce2-home\") pod \"simulator-w4-398828-kserve-6d5bd49c6f-9tw2c\" (UID: \"5b6aecfa-3e3b-4fb3-a80c-35b77de60ce2\") " pod="e2e-models-e2e-worker-w4-398828/simulator-w4-398828-kserve-6d5bd49c6f-9tw2c"
Nov 02 03:10:29.150258 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:10:29.150231    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b6aecfa-3e3b-4fb3-a80c-35b77de60ce2-tmp-dir\") pod \"simulator-w4-398828-kserve-6d5bd49c6f-9tw2c\" (UID: \"5b6aecfa-3e3b-4fb3-a80c-35b77de60ce2\") " pod="e2e-models-e2e-worker-w4-398828/simulator-w4-398828-kserve-6d5bd49c6f-9tw2c"
Nov 02 03:10:29.150388 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:10:29.150364    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b6aecfa-3e3b-4fb3-a80c-35b77de60ce2-home\") pod \"simulator-w4-398828-kserve-6d5bd49c6f-9tw2c\" (UID: \"5b6aecfa-3e3b-4fb3-a80c-35b77de60ce2\") " pod="e2e-models-e2e-worker-w4-398828/simulator-w4-398828-kserve-6d5bd49c6f-9tw2c"
Nov 02 03:10:29.150481 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:10:29.150463    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b6aecfa-3e3b-4fb3-a80c-35b77de60ce2-model-cache\") pod \"simulator-w4-398828-kserve-6d5bd49c6f-9tw2c\" (UID: \"5b6aecfa-3e3b-4fb3-a80c-35b77de60ce2\") " pod="e2e-models-e2e-worker-w4-398828/simulator-w4-398828-kserve-6d5bd49c6f-9tw2c"
Nov 02 03:10:29.152077 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:10:29.152054    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b6aecfa-3e3b-4fb3-a80c-35b77de60ce2-dshm\") pod \"simulator-w4-398828-kserve-6d5bd49c6f-9tw2c\" (UID: \"5b6aecfa-3e3b-4fb3-a80c-35b77de60ce2\") " pod="e2e-models-e2e-worker-w4-398828/simulator-w4-398828-kserve-6d5bd49c6f-9tw2c"
Nov 02 03:10:29.152378 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:10:29.152359    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b6aecfa-3e3b-4fb3-a80c-35b77de60ce2-tls-certs\") pod \"simulator-w4-398828-kserve-6d5bd49c6f-9tw2c\" (UID: \"5b6aecfa-3e3b-4fb3-a80c-35b77de60ce2\") " pod="e2e-models-e2e-worker-w4-398828/simulator-w4-398828-kserve-6d5bd49c6f-9tw2c"
Nov 02 03:10:29.210217 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:10:29.210187    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rh2f2\" (UniqueName: \"kubernetes.io/projected/5b6aecfa-3e3b-4fb3-a80c-35b77de60ce2-kube-api-access-rh2f2\") pod \"simulator-w4-398828-kserve-6d5bd49c6f-9tw2c\" (UID: \"5b6aecfa-3e3b-4fb3-a80c-35b77de60ce2\") " pod="e2e-models-e2e-worker-w4-398828/simulator-w4-398828-kserve-6d5bd49c6f-9tw2c"
Nov 02 03:10:29.505832 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:10:29.505806    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-398828/simulator-w4-398828-kserve-6d5bd49c6f-9tw2c"
Nov 02 03:10:29.883645 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:10:29.883620    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-398828/simulator-w4-398828-kserve-6d5bd49c6f-9tw2c"]
Nov 02 03:10:29.884603 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:10:29.884581    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5b6aecfa_3e3b_4fb3_a80c_35b77de60ce2.slice/crio-eb5bc5b6e8df486632e56587f8ae2368638b08630c2403aafc614dd77c7d2510 WatchSource:0}: Error finding container eb5bc5b6e8df486632e56587f8ae2368638b08630c2403aafc614dd77c7d2510: Status 404 returned error can't find the container with id eb5bc5b6e8df486632e56587f8ae2368638b08630c2403aafc614dd77c7d2510
Nov 02 03:10:30.034413 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:10:30.034377    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-398828/simulator-w4-398828-kserve-6d5bd49c6f-9tw2c" event={"ID":"5b6aecfa-3e3b-4fb3-a80c-35b77de60ce2","Type":"ContainerStarted","Data":"3931bdaf7ea78a43290417fc714792f90b1af645570ea3a027e5faa7ccee9639"}
Nov 02 03:10:30.034928 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:10:30.034426    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-398828/simulator-w4-398828-kserve-6d5bd49c6f-9tw2c" event={"ID":"5b6aecfa-3e3b-4fb3-a80c-35b77de60ce2","Type":"ContainerStarted","Data":"eb5bc5b6e8df486632e56587f8ae2368638b08630c2403aafc614dd77c7d2510"}
Nov 02 03:10:30.149047 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:10:30.148943    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-398828/simulator-w4-398828-kserve-6d5bd49c6f-9tw2c" podStartSLOduration=2.148928004 podStartE2EDuration="2.148928004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:10:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:10:30.148690148 +0000 UTC m=+1370.727297528" watchObservedRunningTime="2026-11-02 03:10:30.148928004 +0000 UTC m=+1370.727535384"
Nov 02 03:10:30.506850 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:10:30.506818    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-398828/simulator-w4-398828-kserve-6d5bd49c6f-9tw2c"
Nov 02 03:10:33.529549 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:10:33.529515    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 03:10:39.507007 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:10:39.506963    4496 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-398828/simulator-w4-398828-kserve-6d5bd49c6f-9tw2c"
Nov 02 03:10:39.520308 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:10:39.520280    4496 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-398828/simulator-w4-398828-kserve-6d5bd49c6f-9tw2c"
Nov 02 03:10:40.129060 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:10:40.129032    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-398828/simulator-w4-398828-kserve-6d5bd49c6f-9tw2c"
Nov 02 03:11:03.527441 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:11:03.527405    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 03:11:33.529665 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:11:33.529623    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 03:12:01.413212 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:01.413170    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29677b55-wmfd7"]
Nov 02 03:12:01.417818 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:01.417793    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29677b55-wmfd7"
Nov 02 03:12:01.471676 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:01.471651    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29677b55-wmfd7"]
Nov 02 03:12:01.496431 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:01.496396    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04fb8775-f59b-44c4-a384-92f3d3a09497-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29677b55-wmfd7\" (UID: \"04fb8775-f59b-44c4-a384-92f3d3a09497\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29677b55-wmfd7"
Nov 02 03:12:01.496590 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:01.496453    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rt4wn\" (UniqueName: \"kubernetes.io/projected/04fb8775-f59b-44c4-a384-92f3d3a09497-kube-api-access-rt4wn\") pod \"maas-api-revoke-sub-29677b55-wmfd7\" (UID: \"04fb8775-f59b-44c4-a384-92f3d3a09497\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29677b55-wmfd7"
Nov 02 03:12:01.496677 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:01.496654    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04fb8775-f59b-44c4-a384-92f3d3a09497-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29677b55-wmfd7\" (UID: \"04fb8775-f59b-44c4-a384-92f3d3a09497\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29677b55-wmfd7"
Nov 02 03:12:01.519296 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:01.519273    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 03:12:01.519296 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:01.519287    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9z5bk\""
Nov 02 03:12:01.519440 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:01.519273    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 03:12:01.521653 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:01.521638    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 03:12:01.598064 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:01.598027    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04fb8775-f59b-44c4-a384-92f3d3a09497-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29677b55-wmfd7\" (UID: \"04fb8775-f59b-44c4-a384-92f3d3a09497\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29677b55-wmfd7"
Nov 02 03:12:01.598241 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:01.598091    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rt4wn\" (UniqueName: \"kubernetes.io/projected/04fb8775-f59b-44c4-a384-92f3d3a09497-kube-api-access-rt4wn\") pod \"maas-api-revoke-sub-29677b55-wmfd7\" (UID: \"04fb8775-f59b-44c4-a384-92f3d3a09497\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29677b55-wmfd7"
Nov 02 03:12:01.598241 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:01.598186    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04fb8775-f59b-44c4-a384-92f3d3a09497-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29677b55-wmfd7\" (UID: \"04fb8775-f59b-44c4-a384-92f3d3a09497\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29677b55-wmfd7"
Nov 02 03:12:01.598703 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:01.598680    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04fb8775-f59b-44c4-a384-92f3d3a09497-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29677b55-wmfd7\" (UID: \"04fb8775-f59b-44c4-a384-92f3d3a09497\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29677b55-wmfd7"
Nov 02 03:12:01.600461 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:01.600443    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04fb8775-f59b-44c4-a384-92f3d3a09497-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29677b55-wmfd7\" (UID: \"04fb8775-f59b-44c4-a384-92f3d3a09497\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29677b55-wmfd7"
Nov 02 03:12:01.656271 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:01.656240    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rt4wn\" (UniqueName: \"kubernetes.io/projected/04fb8775-f59b-44c4-a384-92f3d3a09497-kube-api-access-rt4wn\") pod \"maas-api-revoke-sub-29677b55-wmfd7\" (UID: \"04fb8775-f59b-44c4-a384-92f3d3a09497\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29677b55-wmfd7"
Nov 02 03:12:01.731905 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:01.731876    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29677b55-wmfd7"
Nov 02 03:12:02.107710 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:02.107676    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29677b55-wmfd7"]
Nov 02 03:12:02.108444 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:12:02.108419    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04fb8775_f59b_44c4_a384_92f3d3a09497.slice/crio-06bb20ccbee223c82c320939e042961fb18b6fcef4cbebc594b4e41da75ed6ee WatchSource:0}: Error finding container 06bb20ccbee223c82c320939e042961fb18b6fcef4cbebc594b4e41da75ed6ee: Status 404 returned error can't find the container with id 06bb20ccbee223c82c320939e042961fb18b6fcef4cbebc594b4e41da75ed6ee
Nov 02 03:12:02.678197 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:02.678165    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29677b55-wmfd7" event={"ID":"04fb8775-f59b-44c4-a384-92f3d3a09497","Type":"ContainerStarted","Data":"0b99df6099e2b1f6703db46b3c34de2ab0b75c354bded6ecfcb1fee97818b1f0"}
Nov 02 03:12:02.678576 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:02.678229    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29677b55-wmfd7" event={"ID":"04fb8775-f59b-44c4-a384-92f3d3a09497","Type":"ContainerStarted","Data":"06bb20ccbee223c82c320939e042961fb18b6fcef4cbebc594b4e41da75ed6ee"}
Nov 02 03:12:02.794987 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:02.794934    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29677b55-wmfd7" podStartSLOduration=1.794921626 podStartE2EDuration="1.794921626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:12:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:12:02.791376922 +0000 UTC m=+1463.369984304" watchObservedRunningTime="2026-11-02 03:12:02.794921626 +0000 UTC m=+1463.373529005"
Nov 02 03:12:03.526062 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:12:03.526007    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 03:12:03.686009 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:03.685977    4496 generic.go:356] "Generic (PLEG): container finished" podID="04fb8775-f59b-44c4-a384-92f3d3a09497" containerID="0b99df6099e2b1f6703db46b3c34de2ab0b75c354bded6ecfcb1fee97818b1f0" exitCode=0
Nov 02 03:12:03.686516 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:03.686065    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29677b55-wmfd7" event={"ID":"04fb8775-f59b-44c4-a384-92f3d3a09497","Type":"ContainerDied","Data":"0b99df6099e2b1f6703db46b3c34de2ab0b75c354bded6ecfcb1fee97818b1f0"}
Nov 02 03:12:04.842001 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:04.841974    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29677b55-wmfd7"
Nov 02 03:12:04.931711 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:04.931680    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04fb8775-f59b-44c4-a384-92f3d3a09497-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04fb8775-f59b-44c4-a384-92f3d3a09497-maas-api-openshift-ca\") pod \"04fb8775-f59b-44c4-a384-92f3d3a09497\" (UID: \"04fb8775-f59b-44c4-a384-92f3d3a09497\") "
Nov 02 03:12:04.931892 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:04.931765    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04fb8775-f59b-44c4-a384-92f3d3a09497-kube-api-access-rt4wn\" (UniqueName: \"kubernetes.io/projected/04fb8775-f59b-44c4-a384-92f3d3a09497-kube-api-access-rt4wn\") pod \"04fb8775-f59b-44c4-a384-92f3d3a09497\" (UID: \"04fb8775-f59b-44c4-a384-92f3d3a09497\") "
Nov 02 03:12:04.931892 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:04.931788    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04fb8775-f59b-44c4-a384-92f3d3a09497-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04fb8775-f59b-44c4-a384-92f3d3a09497-maas-api-xks-ca\") pod \"04fb8775-f59b-44c4-a384-92f3d3a09497\" (UID: \"04fb8775-f59b-44c4-a384-92f3d3a09497\") "
Nov 02 03:12:04.932120 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:04.932096    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04fb8775-f59b-44c4-a384-92f3d3a09497-maas-api-openshift-ca" pod "04fb8775-f59b-44c4-a384-92f3d3a09497" (UID: "04fb8775-f59b-44c4-a384-92f3d3a09497"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:12:04.933840 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:04.933809    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04fb8775-f59b-44c4-a384-92f3d3a09497-maas-api-xks-ca" pod "04fb8775-f59b-44c4-a384-92f3d3a09497" (UID: "04fb8775-f59b-44c4-a384-92f3d3a09497"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 03:12:04.934010 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:04.933988    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04fb8775-f59b-44c4-a384-92f3d3a09497-kube-api-access-rt4wn" pod "04fb8775-f59b-44c4-a384-92f3d3a09497" (UID: "04fb8775-f59b-44c4-a384-92f3d3a09497"). InnerVolumeSpecName "kube-api-access-rt4wn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:12:05.033479 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:05.033446    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04fb8775-f59b-44c4-a384-92f3d3a09497-maas-api-openshift-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:12:05.033479 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:05.033475    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rt4wn\" (UniqueName: \"kubernetes.io/projected/04fb8775-f59b-44c4-a384-92f3d3a09497-kube-api-access-rt4wn\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:12:05.033677 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:05.033492    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04fb8775-f59b-44c4-a384-92f3d3a09497-maas-api-xks-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:12:05.701345 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:05.701315    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29677b55-wmfd7"
Nov 02 03:12:05.701345 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:05.701323    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29677b55-wmfd7" event={"ID":"04fb8775-f59b-44c4-a384-92f3d3a09497","Type":"ContainerDied","Data":"06bb20ccbee223c82c320939e042961fb18b6fcef4cbebc594b4e41da75ed6ee"}
Nov 02 03:12:05.701538 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:05.701364    4496 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="06bb20ccbee223c82c320939e042961fb18b6fcef4cbebc594b4e41da75ed6ee"
Nov 02 03:12:18.995819 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:18.995782    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4ec0495-hdpkf"]
Nov 02 03:12:18.996318 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:18.996304    4496 cpu_manager.go:414] "containerMap: removing container" podUID="04fb8775-f59b-44c4-a384-92f3d3a09497" containerName="revoke-keys"
Nov 02 03:12:18.996318 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:18.996318    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04fb8775-f59b-44c4-a384-92f3d3a09497" containerName="revoke-keys"
Nov 02 03:12:18.996431 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:18.996421    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="04fb8775-f59b-44c4-a384-92f3d3a09497" containerName="revoke-keys"
Nov 02 03:12:18.999806 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:18.999787    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4ec0495-hdpkf"
Nov 02 03:12:19.054591 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:19.054560    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4ec0495-hdpkf"]
Nov 02 03:12:19.102546 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:19.102514    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 03:12:19.102546 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:19.102536    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 03:12:19.102789 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:19.102548    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 03:12:19.104055 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:19.104029    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9z5bk\""
Nov 02 03:12:19.176602 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:19.176564    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48727286-46ff-4aae-bfc5-69236842e26e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f4ec0495-hdpkf\" (UID: \"48727286-46ff-4aae-bfc5-69236842e26e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4ec0495-hdpkf"
Nov 02 03:12:19.176602 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:19.176597    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48727286-46ff-4aae-bfc5-69236842e26e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f4ec0495-hdpkf\" (UID: \"48727286-46ff-4aae-bfc5-69236842e26e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4ec0495-hdpkf"
Nov 02 03:12:19.176851 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:19.176765    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nx5rr\" (UniqueName: \"kubernetes.io/projected/48727286-46ff-4aae-bfc5-69236842e26e-kube-api-access-nx5rr\") pod \"maas-api-revoke-sub-f4ec0495-hdpkf\" (UID: \"48727286-46ff-4aae-bfc5-69236842e26e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4ec0495-hdpkf"
Nov 02 03:12:19.278446 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:19.278360    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nx5rr\" (UniqueName: \"kubernetes.io/projected/48727286-46ff-4aae-bfc5-69236842e26e-kube-api-access-nx5rr\") pod \"maas-api-revoke-sub-f4ec0495-hdpkf\" (UID: \"48727286-46ff-4aae-bfc5-69236842e26e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4ec0495-hdpkf"
Nov 02 03:12:19.278594 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:19.278454    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48727286-46ff-4aae-bfc5-69236842e26e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f4ec0495-hdpkf\" (UID: \"48727286-46ff-4aae-bfc5-69236842e26e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4ec0495-hdpkf"
Nov 02 03:12:19.278594 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:19.278472    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48727286-46ff-4aae-bfc5-69236842e26e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f4ec0495-hdpkf\" (UID: \"48727286-46ff-4aae-bfc5-69236842e26e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4ec0495-hdpkf"
Nov 02 03:12:19.279194 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:19.279171    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48727286-46ff-4aae-bfc5-69236842e26e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f4ec0495-hdpkf\" (UID: \"48727286-46ff-4aae-bfc5-69236842e26e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4ec0495-hdpkf"
Nov 02 03:12:19.280777 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:19.280756    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48727286-46ff-4aae-bfc5-69236842e26e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f4ec0495-hdpkf\" (UID: \"48727286-46ff-4aae-bfc5-69236842e26e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4ec0495-hdpkf"
Nov 02 03:12:19.335827 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:19.335792    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nx5rr\" (UniqueName: \"kubernetes.io/projected/48727286-46ff-4aae-bfc5-69236842e26e-kube-api-access-nx5rr\") pod \"maas-api-revoke-sub-f4ec0495-hdpkf\" (UID: \"48727286-46ff-4aae-bfc5-69236842e26e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4ec0495-hdpkf"
Nov 02 03:12:19.613346 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:19.613257    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4ec0495-hdpkf"
Nov 02 03:12:19.991483 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:19.991451    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4ec0495-hdpkf"]
Nov 02 03:12:19.992486 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:12:19.992463    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48727286_46ff_4aae_bfc5_69236842e26e.slice/crio-719ed12fae7b1054726df10d88cccd8fa07b5a66c93c13d2328339eea449cd42 WatchSource:0}: Error finding container 719ed12fae7b1054726df10d88cccd8fa07b5a66c93c13d2328339eea449cd42: Status 404 returned error can't find the container with id 719ed12fae7b1054726df10d88cccd8fa07b5a66c93c13d2328339eea449cd42
Nov 02 03:12:20.799121 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:20.798983    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4ec0495-hdpkf" event={"ID":"48727286-46ff-4aae-bfc5-69236842e26e","Type":"ContainerStarted","Data":"15cfb6c0f9c57eb6b52062aae2fd3f738e23cee30f38cc83d9eb6ae55524bafb"}
Nov 02 03:12:20.799121 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:20.799061    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4ec0495-hdpkf" event={"ID":"48727286-46ff-4aae-bfc5-69236842e26e","Type":"ContainerStarted","Data":"719ed12fae7b1054726df10d88cccd8fa07b5a66c93c13d2328339eea449cd42"}
Nov 02 03:12:20.918824 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:20.918778    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4ec0495-hdpkf" podStartSLOduration=2.918765046 podStartE2EDuration="2.918765046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:12:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:12:20.916659806 +0000 UTC m=+1481.495267185" watchObservedRunningTime="2026-11-02 03:12:20.918765046 +0000 UTC m=+1481.497372425"
Nov 02 03:12:21.807031 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:21.806973    4496 generic.go:356] "Generic (PLEG): container finished" podID="48727286-46ff-4aae-bfc5-69236842e26e" containerID="15cfb6c0f9c57eb6b52062aae2fd3f738e23cee30f38cc83d9eb6ae55524bafb" exitCode=0
Nov 02 03:12:21.807406 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:21.807059    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4ec0495-hdpkf" event={"ID":"48727286-46ff-4aae-bfc5-69236842e26e","Type":"ContainerDied","Data":"15cfb6c0f9c57eb6b52062aae2fd3f738e23cee30f38cc83d9eb6ae55524bafb"}
Nov 02 03:12:23.137704 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:23.137675    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4ec0495-hdpkf"
Nov 02 03:12:23.227538 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:23.227502    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48727286-46ff-4aae-bfc5-69236842e26e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48727286-46ff-4aae-bfc5-69236842e26e-maas-api-xks-ca\") pod \"48727286-46ff-4aae-bfc5-69236842e26e\" (UID: \"48727286-46ff-4aae-bfc5-69236842e26e\") "
Nov 02 03:12:23.227743 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:23.227553    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48727286-46ff-4aae-bfc5-69236842e26e-kube-api-access-nx5rr\" (UniqueName: \"kubernetes.io/projected/48727286-46ff-4aae-bfc5-69236842e26e-kube-api-access-nx5rr\") pod \"48727286-46ff-4aae-bfc5-69236842e26e\" (UID: \"48727286-46ff-4aae-bfc5-69236842e26e\") "
Nov 02 03:12:23.227743 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:23.227724    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48727286-46ff-4aae-bfc5-69236842e26e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48727286-46ff-4aae-bfc5-69236842e26e-maas-api-openshift-ca\") pod \"48727286-46ff-4aae-bfc5-69236842e26e\" (UID: \"48727286-46ff-4aae-bfc5-69236842e26e\") "
Nov 02 03:12:23.228139 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:23.228103    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48727286-46ff-4aae-bfc5-69236842e26e-maas-api-openshift-ca" pod "48727286-46ff-4aae-bfc5-69236842e26e" (UID: "48727286-46ff-4aae-bfc5-69236842e26e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:12:23.229817 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:23.229790    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48727286-46ff-4aae-bfc5-69236842e26e-kube-api-access-nx5rr" pod "48727286-46ff-4aae-bfc5-69236842e26e" (UID: "48727286-46ff-4aae-bfc5-69236842e26e"). InnerVolumeSpecName "kube-api-access-nx5rr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:12:23.230093 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:23.230069    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48727286-46ff-4aae-bfc5-69236842e26e-maas-api-xks-ca" pod "48727286-46ff-4aae-bfc5-69236842e26e" (UID: "48727286-46ff-4aae-bfc5-69236842e26e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 03:12:23.329403 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:23.329371    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48727286-46ff-4aae-bfc5-69236842e26e-maas-api-xks-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:12:23.329403 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:23.329399    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nx5rr\" (UniqueName: \"kubernetes.io/projected/48727286-46ff-4aae-bfc5-69236842e26e-kube-api-access-nx5rr\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:12:23.329403 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:23.329409    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48727286-46ff-4aae-bfc5-69236842e26e-maas-api-openshift-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:12:23.822291 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:23.822258    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4ec0495-hdpkf"
Nov 02 03:12:23.822455 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:23.822268    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4ec0495-hdpkf" event={"ID":"48727286-46ff-4aae-bfc5-69236842e26e","Type":"ContainerDied","Data":"719ed12fae7b1054726df10d88cccd8fa07b5a66c93c13d2328339eea449cd42"}
Nov 02 03:12:23.822455 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:23.822378    4496 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="719ed12fae7b1054726df10d88cccd8fa07b5a66c93c13d2328339eea449cd42"
Nov 02 03:12:33.526772 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:12:33.526736    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 03:12:35.267050 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:35.266997    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mfxjl"]
Nov 02 03:12:35.267612 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:35.267258    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mfxjl" podUID="2532dec7-68f1-4cf4-8cdb-a7c34c3b6438" containerName="istio-proxy" containerID="cri-o://684757fb4233cd1f6a1d5d82185f03bbe9657835e0ab3e5781658c8cf99056cf" gracePeriod=30
Nov 02 03:12:39.754357 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:39.754327    4496 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hww8q_5c797b08-a314-44ed-8026-8a0ce1f6c36f/ovn-acl-logging/0.log"
Nov 02 03:12:39.760349 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:39.760329    4496 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hww8q_5c797b08-a314-44ed-8026-8a0ce1f6c36f/ovn-acl-logging/0.log"
Nov 02 03:12:40.552222 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:40.552195    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mfxjl"
Nov 02 03:12:40.714674 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:40.714643    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2532dec7-68f1-4cf4-8cdb-a7c34c3b6438-kube-api-access-2kv4d\" (UniqueName: \"kubernetes.io/projected/2532dec7-68f1-4cf4-8cdb-a7c34c3b6438-kube-api-access-2kv4d\") pod \"2532dec7-68f1-4cf4-8cdb-a7c34c3b6438\" (UID: \"2532dec7-68f1-4cf4-8cdb-a7c34c3b6438\") "
Nov 02 03:12:40.714846 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:40.714681    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2532dec7-68f1-4cf4-8cdb-a7c34c3b6438-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2532dec7-68f1-4cf4-8cdb-a7c34c3b6438-istiod-ca-cert\") pod \"2532dec7-68f1-4cf4-8cdb-a7c34c3b6438\" (UID: \"2532dec7-68f1-4cf4-8cdb-a7c34c3b6438\") "
Nov 02 03:12:40.714846 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:40.714713    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2532dec7-68f1-4cf4-8cdb-a7c34c3b6438-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2532dec7-68f1-4cf4-8cdb-a7c34c3b6438-istio-podinfo\") pod \"2532dec7-68f1-4cf4-8cdb-a7c34c3b6438\" (UID: \"2532dec7-68f1-4cf4-8cdb-a7c34c3b6438\") "
Nov 02 03:12:40.714846 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:40.714782    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2532dec7-68f1-4cf4-8cdb-a7c34c3b6438-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2532dec7-68f1-4cf4-8cdb-a7c34c3b6438-workload-socket\") pod \"2532dec7-68f1-4cf4-8cdb-a7c34c3b6438\" (UID: \"2532dec7-68f1-4cf4-8cdb-a7c34c3b6438\") "
Nov 02 03:12:40.714846 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:40.714818    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2532dec7-68f1-4cf4-8cdb-a7c34c3b6438-istio-token\" (UniqueName: \"kubernetes.io/projected/2532dec7-68f1-4cf4-8cdb-a7c34c3b6438-istio-token\") pod \"2532dec7-68f1-4cf4-8cdb-a7c34c3b6438\" (UID: \"2532dec7-68f1-4cf4-8cdb-a7c34c3b6438\") "
Nov 02 03:12:40.715080 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:40.714853    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2532dec7-68f1-4cf4-8cdb-a7c34c3b6438-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2532dec7-68f1-4cf4-8cdb-a7c34c3b6438-istio-envoy\") pod \"2532dec7-68f1-4cf4-8cdb-a7c34c3b6438\" (UID: \"2532dec7-68f1-4cf4-8cdb-a7c34c3b6438\") "
Nov 02 03:12:40.715080 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:40.714999    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2532dec7-68f1-4cf4-8cdb-a7c34c3b6438-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2532dec7-68f1-4cf4-8cdb-a7c34c3b6438-workload-certs\") pod \"2532dec7-68f1-4cf4-8cdb-a7c34c3b6438\" (UID: \"2532dec7-68f1-4cf4-8cdb-a7c34c3b6438\") "
Nov 02 03:12:40.715192 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:40.715082    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2532dec7-68f1-4cf4-8cdb-a7c34c3b6438-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2532dec7-68f1-4cf4-8cdb-a7c34c3b6438-istio-data\") pod \"2532dec7-68f1-4cf4-8cdb-a7c34c3b6438\" (UID: \"2532dec7-68f1-4cf4-8cdb-a7c34c3b6438\") "
Nov 02 03:12:40.715192 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:40.715134    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2532dec7-68f1-4cf4-8cdb-a7c34c3b6438-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2532dec7-68f1-4cf4-8cdb-a7c34c3b6438-credential-socket\") pod \"2532dec7-68f1-4cf4-8cdb-a7c34c3b6438\" (UID: \"2532dec7-68f1-4cf4-8cdb-a7c34c3b6438\") "
Nov 02 03:12:40.715192 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:40.715129    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2532dec7-68f1-4cf4-8cdb-a7c34c3b6438-workload-socket" pod "2532dec7-68f1-4cf4-8cdb-a7c34c3b6438" (UID: "2532dec7-68f1-4cf4-8cdb-a7c34c3b6438"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:12:40.715336 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:40.715200    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2532dec7-68f1-4cf4-8cdb-a7c34c3b6438-istiod-ca-cert" pod "2532dec7-68f1-4cf4-8cdb-a7c34c3b6438" (UID: "2532dec7-68f1-4cf4-8cdb-a7c34c3b6438"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:12:40.715392 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:40.715363    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2532dec7-68f1-4cf4-8cdb-a7c34c3b6438-workload-certs" pod "2532dec7-68f1-4cf4-8cdb-a7c34c3b6438" (UID: "2532dec7-68f1-4cf4-8cdb-a7c34c3b6438"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:12:40.715458 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:40.715396    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2532dec7-68f1-4cf4-8cdb-a7c34c3b6438-istio-data" pod "2532dec7-68f1-4cf4-8cdb-a7c34c3b6438" (UID: "2532dec7-68f1-4cf4-8cdb-a7c34c3b6438"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:12:40.715966 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:40.715539    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2532dec7-68f1-4cf4-8cdb-a7c34c3b6438-credential-socket" pod "2532dec7-68f1-4cf4-8cdb-a7c34c3b6438" (UID: "2532dec7-68f1-4cf4-8cdb-a7c34c3b6438"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:12:40.715966 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:40.715872    4496 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2532dec7-68f1-4cf4-8cdb-a7c34c3b6438-workload-socket\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:12:40.715966 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:40.715896    4496 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2532dec7-68f1-4cf4-8cdb-a7c34c3b6438-workload-certs\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:12:40.715966 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:40.715912    4496 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2532dec7-68f1-4cf4-8cdb-a7c34c3b6438-istio-data\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:12:40.715966 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:40.715926    4496 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2532dec7-68f1-4cf4-8cdb-a7c34c3b6438-credential-socket\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:12:40.715966 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:40.715940    4496 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2532dec7-68f1-4cf4-8cdb-a7c34c3b6438-istiod-ca-cert\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:12:40.717129 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:40.717103    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2532dec7-68f1-4cf4-8cdb-a7c34c3b6438-istio-podinfo" pod "2532dec7-68f1-4cf4-8cdb-a7c34c3b6438" (UID: "2532dec7-68f1-4cf4-8cdb-a7c34c3b6438"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 03:12:40.717447 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:40.717425    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2532dec7-68f1-4cf4-8cdb-a7c34c3b6438-istio-token" pod "2532dec7-68f1-4cf4-8cdb-a7c34c3b6438" (UID: "2532dec7-68f1-4cf4-8cdb-a7c34c3b6438"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:12:40.717520 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:40.717428    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2532dec7-68f1-4cf4-8cdb-a7c34c3b6438-istio-envoy" pod "2532dec7-68f1-4cf4-8cdb-a7c34c3b6438" (UID: "2532dec7-68f1-4cf4-8cdb-a7c34c3b6438"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:12:40.717520 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:40.717449    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2532dec7-68f1-4cf4-8cdb-a7c34c3b6438-kube-api-access-2kv4d" pod "2532dec7-68f1-4cf4-8cdb-a7c34c3b6438" (UID: "2532dec7-68f1-4cf4-8cdb-a7c34c3b6438"). InnerVolumeSpecName "kube-api-access-2kv4d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:12:40.816678 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:40.816605    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kv4d\" (UniqueName: \"kubernetes.io/projected/2532dec7-68f1-4cf4-8cdb-a7c34c3b6438-kube-api-access-2kv4d\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:12:40.816678 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:40.816631    4496 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2532dec7-68f1-4cf4-8cdb-a7c34c3b6438-istio-podinfo\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:12:40.816678 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:40.816642    4496 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2532dec7-68f1-4cf4-8cdb-a7c34c3b6438-istio-token\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:12:40.816678 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:40.816652    4496 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2532dec7-68f1-4cf4-8cdb-a7c34c3b6438-istio-envoy\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:12:40.944663 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:40.944629    4496 generic.go:356] "Generic (PLEG): container finished" podID="2532dec7-68f1-4cf4-8cdb-a7c34c3b6438" containerID="684757fb4233cd1f6a1d5d82185f03bbe9657835e0ab3e5781658c8cf99056cf" exitCode=0
Nov 02 03:12:40.944848 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:40.944675    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mfxjl"
Nov 02 03:12:40.944848 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:40.944682    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mfxjl" event={"ID":"2532dec7-68f1-4cf4-8cdb-a7c34c3b6438","Type":"ContainerDied","Data":"684757fb4233cd1f6a1d5d82185f03bbe9657835e0ab3e5781658c8cf99056cf"}
Nov 02 03:12:40.944848 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:40.944735    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mfxjl" event={"ID":"2532dec7-68f1-4cf4-8cdb-a7c34c3b6438","Type":"ContainerDied","Data":"e2d2d2738a597a31d9ddf35119df19f6cfe0b9cfd983ac40980cdd22b05d818b"}
Nov 02 03:12:40.944848 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:40.944765    4496 scope.go:122] "RemoveContainer" containerID="684757fb4233cd1f6a1d5d82185f03bbe9657835e0ab3e5781658c8cf99056cf"
Nov 02 03:12:40.955103 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:40.955087    4496 scope.go:122] "RemoveContainer" containerID="684757fb4233cd1f6a1d5d82185f03bbe9657835e0ab3e5781658c8cf99056cf"
Nov 02 03:12:40.955326 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:12:40.955309    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"684757fb4233cd1f6a1d5d82185f03bbe9657835e0ab3e5781658c8cf99056cf\": container with ID starting with 684757fb4233cd1f6a1d5d82185f03bbe9657835e0ab3e5781658c8cf99056cf not found: ID does not exist" containerID="684757fb4233cd1f6a1d5d82185f03bbe9657835e0ab3e5781658c8cf99056cf"
Nov 02 03:12:40.955375 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:40.955333    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"684757fb4233cd1f6a1d5d82185f03bbe9657835e0ab3e5781658c8cf99056cf"} err="failed to get container status \"684757fb4233cd1f6a1d5d82185f03bbe9657835e0ab3e5781658c8cf99056cf\": rpc error: code = NotFound desc = could not find container \"684757fb4233cd1f6a1d5d82185f03bbe9657835e0ab3e5781658c8cf99056cf\": container with ID starting with 684757fb4233cd1f6a1d5d82185f03bbe9657835e0ab3e5781658c8cf99056cf not found: ID does not exist"
Nov 02 03:12:41.112711 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:41.112635    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mfxjl"]
Nov 02 03:12:41.163700 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:41.163670    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mfxjl"]
Nov 02 03:12:41.574646 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:41.574614    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2532dec7-68f1-4cf4-8cdb-a7c34c3b6438" path="/var/lib/kubelet/pods/2532dec7-68f1-4cf4-8cdb-a7c34c3b6438/volumes"
Nov 02 03:12:44.073935 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:44.073891    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec07dc13-jlg6j"]
Nov 02 03:12:44.074645 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:44.074623    4496 cpu_manager.go:414] "containerMap: removing container" podUID="2532dec7-68f1-4cf4-8cdb-a7c34c3b6438" containerName="istio-proxy"
Nov 02 03:12:44.074736 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:44.074647    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2532dec7-68f1-4cf4-8cdb-a7c34c3b6438" containerName="istio-proxy"
Nov 02 03:12:44.074736 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:44.074672    4496 cpu_manager.go:414] "containerMap: removing container" podUID="48727286-46ff-4aae-bfc5-69236842e26e" containerName="revoke-keys"
Nov 02 03:12:44.074736 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:44.074680    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48727286-46ff-4aae-bfc5-69236842e26e" containerName="revoke-keys"
Nov 02 03:12:44.074882 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:44.074824    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="48727286-46ff-4aae-bfc5-69236842e26e" containerName="revoke-keys"
Nov 02 03:12:44.074882 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:44.074847    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="2532dec7-68f1-4cf4-8cdb-a7c34c3b6438" containerName="istio-proxy"
Nov 02 03:12:44.079230 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:44.079210    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec07dc13-jlg6j"
Nov 02 03:12:44.141175 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:44.141138    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec07dc13-jlg6j"]
Nov 02 03:12:44.187282 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:44.187235    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 03:12:44.187282 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:44.187236    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 03:12:44.187282 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:44.187261    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 03:12:44.187584 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:44.187312    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9z5bk\""
Nov 02 03:12:44.252681 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:44.252637    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6nxs\" (UniqueName: \"kubernetes.io/projected/e7b60965-efe4-4d29-baa5-2e14bc094e1b-kube-api-access-v6nxs\") pod \"maas-api-revoke-sub-ec07dc13-jlg6j\" (UID: \"e7b60965-efe4-4d29-baa5-2e14bc094e1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec07dc13-jlg6j"
Nov 02 03:12:44.252852 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:44.252698    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7b60965-efe4-4d29-baa5-2e14bc094e1b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec07dc13-jlg6j\" (UID: \"e7b60965-efe4-4d29-baa5-2e14bc094e1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec07dc13-jlg6j"
Nov 02 03:12:44.252852 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:44.252767    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7b60965-efe4-4d29-baa5-2e14bc094e1b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec07dc13-jlg6j\" (UID: \"e7b60965-efe4-4d29-baa5-2e14bc094e1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec07dc13-jlg6j"
Nov 02 03:12:44.353365 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:44.353280    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7b60965-efe4-4d29-baa5-2e14bc094e1b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec07dc13-jlg6j\" (UID: \"e7b60965-efe4-4d29-baa5-2e14bc094e1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec07dc13-jlg6j"
Nov 02 03:12:44.353365 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:44.353328    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7b60965-efe4-4d29-baa5-2e14bc094e1b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec07dc13-jlg6j\" (UID: \"e7b60965-efe4-4d29-baa5-2e14bc094e1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec07dc13-jlg6j"
Nov 02 03:12:44.353594 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:44.353454    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6nxs\" (UniqueName: \"kubernetes.io/projected/e7b60965-efe4-4d29-baa5-2e14bc094e1b-kube-api-access-v6nxs\") pod \"maas-api-revoke-sub-ec07dc13-jlg6j\" (UID: \"e7b60965-efe4-4d29-baa5-2e14bc094e1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec07dc13-jlg6j"
Nov 02 03:12:44.354077 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:44.354056    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7b60965-efe4-4d29-baa5-2e14bc094e1b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec07dc13-jlg6j\" (UID: \"e7b60965-efe4-4d29-baa5-2e14bc094e1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec07dc13-jlg6j"
Nov 02 03:12:44.355771 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:44.355744    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7b60965-efe4-4d29-baa5-2e14bc094e1b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec07dc13-jlg6j\" (UID: \"e7b60965-efe4-4d29-baa5-2e14bc094e1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec07dc13-jlg6j"
Nov 02 03:12:44.411053 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:44.411010    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6nxs\" (UniqueName: \"kubernetes.io/projected/e7b60965-efe4-4d29-baa5-2e14bc094e1b-kube-api-access-v6nxs\") pod \"maas-api-revoke-sub-ec07dc13-jlg6j\" (UID: \"e7b60965-efe4-4d29-baa5-2e14bc094e1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec07dc13-jlg6j"
Nov 02 03:12:44.456480 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:44.456445    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-181e39b9-mhtxr"]
Nov 02 03:12:44.461318 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:44.461291    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-181e39b9-mhtxr"
Nov 02 03:12:44.516094 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:44.516060    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-181e39b9-mhtxr"]
Nov 02 03:12:44.555406 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:44.555373    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c2c1ce9-e204-4204-9793-7d46595601ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-181e39b9-mhtxr\" (UID: \"9c2c1ce9-e204-4204-9793-7d46595601ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-181e39b9-mhtxr"
Nov 02 03:12:44.555608 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:44.555429    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c2c1ce9-e204-4204-9793-7d46595601ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-181e39b9-mhtxr\" (UID: \"9c2c1ce9-e204-4204-9793-7d46595601ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-181e39b9-mhtxr"
Nov 02 03:12:44.555608 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:44.555448    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9vqg\" (UniqueName: \"kubernetes.io/projected/9c2c1ce9-e204-4204-9793-7d46595601ba-kube-api-access-d9vqg\") pod \"maas-api-revoke-sub-181e39b9-mhtxr\" (UID: \"9c2c1ce9-e204-4204-9793-7d46595601ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-181e39b9-mhtxr"
Nov 02 03:12:44.656490 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:44.656411    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c2c1ce9-e204-4204-9793-7d46595601ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-181e39b9-mhtxr\" (UID: \"9c2c1ce9-e204-4204-9793-7d46595601ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-181e39b9-mhtxr"
Nov 02 03:12:44.656490 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:44.656464    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c2c1ce9-e204-4204-9793-7d46595601ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-181e39b9-mhtxr\" (UID: \"9c2c1ce9-e204-4204-9793-7d46595601ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-181e39b9-mhtxr"
Nov 02 03:12:44.656673 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:44.656663    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9vqg\" (UniqueName: \"kubernetes.io/projected/9c2c1ce9-e204-4204-9793-7d46595601ba-kube-api-access-d9vqg\") pod \"maas-api-revoke-sub-181e39b9-mhtxr\" (UID: \"9c2c1ce9-e204-4204-9793-7d46595601ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-181e39b9-mhtxr"
Nov 02 03:12:44.657223 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:44.657201    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c2c1ce9-e204-4204-9793-7d46595601ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-181e39b9-mhtxr\" (UID: \"9c2c1ce9-e204-4204-9793-7d46595601ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-181e39b9-mhtxr"
Nov 02 03:12:44.658784 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:44.658762    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c2c1ce9-e204-4204-9793-7d46595601ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-181e39b9-mhtxr\" (UID: \"9c2c1ce9-e204-4204-9793-7d46595601ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-181e39b9-mhtxr"
Nov 02 03:12:44.692923 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:44.692896    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec07dc13-jlg6j"
Nov 02 03:12:44.707979 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:44.707950    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-jx2rx"]
Nov 02 03:12:44.712113 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:44.712089    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-jx2rx"
Nov 02 03:12:44.714518 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:44.714493    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9vqg\" (UniqueName: \"kubernetes.io/projected/9c2c1ce9-e204-4204-9793-7d46595601ba-kube-api-access-d9vqg\") pod \"maas-api-revoke-sub-181e39b9-mhtxr\" (UID: \"9c2c1ce9-e204-4204-9793-7d46595601ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-181e39b9-mhtxr"
Nov 02 03:12:44.772053 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:44.771186    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-ca6546-openshift-default-dockercfg-c2jsr\""
Nov 02 03:12:44.772633 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:44.772611    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-jx2rx"]
Nov 02 03:12:44.783958 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:44.783935    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-181e39b9-mhtxr"
Nov 02 03:12:44.859408 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:44.858839    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ecf68d6-0a44-436c-938f-e75696a37e6a-workload-socket\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-jx2rx\" (UID: \"2ecf68d6-0a44-436c-938f-e75696a37e6a\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-jx2rx"
Nov 02 03:12:44.859408 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:44.858902    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ecf68d6-0a44-436c-938f-e75696a37e6a-istio-token\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-jx2rx\" (UID: \"2ecf68d6-0a44-436c-938f-e75696a37e6a\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-jx2rx"
Nov 02 03:12:44.859408 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:44.858929    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ecf68d6-0a44-436c-938f-e75696a37e6a-istiod-ca-cert\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-jx2rx\" (UID: \"2ecf68d6-0a44-436c-938f-e75696a37e6a\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-jx2rx"
Nov 02 03:12:44.859408 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:44.858987    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ecf68d6-0a44-436c-938f-e75696a37e6a-istio-data\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-jx2rx\" (UID: \"2ecf68d6-0a44-436c-938f-e75696a37e6a\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-jx2rx"
Nov 02 03:12:44.859408 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:44.859041    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9h4pn\" (UniqueName: \"kubernetes.io/projected/2ecf68d6-0a44-436c-938f-e75696a37e6a-kube-api-access-9h4pn\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-jx2rx\" (UID: \"2ecf68d6-0a44-436c-938f-e75696a37e6a\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-jx2rx"
Nov 02 03:12:44.859408 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:44.859069    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ecf68d6-0a44-436c-938f-e75696a37e6a-istio-podinfo\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-jx2rx\" (UID: \"2ecf68d6-0a44-436c-938f-e75696a37e6a\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-jx2rx"
Nov 02 03:12:44.859408 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:44.859106    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ecf68d6-0a44-436c-938f-e75696a37e6a-credential-socket\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-jx2rx\" (UID: \"2ecf68d6-0a44-436c-938f-e75696a37e6a\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-jx2rx"
Nov 02 03:12:44.859408 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:44.859137    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ecf68d6-0a44-436c-938f-e75696a37e6a-workload-certs\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-jx2rx\" (UID: \"2ecf68d6-0a44-436c-938f-e75696a37e6a\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-jx2rx"
Nov 02 03:12:44.859408 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:44.859191    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ecf68d6-0a44-436c-938f-e75696a37e6a-istio-envoy\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-jx2rx\" (UID: \"2ecf68d6-0a44-436c-938f-e75696a37e6a\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-jx2rx"
Nov 02 03:12:44.960456 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:44.960382    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ecf68d6-0a44-436c-938f-e75696a37e6a-credential-socket\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-jx2rx\" (UID: \"2ecf68d6-0a44-436c-938f-e75696a37e6a\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-jx2rx"
Nov 02 03:12:44.960456 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:44.960422    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ecf68d6-0a44-436c-938f-e75696a37e6a-workload-certs\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-jx2rx\" (UID: \"2ecf68d6-0a44-436c-938f-e75696a37e6a\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-jx2rx"
Nov 02 03:12:44.960456 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:44.960455    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ecf68d6-0a44-436c-938f-e75696a37e6a-istio-envoy\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-jx2rx\" (UID: \"2ecf68d6-0a44-436c-938f-e75696a37e6a\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-jx2rx"
Nov 02 03:12:44.960700 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:44.960524    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ecf68d6-0a44-436c-938f-e75696a37e6a-workload-socket\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-jx2rx\" (UID: \"2ecf68d6-0a44-436c-938f-e75696a37e6a\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-jx2rx"
Nov 02 03:12:44.960700 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:44.960560    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ecf68d6-0a44-436c-938f-e75696a37e6a-istio-token\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-jx2rx\" (UID: \"2ecf68d6-0a44-436c-938f-e75696a37e6a\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-jx2rx"
Nov 02 03:12:44.960700 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:44.960578    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ecf68d6-0a44-436c-938f-e75696a37e6a-istiod-ca-cert\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-jx2rx\" (UID: \"2ecf68d6-0a44-436c-938f-e75696a37e6a\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-jx2rx"
Nov 02 03:12:44.960700 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:44.960630    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ecf68d6-0a44-436c-938f-e75696a37e6a-istio-data\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-jx2rx\" (UID: \"2ecf68d6-0a44-436c-938f-e75696a37e6a\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-jx2rx"
Nov 02 03:12:44.960700 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:44.960659    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9h4pn\" (UniqueName: \"kubernetes.io/projected/2ecf68d6-0a44-436c-938f-e75696a37e6a-kube-api-access-9h4pn\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-jx2rx\" (UID: \"2ecf68d6-0a44-436c-938f-e75696a37e6a\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-jx2rx"
Nov 02 03:12:44.960700 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:44.960684    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ecf68d6-0a44-436c-938f-e75696a37e6a-istio-podinfo\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-jx2rx\" (UID: \"2ecf68d6-0a44-436c-938f-e75696a37e6a\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-jx2rx"
Nov 02 03:12:44.960985 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:44.960770    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ecf68d6-0a44-436c-938f-e75696a37e6a-credential-socket\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-jx2rx\" (UID: \"2ecf68d6-0a44-436c-938f-e75696a37e6a\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-jx2rx"
Nov 02 03:12:44.960985 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:44.960821    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ecf68d6-0a44-436c-938f-e75696a37e6a-workload-certs\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-jx2rx\" (UID: \"2ecf68d6-0a44-436c-938f-e75696a37e6a\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-jx2rx"
Nov 02 03:12:44.960985 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:44.960962    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ecf68d6-0a44-436c-938f-e75696a37e6a-workload-socket\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-jx2rx\" (UID: \"2ecf68d6-0a44-436c-938f-e75696a37e6a\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-jx2rx"
Nov 02 03:12:44.961136 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:44.961062    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ecf68d6-0a44-436c-938f-e75696a37e6a-istio-data\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-jx2rx\" (UID: \"2ecf68d6-0a44-436c-938f-e75696a37e6a\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-jx2rx"
Nov 02 03:12:44.961320 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:44.961298    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ecf68d6-0a44-436c-938f-e75696a37e6a-istiod-ca-cert\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-jx2rx\" (UID: \"2ecf68d6-0a44-436c-938f-e75696a37e6a\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-jx2rx"
Nov 02 03:12:44.962774 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:44.962750    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ecf68d6-0a44-436c-938f-e75696a37e6a-istio-envoy\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-jx2rx\" (UID: \"2ecf68d6-0a44-436c-938f-e75696a37e6a\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-jx2rx"
Nov 02 03:12:44.962979 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:44.962962    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ecf68d6-0a44-436c-938f-e75696a37e6a-istio-podinfo\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-jx2rx\" (UID: \"2ecf68d6-0a44-436c-938f-e75696a37e6a\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-jx2rx"
Nov 02 03:12:45.017980 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:45.017948    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9h4pn\" (UniqueName: \"kubernetes.io/projected/2ecf68d6-0a44-436c-938f-e75696a37e6a-kube-api-access-9h4pn\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-jx2rx\" (UID: \"2ecf68d6-0a44-436c-938f-e75696a37e6a\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-jx2rx"
Nov 02 03:12:45.018517 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:45.018495    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ecf68d6-0a44-436c-938f-e75696a37e6a-istio-token\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-jx2rx\" (UID: \"2ecf68d6-0a44-436c-938f-e75696a37e6a\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-jx2rx"
Nov 02 03:12:45.036653 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:45.036631    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-jx2rx"
Nov 02 03:12:45.070049 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:45.069345    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec07dc13-jlg6j"]
Nov 02 03:12:45.070049 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:12:45.069562    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7b60965_efe4_4d29_baa5_2e14bc094e1b.slice/crio-f19160cb1dbdfa1e5b95af645dbc60740eaccafbc31b805e4ab675880fe949af WatchSource:0}: Error finding container f19160cb1dbdfa1e5b95af645dbc60740eaccafbc31b805e4ab675880fe949af: Status 404 returned error can't find the container with id f19160cb1dbdfa1e5b95af645dbc60740eaccafbc31b805e4ab675880fe949af
Nov 02 03:12:45.178107 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:45.178080    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-181e39b9-mhtxr"]
Nov 02 03:12:45.235288 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:12:45.235248    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c2c1ce9_e204_4204_9793_7d46595601ba.slice/crio-a142f5daf756667a2164919591c1add821458d3725c39dcefc4964202ebac900 WatchSource:0}: Error finding container a142f5daf756667a2164919591c1add821458d3725c39dcefc4964202ebac900: Status 404 returned error can't find the container with id a142f5daf756667a2164919591c1add821458d3725c39dcefc4964202ebac900
Nov 02 03:12:45.445855 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:45.445816    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-jx2rx"]
Nov 02 03:12:45.446426 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:12:45.446397    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ecf68d6_0a44_436c_938f_e75696a37e6a.slice/crio-43af5d8c6176e55c48b8923b427d27b0d3624b208d684a20a981cbde2f75f0f9 WatchSource:0}: Error finding container 43af5d8c6176e55c48b8923b427d27b0d3624b208d684a20a981cbde2f75f0f9: Status 404 returned error can't find the container with id 43af5d8c6176e55c48b8923b427d27b0d3624b208d684a20a981cbde2f75f0f9
Nov 02 03:12:45.449194 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:45.449163    4496 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 03:12:45.449304 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:45.449243    4496 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 03:12:45.449304 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:45.449288    4496 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 03:12:45.986083 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:45.986042    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-181e39b9-mhtxr" event={"ID":"9c2c1ce9-e204-4204-9793-7d46595601ba","Type":"ContainerStarted","Data":"6a8eae85c0e24b7350fc89dc3b43b487e6786840dc69056689ee8059e7d63fea"}
Nov 02 03:12:45.986382 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:45.986119    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-181e39b9-mhtxr" event={"ID":"9c2c1ce9-e204-4204-9793-7d46595601ba","Type":"ContainerStarted","Data":"a142f5daf756667a2164919591c1add821458d3725c39dcefc4964202ebac900"}
Nov 02 03:12:45.987999 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:45.987970    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-jx2rx" event={"ID":"2ecf68d6-0a44-436c-938f-e75696a37e6a","Type":"ContainerStarted","Data":"f9985833faa6c86d92e567ccfa70c3e7dc7bc99671ec8e3f78c5b4530250ad7a"}
Nov 02 03:12:45.988149 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:45.988032    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-jx2rx" event={"ID":"2ecf68d6-0a44-436c-938f-e75696a37e6a","Type":"ContainerStarted","Data":"43af5d8c6176e55c48b8923b427d27b0d3624b208d684a20a981cbde2f75f0f9"}
Nov 02 03:12:45.989844 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:45.989815    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec07dc13-jlg6j" event={"ID":"e7b60965-efe4-4d29-baa5-2e14bc094e1b","Type":"ContainerStarted","Data":"1b0a1c915f45520f1ba81be8c1d45c6f07c4f9590450fd6506c3fc9e027c6aff"}
Nov 02 03:12:45.989978 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:45.989869    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec07dc13-jlg6j" event={"ID":"e7b60965-efe4-4d29-baa5-2e14bc094e1b","Type":"ContainerStarted","Data":"f19160cb1dbdfa1e5b95af645dbc60740eaccafbc31b805e4ab675880fe949af"}
Nov 02 03:12:46.037208 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:46.037166    4496 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-jx2rx"
Nov 02 03:12:46.039128 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:46.039096    4496 patch_prober.go:29] interesting pod/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-jx2rx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Nov 02 03:12:46.039269 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:46.039153    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-jx2rx" podUID="2ecf68d6-0a44-436c-938f-e75696a37e6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Nov 02 03:12:46.101523 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:46.101463    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-181e39b9-mhtxr" podStartSLOduration=2.101449053 podStartE2EDuration="2.101449053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:12:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:12:46.100709967 +0000 UTC m=+1506.679317348" watchObservedRunningTime="2026-11-02 03:12:46.101449053 +0000 UTC m=+1506.680056457"
Nov 02 03:12:46.216251 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:46.216194    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec07dc13-jlg6j" podStartSLOduration=2.216178798 podStartE2EDuration="2.216178798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:12:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:12:46.213984606 +0000 UTC m=+1506.792591987" watchObservedRunningTime="2026-11-02 03:12:46.216178798 +0000 UTC m=+1506.794786218"
Nov 02 03:12:46.370040 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:46.363529    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-jx2rx" podStartSLOduration=2.363507785 podStartE2EDuration="2.363507785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:12:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:12:46.357668139 +0000 UTC m=+1506.936275523" watchObservedRunningTime="2026-11-02 03:12:46.363507785 +0000 UTC m=+1506.942115167"
Nov 02 03:12:46.997799 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:46.997764    4496 generic.go:356] "Generic (PLEG): container finished" podID="9c2c1ce9-e204-4204-9793-7d46595601ba" containerID="6a8eae85c0e24b7350fc89dc3b43b487e6786840dc69056689ee8059e7d63fea" exitCode=0
Nov 02 03:12:46.997986 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:46.997846    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-181e39b9-mhtxr" event={"ID":"9c2c1ce9-e204-4204-9793-7d46595601ba","Type":"ContainerDied","Data":"6a8eae85c0e24b7350fc89dc3b43b487e6786840dc69056689ee8059e7d63fea"}
Nov 02 03:12:46.999563 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:46.999538    4496 generic.go:356] "Generic (PLEG): container finished" podID="e7b60965-efe4-4d29-baa5-2e14bc094e1b" containerID="1b0a1c915f45520f1ba81be8c1d45c6f07c4f9590450fd6506c3fc9e027c6aff" exitCode=0
Nov 02 03:12:46.999694 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:46.999590    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec07dc13-jlg6j" event={"ID":"e7b60965-efe4-4d29-baa5-2e14bc094e1b","Type":"ContainerDied","Data":"1b0a1c915f45520f1ba81be8c1d45c6f07c4f9590450fd6506c3fc9e027c6aff"}
Nov 02 03:12:47.047041 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:47.041408    4496 patch_prober.go:29] interesting pod/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-jx2rx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Nov 02 03:12:47.047041 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:47.041496    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-jx2rx" podUID="2ecf68d6-0a44-436c-938f-e75696a37e6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Nov 02 03:12:48.060895 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:48.060421    4496 patch_prober.go:29] interesting pod/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-jx2rx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Nov 02 03:12:48.060895 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:48.060507    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-jx2rx" podUID="2ecf68d6-0a44-436c-938f-e75696a37e6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Nov 02 03:12:48.233047 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:48.232928    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-181e39b9-mhtxr"
Nov 02 03:12:48.295676 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:48.295650    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec07dc13-jlg6j"
Nov 02 03:12:48.399639 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:48.399602    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c2c1ce9-e204-4204-9793-7d46595601ba-kube-api-access-d9vqg\" (UniqueName: \"kubernetes.io/projected/9c2c1ce9-e204-4204-9793-7d46595601ba-kube-api-access-d9vqg\") pod \"9c2c1ce9-e204-4204-9793-7d46595601ba\" (UID: \"9c2c1ce9-e204-4204-9793-7d46595601ba\") "
Nov 02 03:12:48.399813 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:48.399670    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7b60965-efe4-4d29-baa5-2e14bc094e1b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7b60965-efe4-4d29-baa5-2e14bc094e1b-maas-api-xks-ca\") pod \"e7b60965-efe4-4d29-baa5-2e14bc094e1b\" (UID: \"e7b60965-efe4-4d29-baa5-2e14bc094e1b\") "
Nov 02 03:12:48.399813 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:48.399772    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7b60965-efe4-4d29-baa5-2e14bc094e1b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7b60965-efe4-4d29-baa5-2e14bc094e1b-maas-api-openshift-ca\") pod \"e7b60965-efe4-4d29-baa5-2e14bc094e1b\" (UID: \"e7b60965-efe4-4d29-baa5-2e14bc094e1b\") "
Nov 02 03:12:48.399940 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:48.399837    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7b60965-efe4-4d29-baa5-2e14bc094e1b-kube-api-access-v6nxs\" (UniqueName: \"kubernetes.io/projected/e7b60965-efe4-4d29-baa5-2e14bc094e1b-kube-api-access-v6nxs\") pod \"e7b60965-efe4-4d29-baa5-2e14bc094e1b\" (UID: \"e7b60965-efe4-4d29-baa5-2e14bc094e1b\") "
Nov 02 03:12:48.400006 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:48.399960    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c2c1ce9-e204-4204-9793-7d46595601ba-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c2c1ce9-e204-4204-9793-7d46595601ba-maas-api-openshift-ca\") pod \"9c2c1ce9-e204-4204-9793-7d46595601ba\" (UID: \"9c2c1ce9-e204-4204-9793-7d46595601ba\") "
Nov 02 03:12:48.400152 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:48.400130    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c2c1ce9-e204-4204-9793-7d46595601ba-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c2c1ce9-e204-4204-9793-7d46595601ba-maas-api-xks-ca\") pod \"9c2c1ce9-e204-4204-9793-7d46595601ba\" (UID: \"9c2c1ce9-e204-4204-9793-7d46595601ba\") "
Nov 02 03:12:48.400427 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:48.400403    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c2c1ce9-e204-4204-9793-7d46595601ba-maas-api-openshift-ca" pod "9c2c1ce9-e204-4204-9793-7d46595601ba" (UID: "9c2c1ce9-e204-4204-9793-7d46595601ba"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:12:48.400575 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:48.400559    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c2c1ce9-e204-4204-9793-7d46595601ba-maas-api-openshift-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:12:48.401147 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:48.401104    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7b60965-efe4-4d29-baa5-2e14bc094e1b-maas-api-openshift-ca" pod "e7b60965-efe4-4d29-baa5-2e14bc094e1b" (UID: "e7b60965-efe4-4d29-baa5-2e14bc094e1b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:12:48.402600 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:48.402569    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c2c1ce9-e204-4204-9793-7d46595601ba-maas-api-xks-ca" pod "9c2c1ce9-e204-4204-9793-7d46595601ba" (UID: "9c2c1ce9-e204-4204-9793-7d46595601ba"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 03:12:48.403165 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:48.403138    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c2c1ce9-e204-4204-9793-7d46595601ba-kube-api-access-d9vqg" pod "9c2c1ce9-e204-4204-9793-7d46595601ba" (UID: "9c2c1ce9-e204-4204-9793-7d46595601ba"). InnerVolumeSpecName "kube-api-access-d9vqg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:12:48.403267 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:48.403245    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7b60965-efe4-4d29-baa5-2e14bc094e1b-kube-api-access-v6nxs" pod "e7b60965-efe4-4d29-baa5-2e14bc094e1b" (UID: "e7b60965-efe4-4d29-baa5-2e14bc094e1b"). InnerVolumeSpecName "kube-api-access-v6nxs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:12:48.404299 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:48.404277    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7b60965-efe4-4d29-baa5-2e14bc094e1b-maas-api-xks-ca" pod "e7b60965-efe4-4d29-baa5-2e14bc094e1b" (UID: "e7b60965-efe4-4d29-baa5-2e14bc094e1b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 03:12:48.502153 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:48.502110    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c2c1ce9-e204-4204-9793-7d46595601ba-maas-api-xks-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:12:48.502153 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:48.502146    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9vqg\" (UniqueName: \"kubernetes.io/projected/9c2c1ce9-e204-4204-9793-7d46595601ba-kube-api-access-d9vqg\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:12:48.502410 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:48.502163    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7b60965-efe4-4d29-baa5-2e14bc094e1b-maas-api-xks-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:12:48.502410 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:48.502189    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7b60965-efe4-4d29-baa5-2e14bc094e1b-maas-api-openshift-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:12:48.502410 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:48.502203    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6nxs\" (UniqueName: \"kubernetes.io/projected/e7b60965-efe4-4d29-baa5-2e14bc094e1b-kube-api-access-v6nxs\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:12:49.014098 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:49.014069    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-181e39b9-mhtxr"
Nov 02 03:12:49.014286 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:49.014071    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-181e39b9-mhtxr" event={"ID":"9c2c1ce9-e204-4204-9793-7d46595601ba","Type":"ContainerDied","Data":"a142f5daf756667a2164919591c1add821458d3725c39dcefc4964202ebac900"}
Nov 02 03:12:49.014286 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:49.014214    4496 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a142f5daf756667a2164919591c1add821458d3725c39dcefc4964202ebac900"
Nov 02 03:12:49.015638 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:49.015616    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec07dc13-jlg6j" event={"ID":"e7b60965-efe4-4d29-baa5-2e14bc094e1b","Type":"ContainerDied","Data":"f19160cb1dbdfa1e5b95af645dbc60740eaccafbc31b805e4ab675880fe949af"}
Nov 02 03:12:49.015746 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:49.015644    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec07dc13-jlg6j"
Nov 02 03:12:49.015746 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:49.015651    4496 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f19160cb1dbdfa1e5b95af645dbc60740eaccafbc31b805e4ab675880fe949af"
Nov 02 03:12:49.060094 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:49.049256    4496 patch_prober.go:29] interesting pod/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-jx2rx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Nov 02 03:12:49.060094 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:49.049321    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-jx2rx" podUID="2ecf68d6-0a44-436c-938f-e75696a37e6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Nov 02 03:12:50.042039 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:50.037378    4496 patch_prober.go:29] interesting pod/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-jx2rx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Nov 02 03:12:50.042039 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:50.037441    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-jx2rx" podUID="2ecf68d6-0a44-436c-938f-e75696a37e6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Nov 02 03:12:51.037395 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:51.037358    4496 patch_prober.go:29] interesting pod/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-jx2rx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Nov 02 03:12:51.037676 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:51.037646    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-jx2rx" podUID="2ecf68d6-0a44-436c-938f-e75696a37e6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Nov 02 03:12:52.037846 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:52.037807    4496 patch_prober.go:29] interesting pod/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-jx2rx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Nov 02 03:12:52.038338 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:52.037870    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-jx2rx" podUID="2ecf68d6-0a44-436c-938f-e75696a37e6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Nov 02 03:12:53.094255 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:53.094219    4496 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-jx2rx"
Nov 02 03:12:53.094712 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:53.094544    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-jx2rx"
Nov 02 03:12:53.095396 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:53.095378    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-jx2rx"
Nov 02 03:12:59.559008 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:59.558964    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ce700f8-fc6fc"]
Nov 02 03:12:59.559539 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:59.559525    4496 cpu_manager.go:414] "containerMap: removing container" podUID="9c2c1ce9-e204-4204-9793-7d46595601ba" containerName="revoke-keys"
Nov 02 03:12:59.559582 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:59.559541    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c2c1ce9-e204-4204-9793-7d46595601ba" containerName="revoke-keys"
Nov 02 03:12:59.559582 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:59.559561    4496 cpu_manager.go:414] "containerMap: removing container" podUID="e7b60965-efe4-4d29-baa5-2e14bc094e1b" containerName="revoke-keys"
Nov 02 03:12:59.559582 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:59.559565    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7b60965-efe4-4d29-baa5-2e14bc094e1b" containerName="revoke-keys"
Nov 02 03:12:59.559675 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:59.559668    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c2c1ce9-e204-4204-9793-7d46595601ba" containerName="revoke-keys"
Nov 02 03:12:59.559711 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:59.559679    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7b60965-efe4-4d29-baa5-2e14bc094e1b" containerName="revoke-keys"
Nov 02 03:12:59.564628 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:59.564611    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ce700f8-fc6fc"
Nov 02 03:12:59.608893 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:59.608862    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a95cfe25-614a-4e18-a401-5ff9c2b2772a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ce700f8-fc6fc\" (UID: \"a95cfe25-614a-4e18-a401-5ff9c2b2772a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ce700f8-fc6fc"
Nov 02 03:12:59.609089 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:59.608977    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7j7x\" (UniqueName: \"kubernetes.io/projected/a95cfe25-614a-4e18-a401-5ff9c2b2772a-kube-api-access-s7j7x\") pod \"maas-api-revoke-sub-6ce700f8-fc6fc\" (UID: \"a95cfe25-614a-4e18-a401-5ff9c2b2772a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ce700f8-fc6fc"
Nov 02 03:12:59.609174 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:59.609115    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a95cfe25-614a-4e18-a401-5ff9c2b2772a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ce700f8-fc6fc\" (UID: \"a95cfe25-614a-4e18-a401-5ff9c2b2772a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ce700f8-fc6fc"
Nov 02 03:12:59.619883 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:59.619856    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ce700f8-fc6fc"]
Nov 02 03:12:59.666151 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:59.666130    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 03:12:59.666151 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:59.666138    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9z5bk\""
Nov 02 03:12:59.666322 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:59.666180    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 03:12:59.668082 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:59.668059    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 03:12:59.710476 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:59.710450    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a95cfe25-614a-4e18-a401-5ff9c2b2772a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ce700f8-fc6fc\" (UID: \"a95cfe25-614a-4e18-a401-5ff9c2b2772a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ce700f8-fc6fc"
Nov 02 03:12:59.710598 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:59.710510    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a95cfe25-614a-4e18-a401-5ff9c2b2772a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ce700f8-fc6fc\" (UID: \"a95cfe25-614a-4e18-a401-5ff9c2b2772a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ce700f8-fc6fc"
Nov 02 03:12:59.710598 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:59.710565    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7j7x\" (UniqueName: \"kubernetes.io/projected/a95cfe25-614a-4e18-a401-5ff9c2b2772a-kube-api-access-s7j7x\") pod \"maas-api-revoke-sub-6ce700f8-fc6fc\" (UID: \"a95cfe25-614a-4e18-a401-5ff9c2b2772a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ce700f8-fc6fc"
Nov 02 03:12:59.711135 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:59.711109    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a95cfe25-614a-4e18-a401-5ff9c2b2772a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ce700f8-fc6fc\" (UID: \"a95cfe25-614a-4e18-a401-5ff9c2b2772a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ce700f8-fc6fc"
Nov 02 03:12:59.712825 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:59.712803    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a95cfe25-614a-4e18-a401-5ff9c2b2772a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ce700f8-fc6fc\" (UID: \"a95cfe25-614a-4e18-a401-5ff9c2b2772a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ce700f8-fc6fc"
Nov 02 03:12:59.768063 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:59.768038    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7j7x\" (UniqueName: \"kubernetes.io/projected/a95cfe25-614a-4e18-a401-5ff9c2b2772a-kube-api-access-s7j7x\") pod \"maas-api-revoke-sub-6ce700f8-fc6fc\" (UID: \"a95cfe25-614a-4e18-a401-5ff9c2b2772a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ce700f8-fc6fc"
Nov 02 03:12:59.879978 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:12:59.879895    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ce700f8-fc6fc"
Nov 02 03:13:00.259561 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:13:00.259526    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda95cfe25_614a_4e18_a401_5ff9c2b2772a.slice/crio-e3ae33aa22842c4679f8f66cbaf51544129a35d19d41f7aec0f6802d1bec00f1 WatchSource:0}: Error finding container e3ae33aa22842c4679f8f66cbaf51544129a35d19d41f7aec0f6802d1bec00f1: Status 404 returned error can't find the container with id e3ae33aa22842c4679f8f66cbaf51544129a35d19d41f7aec0f6802d1bec00f1
Nov 02 03:13:00.260844 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:00.260814    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ce700f8-fc6fc"]
Nov 02 03:13:01.096431 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:01.096249    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ce700f8-fc6fc" event={"ID":"a95cfe25-614a-4e18-a401-5ff9c2b2772a","Type":"ContainerStarted","Data":"617bacde990926e12001bd1e6dbf2856dd1f58c6d4c9b85277bbdf94411f3314"}
Nov 02 03:13:01.096431 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:01.096300    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ce700f8-fc6fc" event={"ID":"a95cfe25-614a-4e18-a401-5ff9c2b2772a","Type":"ContainerStarted","Data":"e3ae33aa22842c4679f8f66cbaf51544129a35d19d41f7aec0f6802d1bec00f1"}
Nov 02 03:13:01.213564 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:01.213512    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ce700f8-fc6fc" podStartSLOduration=2.213499347 podStartE2EDuration="2.213499347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:12:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:13:01.211309435 +0000 UTC m=+1521.789916815" watchObservedRunningTime="2026-11-02 03:13:01.213499347 +0000 UTC m=+1521.792106742"
Nov 02 03:13:02.103675 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:02.103635    4496 generic.go:356] "Generic (PLEG): container finished" podID="a95cfe25-614a-4e18-a401-5ff9c2b2772a" containerID="617bacde990926e12001bd1e6dbf2856dd1f58c6d4c9b85277bbdf94411f3314" exitCode=0
Nov 02 03:13:02.104159 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:02.103731    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ce700f8-fc6fc" event={"ID":"a95cfe25-614a-4e18-a401-5ff9c2b2772a","Type":"ContainerDied","Data":"617bacde990926e12001bd1e6dbf2856dd1f58c6d4c9b85277bbdf94411f3314"}
Nov 02 03:13:03.277654 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:03.277628    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ce700f8-fc6fc"
Nov 02 03:13:03.351319 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:03.351284    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a95cfe25-614a-4e18-a401-5ff9c2b2772a-kube-api-access-s7j7x\" (UniqueName: \"kubernetes.io/projected/a95cfe25-614a-4e18-a401-5ff9c2b2772a-kube-api-access-s7j7x\") pod \"a95cfe25-614a-4e18-a401-5ff9c2b2772a\" (UID: \"a95cfe25-614a-4e18-a401-5ff9c2b2772a\") "
Nov 02 03:13:03.351483 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:03.351328    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a95cfe25-614a-4e18-a401-5ff9c2b2772a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a95cfe25-614a-4e18-a401-5ff9c2b2772a-maas-api-openshift-ca\") pod \"a95cfe25-614a-4e18-a401-5ff9c2b2772a\" (UID: \"a95cfe25-614a-4e18-a401-5ff9c2b2772a\") "
Nov 02 03:13:03.351483 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:03.351454    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a95cfe25-614a-4e18-a401-5ff9c2b2772a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a95cfe25-614a-4e18-a401-5ff9c2b2772a-maas-api-xks-ca\") pod \"a95cfe25-614a-4e18-a401-5ff9c2b2772a\" (UID: \"a95cfe25-614a-4e18-a401-5ff9c2b2772a\") "
Nov 02 03:13:03.351695 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:03.351674    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a95cfe25-614a-4e18-a401-5ff9c2b2772a-maas-api-openshift-ca" pod "a95cfe25-614a-4e18-a401-5ff9c2b2772a" (UID: "a95cfe25-614a-4e18-a401-5ff9c2b2772a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:13:03.351806 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:03.351792    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a95cfe25-614a-4e18-a401-5ff9c2b2772a-maas-api-openshift-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:13:03.353487 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:03.353447    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a95cfe25-614a-4e18-a401-5ff9c2b2772a-maas-api-xks-ca" pod "a95cfe25-614a-4e18-a401-5ff9c2b2772a" (UID: "a95cfe25-614a-4e18-a401-5ff9c2b2772a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 03:13:03.353597 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:03.353533    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a95cfe25-614a-4e18-a401-5ff9c2b2772a-kube-api-access-s7j7x" pod "a95cfe25-614a-4e18-a401-5ff9c2b2772a" (UID: "a95cfe25-614a-4e18-a401-5ff9c2b2772a"). InnerVolumeSpecName "kube-api-access-s7j7x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:13:03.452498 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:03.452457    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7j7x\" (UniqueName: \"kubernetes.io/projected/a95cfe25-614a-4e18-a401-5ff9c2b2772a-kube-api-access-s7j7x\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:13:03.452498 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:03.452495    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a95cfe25-614a-4e18-a401-5ff9c2b2772a-maas-api-xks-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:13:03.526987 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:13:03.526948    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 03:13:04.120039 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:04.119968    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ce700f8-fc6fc" event={"ID":"a95cfe25-614a-4e18-a401-5ff9c2b2772a","Type":"ContainerDied","Data":"e3ae33aa22842c4679f8f66cbaf51544129a35d19d41f7aec0f6802d1bec00f1"}
Nov 02 03:13:04.120212 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:04.120051    4496 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e3ae33aa22842c4679f8f66cbaf51544129a35d19d41f7aec0f6802d1bec00f1"
Nov 02 03:13:04.120212 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:04.120124    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ce700f8-fc6fc"
Nov 02 03:13:04.843396 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:04.843358    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1744e3a8-bsqhh"]
Nov 02 03:13:04.845209 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:04.845181    4496 cpu_manager.go:414] "containerMap: removing container" podUID="a95cfe25-614a-4e18-a401-5ff9c2b2772a" containerName="revoke-keys"
Nov 02 03:13:04.845209 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:04.845208    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a95cfe25-614a-4e18-a401-5ff9c2b2772a" containerName="revoke-keys"
Nov 02 03:13:04.845448 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:04.845432    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="a95cfe25-614a-4e18-a401-5ff9c2b2772a" containerName="revoke-keys"
Nov 02 03:13:04.849587 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:04.849562    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1744e3a8-bsqhh"
Nov 02 03:13:04.903075 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:04.903043    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 03:13:04.954038 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:04.953855    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 03:13:04.954038 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:04.953989    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9z5bk\""
Nov 02 03:13:04.954483 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:04.954459    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 03:13:04.959301 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:04.959272    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1744e3a8-bsqhh"]
Nov 02 03:13:04.968780 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:04.968752    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d15dd94-66d5-475e-9ec8-55b043fe3597-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1744e3a8-bsqhh\" (UID: \"4d15dd94-66d5-475e-9ec8-55b043fe3597\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1744e3a8-bsqhh"
Nov 02 03:13:04.968900 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:04.968798    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d15dd94-66d5-475e-9ec8-55b043fe3597-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1744e3a8-bsqhh\" (UID: \"4d15dd94-66d5-475e-9ec8-55b043fe3597\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1744e3a8-bsqhh"
Nov 02 03:13:04.968900 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:04.968831    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmwxm\" (UniqueName: \"kubernetes.io/projected/4d15dd94-66d5-475e-9ec8-55b043fe3597-kube-api-access-lmwxm\") pod \"maas-api-revoke-sub-1744e3a8-bsqhh\" (UID: \"4d15dd94-66d5-475e-9ec8-55b043fe3597\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1744e3a8-bsqhh"
Nov 02 03:13:05.069554 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:05.069513    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d15dd94-66d5-475e-9ec8-55b043fe3597-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1744e3a8-bsqhh\" (UID: \"4d15dd94-66d5-475e-9ec8-55b043fe3597\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1744e3a8-bsqhh"
Nov 02 03:13:05.069554 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:05.069552    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d15dd94-66d5-475e-9ec8-55b043fe3597-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1744e3a8-bsqhh\" (UID: \"4d15dd94-66d5-475e-9ec8-55b043fe3597\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1744e3a8-bsqhh"
Nov 02 03:13:05.069791 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:05.069588    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmwxm\" (UniqueName: \"kubernetes.io/projected/4d15dd94-66d5-475e-9ec8-55b043fe3597-kube-api-access-lmwxm\") pod \"maas-api-revoke-sub-1744e3a8-bsqhh\" (UID: \"4d15dd94-66d5-475e-9ec8-55b043fe3597\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1744e3a8-bsqhh"
Nov 02 03:13:05.070327 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:05.070304    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d15dd94-66d5-475e-9ec8-55b043fe3597-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1744e3a8-bsqhh\" (UID: \"4d15dd94-66d5-475e-9ec8-55b043fe3597\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1744e3a8-bsqhh"
Nov 02 03:13:05.072050 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:05.072005    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d15dd94-66d5-475e-9ec8-55b043fe3597-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1744e3a8-bsqhh\" (UID: \"4d15dd94-66d5-475e-9ec8-55b043fe3597\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1744e3a8-bsqhh"
Nov 02 03:13:05.128145 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:05.128073    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmwxm\" (UniqueName: \"kubernetes.io/projected/4d15dd94-66d5-475e-9ec8-55b043fe3597-kube-api-access-lmwxm\") pod \"maas-api-revoke-sub-1744e3a8-bsqhh\" (UID: \"4d15dd94-66d5-475e-9ec8-55b043fe3597\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1744e3a8-bsqhh"
Nov 02 03:13:05.169680 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:05.169648    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1744e3a8-bsqhh"
Nov 02 03:13:05.547201 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:05.547167    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1744e3a8-bsqhh"]
Nov 02 03:13:05.548453 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:13:05.548427    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d15dd94_66d5_475e_9ec8_55b043fe3597.slice/crio-38d9f6944e17bb172601db624535b4c2e184590b17b86a74304329c14598edc1 WatchSource:0}: Error finding container 38d9f6944e17bb172601db624535b4c2e184590b17b86a74304329c14598edc1: Status 404 returned error can't find the container with id 38d9f6944e17bb172601db624535b4c2e184590b17b86a74304329c14598edc1
Nov 02 03:13:06.164096 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:06.163070    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1744e3a8-bsqhh" event={"ID":"4d15dd94-66d5-475e-9ec8-55b043fe3597","Type":"ContainerStarted","Data":"3349496a244a764722e55b12fd44d4731a331c34f40853527e4b1161529dc196"}
Nov 02 03:13:06.164096 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:06.163138    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1744e3a8-bsqhh" event={"ID":"4d15dd94-66d5-475e-9ec8-55b043fe3597","Type":"ContainerStarted","Data":"38d9f6944e17bb172601db624535b4c2e184590b17b86a74304329c14598edc1"}
Nov 02 03:13:06.282489 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:06.282423    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1744e3a8-bsqhh" podStartSLOduration=2.282401386 podStartE2EDuration="2.282401386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:13:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:13:06.280871871 +0000 UTC m=+1526.859479251" watchObservedRunningTime="2026-11-02 03:13:06.282401386 +0000 UTC m=+1526.861008766"
Nov 02 03:13:07.171439 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:07.171406    4496 generic.go:356] "Generic (PLEG): container finished" podID="4d15dd94-66d5-475e-9ec8-55b043fe3597" containerID="3349496a244a764722e55b12fd44d4731a331c34f40853527e4b1161529dc196" exitCode=0
Nov 02 03:13:07.171858 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:07.171496    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1744e3a8-bsqhh" event={"ID":"4d15dd94-66d5-475e-9ec8-55b043fe3597","Type":"ContainerDied","Data":"3349496a244a764722e55b12fd44d4731a331c34f40853527e4b1161529dc196"}
Nov 02 03:13:08.332163 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:08.332135    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1744e3a8-bsqhh"
Nov 02 03:13:08.450210 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:08.450106    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d15dd94-66d5-475e-9ec8-55b043fe3597-kube-api-access-lmwxm\" (UniqueName: \"kubernetes.io/projected/4d15dd94-66d5-475e-9ec8-55b043fe3597-kube-api-access-lmwxm\") pod \"4d15dd94-66d5-475e-9ec8-55b043fe3597\" (UID: \"4d15dd94-66d5-475e-9ec8-55b043fe3597\") "
Nov 02 03:13:08.450572 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:08.450211    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d15dd94-66d5-475e-9ec8-55b043fe3597-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d15dd94-66d5-475e-9ec8-55b043fe3597-maas-api-openshift-ca\") pod \"4d15dd94-66d5-475e-9ec8-55b043fe3597\" (UID: \"4d15dd94-66d5-475e-9ec8-55b043fe3597\") "
Nov 02 03:13:08.450851 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:08.450806    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d15dd94-66d5-475e-9ec8-55b043fe3597-maas-api-openshift-ca" pod "4d15dd94-66d5-475e-9ec8-55b043fe3597" (UID: "4d15dd94-66d5-475e-9ec8-55b043fe3597"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:13:08.451006 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:08.450982    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d15dd94-66d5-475e-9ec8-55b043fe3597-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d15dd94-66d5-475e-9ec8-55b043fe3597-maas-api-xks-ca\") pod \"4d15dd94-66d5-475e-9ec8-55b043fe3597\" (UID: \"4d15dd94-66d5-475e-9ec8-55b043fe3597\") "
Nov 02 03:13:08.451372 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:08.451347    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d15dd94-66d5-475e-9ec8-55b043fe3597-maas-api-openshift-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:13:08.453321 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:08.453292    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d15dd94-66d5-475e-9ec8-55b043fe3597-kube-api-access-lmwxm" pod "4d15dd94-66d5-475e-9ec8-55b043fe3597" (UID: "4d15dd94-66d5-475e-9ec8-55b043fe3597"). InnerVolumeSpecName "kube-api-access-lmwxm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:13:08.456885 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:08.456853    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d15dd94-66d5-475e-9ec8-55b043fe3597-maas-api-xks-ca" pod "4d15dd94-66d5-475e-9ec8-55b043fe3597" (UID: "4d15dd94-66d5-475e-9ec8-55b043fe3597"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 03:13:08.552234 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:08.552202    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d15dd94-66d5-475e-9ec8-55b043fe3597-maas-api-xks-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:13:08.552234 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:08.552228    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmwxm\" (UniqueName: \"kubernetes.io/projected/4d15dd94-66d5-475e-9ec8-55b043fe3597-kube-api-access-lmwxm\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:13:09.186107 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:09.186064    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1744e3a8-bsqhh" event={"ID":"4d15dd94-66d5-475e-9ec8-55b043fe3597","Type":"ContainerDied","Data":"38d9f6944e17bb172601db624535b4c2e184590b17b86a74304329c14598edc1"}
Nov 02 03:13:09.186257 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:09.186132    4496 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="38d9f6944e17bb172601db624535b4c2e184590b17b86a74304329c14598edc1"
Nov 02 03:13:09.186257 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:09.186075    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1744e3a8-bsqhh"
Nov 02 03:13:15.627918 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:15.627886    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b3a374f-mclzj"]
Nov 02 03:13:15.628455 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:15.628441    4496 cpu_manager.go:414] "containerMap: removing container" podUID="4d15dd94-66d5-475e-9ec8-55b043fe3597" containerName="revoke-keys"
Nov 02 03:13:15.628497 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:15.628456    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d15dd94-66d5-475e-9ec8-55b043fe3597" containerName="revoke-keys"
Nov 02 03:13:15.628547 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:15.628538    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d15dd94-66d5-475e-9ec8-55b043fe3597" containerName="revoke-keys"
Nov 02 03:13:15.632083 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:15.632063    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b3a374f-mclzj"
Nov 02 03:13:15.687516 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:15.687488    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b3a374f-mclzj"]
Nov 02 03:13:15.734035 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:15.733993    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 03:13:15.734185 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:15.734043    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 03:13:15.734185 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:15.734065    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 03:13:15.735652 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:15.735640    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9z5bk\""
Nov 02 03:13:15.826475 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:15.826445    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/901d77d5-60d5-4708-b970-62d5411c2f3f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b3a374f-mclzj\" (UID: \"901d77d5-60d5-4708-b970-62d5411c2f3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b3a374f-mclzj"
Nov 02 03:13:15.826618 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:15.826578    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/901d77d5-60d5-4708-b970-62d5411c2f3f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b3a374f-mclzj\" (UID: \"901d77d5-60d5-4708-b970-62d5411c2f3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b3a374f-mclzj"
Nov 02 03:13:15.826618 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:15.826609    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5pj8\" (UniqueName: \"kubernetes.io/projected/901d77d5-60d5-4708-b970-62d5411c2f3f-kube-api-access-c5pj8\") pod \"maas-api-revoke-sub-1b3a374f-mclzj\" (UID: \"901d77d5-60d5-4708-b970-62d5411c2f3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b3a374f-mclzj"
Nov 02 03:13:15.927557 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:15.927486    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/901d77d5-60d5-4708-b970-62d5411c2f3f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b3a374f-mclzj\" (UID: \"901d77d5-60d5-4708-b970-62d5411c2f3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b3a374f-mclzj"
Nov 02 03:13:15.927557 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:15.927518    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5pj8\" (UniqueName: \"kubernetes.io/projected/901d77d5-60d5-4708-b970-62d5411c2f3f-kube-api-access-c5pj8\") pod \"maas-api-revoke-sub-1b3a374f-mclzj\" (UID: \"901d77d5-60d5-4708-b970-62d5411c2f3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b3a374f-mclzj"
Nov 02 03:13:15.927752 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:15.927581    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/901d77d5-60d5-4708-b970-62d5411c2f3f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b3a374f-mclzj\" (UID: \"901d77d5-60d5-4708-b970-62d5411c2f3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b3a374f-mclzj"
Nov 02 03:13:15.928250 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:15.928230    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/901d77d5-60d5-4708-b970-62d5411c2f3f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b3a374f-mclzj\" (UID: \"901d77d5-60d5-4708-b970-62d5411c2f3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b3a374f-mclzj"
Nov 02 03:13:15.929703 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:15.929684    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/901d77d5-60d5-4708-b970-62d5411c2f3f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b3a374f-mclzj\" (UID: \"901d77d5-60d5-4708-b970-62d5411c2f3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b3a374f-mclzj"
Nov 02 03:13:15.984853 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:15.984829    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5pj8\" (UniqueName: \"kubernetes.io/projected/901d77d5-60d5-4708-b970-62d5411c2f3f-kube-api-access-c5pj8\") pod \"maas-api-revoke-sub-1b3a374f-mclzj\" (UID: \"901d77d5-60d5-4708-b970-62d5411c2f3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b3a374f-mclzj"
Nov 02 03:13:16.246094 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:16.246064    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b3a374f-mclzj"
Nov 02 03:13:16.622892 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:16.622861    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b3a374f-mclzj"]
Nov 02 03:13:16.623588 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:13:16.623562    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod901d77d5_60d5_4708_b970_62d5411c2f3f.slice/crio-d4460da406cf69a523a30c18a0fc33b2dd2845fec51d149b4e18a33b2d3e62b3 WatchSource:0}: Error finding container d4460da406cf69a523a30c18a0fc33b2dd2845fec51d149b4e18a33b2d3e62b3: Status 404 returned error can't find the container with id d4460da406cf69a523a30c18a0fc33b2dd2845fec51d149b4e18a33b2d3e62b3
Nov 02 03:13:17.242666 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:17.242627    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b3a374f-mclzj" event={"ID":"901d77d5-60d5-4708-b970-62d5411c2f3f","Type":"ContainerStarted","Data":"c773fe88a57ed083ce65d5420da0bd5cfa2905e082de5d0bf9105aae6ff0070f"}
Nov 02 03:13:17.243150 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:17.242688    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b3a374f-mclzj" event={"ID":"901d77d5-60d5-4708-b970-62d5411c2f3f","Type":"ContainerStarted","Data":"d4460da406cf69a523a30c18a0fc33b2dd2845fec51d149b4e18a33b2d3e62b3"}
Nov 02 03:13:17.360211 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:17.360161    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b3a374f-mclzj" podStartSLOduration=2.360147674 podStartE2EDuration="2.360147674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:13:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:13:17.356596811 +0000 UTC m=+1537.935204192" watchObservedRunningTime="2026-11-02 03:13:17.360147674 +0000 UTC m=+1537.938755081"
Nov 02 03:13:18.254280 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:18.254238    4496 generic.go:356] "Generic (PLEG): container finished" podID="901d77d5-60d5-4708-b970-62d5411c2f3f" containerID="c773fe88a57ed083ce65d5420da0bd5cfa2905e082de5d0bf9105aae6ff0070f" exitCode=0
Nov 02 03:13:18.254775 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:18.254299    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b3a374f-mclzj" event={"ID":"901d77d5-60d5-4708-b970-62d5411c2f3f","Type":"ContainerDied","Data":"c773fe88a57ed083ce65d5420da0bd5cfa2905e082de5d0bf9105aae6ff0070f"}
Nov 02 03:13:18.582958 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:18.582882    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-6c456c64b9-drg8b"]
Nov 02 03:13:18.587160 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:18.587143    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6c456c64b9-drg8b"
Nov 02 03:13:18.640430 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:18.640397    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-cb01f95f8193\""
Nov 02 03:13:18.640430 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:18.640413    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 02 03:13:18.640662 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:18.640471    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 02 03:13:18.640662 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:18.640498    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 02 03:13:18.640662 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:18.640593    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-7d506ce743a8\""
Nov 02 03:13:18.640805 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:18.640662    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 02 03:13:18.642059 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:18.642032    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6c456c64b9-drg8b"]
Nov 02 03:13:18.655275 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:18.655247    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07a0a43a-4232-467a-b7f4-ff4772b34e33-plugins-config-volume\") pod \"payload-processing-external-model-6c456c64b9-drg8b\" (UID: \"07a0a43a-4232-467a-b7f4-ff4772b34e33\") " pod="models-as-a-service/payload-processing-external-model-6c456c64b9-drg8b"
Nov 02 03:13:18.655547 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:18.655525    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/07a0a43a-4232-467a-b7f4-ff4772b34e33-routing-overlay-volume\") pod \"payload-processing-external-model-6c456c64b9-drg8b\" (UID: \"07a0a43a-4232-467a-b7f4-ff4772b34e33\") " pod="models-as-a-service/payload-processing-external-model-6c456c64b9-drg8b"
Nov 02 03:13:18.655720 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:18.655698    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldccv\" (UniqueName: \"kubernetes.io/projected/07a0a43a-4232-467a-b7f4-ff4772b34e33-kube-api-access-ldccv\") pod \"payload-processing-external-model-6c456c64b9-drg8b\" (UID: \"07a0a43a-4232-467a-b7f4-ff4772b34e33\") " pod="models-as-a-service/payload-processing-external-model-6c456c64b9-drg8b"
Nov 02 03:13:18.655831 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:18.655809    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/07a0a43a-4232-467a-b7f4-ff4772b34e33-provider-credentials\") pod \"payload-processing-external-model-6c456c64b9-drg8b\" (UID: \"07a0a43a-4232-467a-b7f4-ff4772b34e33\") " pod="models-as-a-service/payload-processing-external-model-6c456c64b9-drg8b"
Nov 02 03:13:18.756504 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:18.756471    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldccv\" (UniqueName: \"kubernetes.io/projected/07a0a43a-4232-467a-b7f4-ff4772b34e33-kube-api-access-ldccv\") pod \"payload-processing-external-model-6c456c64b9-drg8b\" (UID: \"07a0a43a-4232-467a-b7f4-ff4772b34e33\") " pod="models-as-a-service/payload-processing-external-model-6c456c64b9-drg8b"
Nov 02 03:13:18.756691 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:18.756517    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/07a0a43a-4232-467a-b7f4-ff4772b34e33-provider-credentials\") pod \"payload-processing-external-model-6c456c64b9-drg8b\" (UID: \"07a0a43a-4232-467a-b7f4-ff4772b34e33\") " pod="models-as-a-service/payload-processing-external-model-6c456c64b9-drg8b"
Nov 02 03:13:18.756691 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:18.756553    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07a0a43a-4232-467a-b7f4-ff4772b34e33-plugins-config-volume\") pod \"payload-processing-external-model-6c456c64b9-drg8b\" (UID: \"07a0a43a-4232-467a-b7f4-ff4772b34e33\") " pod="models-as-a-service/payload-processing-external-model-6c456c64b9-drg8b"
Nov 02 03:13:18.756691 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:18.756629    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/07a0a43a-4232-467a-b7f4-ff4772b34e33-routing-overlay-volume\") pod \"payload-processing-external-model-6c456c64b9-drg8b\" (UID: \"07a0a43a-4232-467a-b7f4-ff4772b34e33\") " pod="models-as-a-service/payload-processing-external-model-6c456c64b9-drg8b"
Nov 02 03:13:18.757620 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:18.757581    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/07a0a43a-4232-467a-b7f4-ff4772b34e33-routing-overlay-volume\") pod \"payload-processing-external-model-6c456c64b9-drg8b\" (UID: \"07a0a43a-4232-467a-b7f4-ff4772b34e33\") " pod="models-as-a-service/payload-processing-external-model-6c456c64b9-drg8b"
Nov 02 03:13:18.757620 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:18.757603    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07a0a43a-4232-467a-b7f4-ff4772b34e33-plugins-config-volume\") pod \"payload-processing-external-model-6c456c64b9-drg8b\" (UID: \"07a0a43a-4232-467a-b7f4-ff4772b34e33\") " pod="models-as-a-service/payload-processing-external-model-6c456c64b9-drg8b"
Nov 02 03:13:18.759261 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:18.759233    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/07a0a43a-4232-467a-b7f4-ff4772b34e33-provider-credentials\") pod \"payload-processing-external-model-6c456c64b9-drg8b\" (UID: \"07a0a43a-4232-467a-b7f4-ff4772b34e33\") " pod="models-as-a-service/payload-processing-external-model-6c456c64b9-drg8b"
Nov 02 03:13:18.813921 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:18.813893    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldccv\" (UniqueName: \"kubernetes.io/projected/07a0a43a-4232-467a-b7f4-ff4772b34e33-kube-api-access-ldccv\") pod \"payload-processing-external-model-6c456c64b9-drg8b\" (UID: \"07a0a43a-4232-467a-b7f4-ff4772b34e33\") " pod="models-as-a-service/payload-processing-external-model-6c456c64b9-drg8b"
Nov 02 03:13:18.901646 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:18.901564    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6c456c64b9-drg8b"
Nov 02 03:13:19.284497 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:19.284458    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6c456c64b9-drg8b"]
Nov 02 03:13:19.288440 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:13:19.288396    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07a0a43a_4232_467a_b7f4_ff4772b34e33.slice/crio-72adb85ccaea8e8b75014ec69f4ea280d27a37ffefd375b26d8a43519d18bb65 WatchSource:0}: Error finding container 72adb85ccaea8e8b75014ec69f4ea280d27a37ffefd375b26d8a43519d18bb65: Status 404 returned error can't find the container with id 72adb85ccaea8e8b75014ec69f4ea280d27a37ffefd375b26d8a43519d18bb65
Nov 02 03:13:19.394995 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:19.394973    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b3a374f-mclzj"
Nov 02 03:13:19.463341 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:19.463315    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/901d77d5-60d5-4708-b970-62d5411c2f3f-kube-api-access-c5pj8\" (UniqueName: \"kubernetes.io/projected/901d77d5-60d5-4708-b970-62d5411c2f3f-kube-api-access-c5pj8\") pod \"901d77d5-60d5-4708-b970-62d5411c2f3f\" (UID: \"901d77d5-60d5-4708-b970-62d5411c2f3f\") "
Nov 02 03:13:19.463503 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:19.463373    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/901d77d5-60d5-4708-b970-62d5411c2f3f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/901d77d5-60d5-4708-b970-62d5411c2f3f-maas-api-openshift-ca\") pod \"901d77d5-60d5-4708-b970-62d5411c2f3f\" (UID: \"901d77d5-60d5-4708-b970-62d5411c2f3f\") "
Nov 02 03:13:19.463503 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:19.463486    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/901d77d5-60d5-4708-b970-62d5411c2f3f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/901d77d5-60d5-4708-b970-62d5411c2f3f-maas-api-xks-ca\") pod \"901d77d5-60d5-4708-b970-62d5411c2f3f\" (UID: \"901d77d5-60d5-4708-b970-62d5411c2f3f\") "
Nov 02 03:13:19.463774 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:19.463750    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/901d77d5-60d5-4708-b970-62d5411c2f3f-maas-api-openshift-ca" pod "901d77d5-60d5-4708-b970-62d5411c2f3f" (UID: "901d77d5-60d5-4708-b970-62d5411c2f3f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:13:19.463931 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:19.463915    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/901d77d5-60d5-4708-b970-62d5411c2f3f-maas-api-openshift-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:13:19.465511 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:19.465489    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/901d77d5-60d5-4708-b970-62d5411c2f3f-maas-api-xks-ca" pod "901d77d5-60d5-4708-b970-62d5411c2f3f" (UID: "901d77d5-60d5-4708-b970-62d5411c2f3f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 03:13:19.465608 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:19.465523    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/901d77d5-60d5-4708-b970-62d5411c2f3f-kube-api-access-c5pj8" pod "901d77d5-60d5-4708-b970-62d5411c2f3f" (UID: "901d77d5-60d5-4708-b970-62d5411c2f3f"). InnerVolumeSpecName "kube-api-access-c5pj8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:13:19.564522 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:19.564435    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/901d77d5-60d5-4708-b970-62d5411c2f3f-maas-api-xks-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:13:19.564522 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:19.564466    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5pj8\" (UniqueName: \"kubernetes.io/projected/901d77d5-60d5-4708-b970-62d5411c2f3f-kube-api-access-c5pj8\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:13:20.270618 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:20.270584    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b3a374f-mclzj"
Nov 02 03:13:20.270618 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:20.270596    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b3a374f-mclzj" event={"ID":"901d77d5-60d5-4708-b970-62d5411c2f3f","Type":"ContainerDied","Data":"d4460da406cf69a523a30c18a0fc33b2dd2845fec51d149b4e18a33b2d3e62b3"}
Nov 02 03:13:20.270846 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:20.270647    4496 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d4460da406cf69a523a30c18a0fc33b2dd2845fec51d149b4e18a33b2d3e62b3"
Nov 02 03:13:20.272419 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:20.272394    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6c456c64b9-drg8b" event={"ID":"07a0a43a-4232-467a-b7f4-ff4772b34e33","Type":"ContainerStarted","Data":"f69e2511634fff187da3d8b04aae287a979d0a672f38986d9567eb3a7804bbea"}
Nov 02 03:13:20.272608 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:20.272585    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6c456c64b9-drg8b" event={"ID":"07a0a43a-4232-467a-b7f4-ff4772b34e33","Type":"ContainerStarted","Data":"72adb85ccaea8e8b75014ec69f4ea280d27a37ffefd375b26d8a43519d18bb65"}
Nov 02 03:13:20.272737 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:20.272624    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-6c456c64b9-drg8b"
Nov 02 03:13:20.409645 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:20.409593    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-6c456c64b9-drg8b" podStartSLOduration=2.409578772 podStartE2EDuration="2.409578772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:13:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:13:20.408583371 +0000 UTC m=+1540.987190752" watchObservedRunningTime="2026-11-02 03:13:20.409578772 +0000 UTC m=+1540.988186152"
Nov 02 03:13:31.283808 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:31.283778    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-6c456c64b9-drg8b"
Nov 02 03:13:33.530369 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:13:33.530325    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 03:13:36.580463 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:36.580427    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-ca6546/e2e-conflict-a-5eca459f-kserve-5744ffc845-xgm78"]
Nov 02 03:13:36.580980 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:36.580965    4496 cpu_manager.go:414] "containerMap: removing container" podUID="901d77d5-60d5-4708-b970-62d5411c2f3f" containerName="revoke-keys"
Nov 02 03:13:36.581047 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:36.580982    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="901d77d5-60d5-4708-b970-62d5411c2f3f" containerName="revoke-keys"
Nov 02 03:13:36.581132 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:36.581121    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="901d77d5-60d5-4708-b970-62d5411c2f3f" containerName="revoke-keys"
Nov 02 03:13:36.584490 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:36.584470    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-ca6546/e2e-conflict-a-5eca459f-kserve-5744ffc845-xgm78"
Nov 02 03:13:36.644410 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:36.644379    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-ca6546/e2e-conflict-a-5eca459f-kserve-5744ffc845-xgm78"]
Nov 02 03:13:36.646783 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:36.646757    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c862a274-1d99-4f90-926e-4a72041456ab-tmp-dir\") pod \"e2e-conflict-a-5eca459f-kserve-5744ffc845-xgm78\" (UID: \"c862a274-1d99-4f90-926e-4a72041456ab\") " pod="e2e-models-e2e-worker-w2-ca6546/e2e-conflict-a-5eca459f-kserve-5744ffc845-xgm78"
Nov 02 03:13:36.646895 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:36.646804    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c862a274-1d99-4f90-926e-4a72041456ab-model-cache\") pod \"e2e-conflict-a-5eca459f-kserve-5744ffc845-xgm78\" (UID: \"c862a274-1d99-4f90-926e-4a72041456ab\") " pod="e2e-models-e2e-worker-w2-ca6546/e2e-conflict-a-5eca459f-kserve-5744ffc845-xgm78"
Nov 02 03:13:36.646952 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:36.646926    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c862a274-1d99-4f90-926e-4a72041456ab-tls-certs\") pod \"e2e-conflict-a-5eca459f-kserve-5744ffc845-xgm78\" (UID: \"c862a274-1d99-4f90-926e-4a72041456ab\") " pod="e2e-models-e2e-worker-w2-ca6546/e2e-conflict-a-5eca459f-kserve-5744ffc845-xgm78"
Nov 02 03:13:36.647000 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:36.646952    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c862a274-1d99-4f90-926e-4a72041456ab-home\") pod \"e2e-conflict-a-5eca459f-kserve-5744ffc845-xgm78\" (UID: \"c862a274-1d99-4f90-926e-4a72041456ab\") " pod="e2e-models-e2e-worker-w2-ca6546/e2e-conflict-a-5eca459f-kserve-5744ffc845-xgm78"
Nov 02 03:13:36.647000 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:36.646985    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28mvw\" (UniqueName: \"kubernetes.io/projected/c862a274-1d99-4f90-926e-4a72041456ab-kube-api-access-28mvw\") pod \"e2e-conflict-a-5eca459f-kserve-5744ffc845-xgm78\" (UID: \"c862a274-1d99-4f90-926e-4a72041456ab\") " pod="e2e-models-e2e-worker-w2-ca6546/e2e-conflict-a-5eca459f-kserve-5744ffc845-xgm78"
Nov 02 03:13:36.647092 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:36.647069    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c862a274-1d99-4f90-926e-4a72041456ab-dshm\") pod \"e2e-conflict-a-5eca459f-kserve-5744ffc845-xgm78\" (UID: \"c862a274-1d99-4f90-926e-4a72041456ab\") " pod="e2e-models-e2e-worker-w2-ca6546/e2e-conflict-a-5eca459f-kserve-5744ffc845-xgm78"
Nov 02 03:13:36.689143 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:36.689118    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-ca6546\"/\"e2e-conflict-a-5eca459f-kserve-self-signed-certs\""
Nov 02 03:13:36.747697 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:36.747662    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c862a274-1d99-4f90-926e-4a72041456ab-tmp-dir\") pod \"e2e-conflict-a-5eca459f-kserve-5744ffc845-xgm78\" (UID: \"c862a274-1d99-4f90-926e-4a72041456ab\") " pod="e2e-models-e2e-worker-w2-ca6546/e2e-conflict-a-5eca459f-kserve-5744ffc845-xgm78"
Nov 02 03:13:36.747913 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:36.747723    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c862a274-1d99-4f90-926e-4a72041456ab-model-cache\") pod \"e2e-conflict-a-5eca459f-kserve-5744ffc845-xgm78\" (UID: \"c862a274-1d99-4f90-926e-4a72041456ab\") " pod="e2e-models-e2e-worker-w2-ca6546/e2e-conflict-a-5eca459f-kserve-5744ffc845-xgm78"
Nov 02 03:13:36.747913 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:36.747812    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c862a274-1d99-4f90-926e-4a72041456ab-tls-certs\") pod \"e2e-conflict-a-5eca459f-kserve-5744ffc845-xgm78\" (UID: \"c862a274-1d99-4f90-926e-4a72041456ab\") " pod="e2e-models-e2e-worker-w2-ca6546/e2e-conflict-a-5eca459f-kserve-5744ffc845-xgm78"
Nov 02 03:13:36.747913 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:36.747842    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c862a274-1d99-4f90-926e-4a72041456ab-home\") pod \"e2e-conflict-a-5eca459f-kserve-5744ffc845-xgm78\" (UID: \"c862a274-1d99-4f90-926e-4a72041456ab\") " pod="e2e-models-e2e-worker-w2-ca6546/e2e-conflict-a-5eca459f-kserve-5744ffc845-xgm78"
Nov 02 03:13:36.748192 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:36.747909    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28mvw\" (UniqueName: \"kubernetes.io/projected/c862a274-1d99-4f90-926e-4a72041456ab-kube-api-access-28mvw\") pod \"e2e-conflict-a-5eca459f-kserve-5744ffc845-xgm78\" (UID: \"c862a274-1d99-4f90-926e-4a72041456ab\") " pod="e2e-models-e2e-worker-w2-ca6546/e2e-conflict-a-5eca459f-kserve-5744ffc845-xgm78"
Nov 02 03:13:36.748192 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:36.747941    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c862a274-1d99-4f90-926e-4a72041456ab-dshm\") pod \"e2e-conflict-a-5eca459f-kserve-5744ffc845-xgm78\" (UID: \"c862a274-1d99-4f90-926e-4a72041456ab\") " pod="e2e-models-e2e-worker-w2-ca6546/e2e-conflict-a-5eca459f-kserve-5744ffc845-xgm78"
Nov 02 03:13:36.748192 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:36.748183    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c862a274-1d99-4f90-926e-4a72041456ab-tmp-dir\") pod \"e2e-conflict-a-5eca459f-kserve-5744ffc845-xgm78\" (UID: \"c862a274-1d99-4f90-926e-4a72041456ab\") " pod="e2e-models-e2e-worker-w2-ca6546/e2e-conflict-a-5eca459f-kserve-5744ffc845-xgm78"
Nov 02 03:13:36.748672 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:36.748644    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c862a274-1d99-4f90-926e-4a72041456ab-home\") pod \"e2e-conflict-a-5eca459f-kserve-5744ffc845-xgm78\" (UID: \"c862a274-1d99-4f90-926e-4a72041456ab\") " pod="e2e-models-e2e-worker-w2-ca6546/e2e-conflict-a-5eca459f-kserve-5744ffc845-xgm78"
Nov 02 03:13:36.748891 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:36.748861    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c862a274-1d99-4f90-926e-4a72041456ab-model-cache\") pod \"e2e-conflict-a-5eca459f-kserve-5744ffc845-xgm78\" (UID: \"c862a274-1d99-4f90-926e-4a72041456ab\") " pod="e2e-models-e2e-worker-w2-ca6546/e2e-conflict-a-5eca459f-kserve-5744ffc845-xgm78"
Nov 02 03:13:36.750898 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:36.750877    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c862a274-1d99-4f90-926e-4a72041456ab-dshm\") pod \"e2e-conflict-a-5eca459f-kserve-5744ffc845-xgm78\" (UID: \"c862a274-1d99-4f90-926e-4a72041456ab\") " pod="e2e-models-e2e-worker-w2-ca6546/e2e-conflict-a-5eca459f-kserve-5744ffc845-xgm78"
Nov 02 03:13:36.751294 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:36.751271    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c862a274-1d99-4f90-926e-4a72041456ab-tls-certs\") pod \"e2e-conflict-a-5eca459f-kserve-5744ffc845-xgm78\" (UID: \"c862a274-1d99-4f90-926e-4a72041456ab\") " pod="e2e-models-e2e-worker-w2-ca6546/e2e-conflict-a-5eca459f-kserve-5744ffc845-xgm78"
Nov 02 03:13:36.805738 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:36.805707    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28mvw\" (UniqueName: \"kubernetes.io/projected/c862a274-1d99-4f90-926e-4a72041456ab-kube-api-access-28mvw\") pod \"e2e-conflict-a-5eca459f-kserve-5744ffc845-xgm78\" (UID: \"c862a274-1d99-4f90-926e-4a72041456ab\") " pod="e2e-models-e2e-worker-w2-ca6546/e2e-conflict-a-5eca459f-kserve-5744ffc845-xgm78"
Nov 02 03:13:36.896838 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:36.896767    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-ca6546/e2e-conflict-a-5eca459f-kserve-5744ffc845-xgm78"
Nov 02 03:13:37.274833 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:37.274804    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-ca6546/e2e-conflict-a-5eca459f-kserve-5744ffc845-xgm78"]
Nov 02 03:13:37.275305 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:13:37.275280    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc862a274_1d99_4f90_926e_4a72041456ab.slice/crio-d1de3356e1478bf1a600eff01d8073d980aeb7f60a896dab0dea4595d9e0aa60 WatchSource:0}: Error finding container d1de3356e1478bf1a600eff01d8073d980aeb7f60a896dab0dea4595d9e0aa60: Status 404 returned error can't find the container with id d1de3356e1478bf1a600eff01d8073d980aeb7f60a896dab0dea4595d9e0aa60
Nov 02 03:13:37.407100 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:37.402316    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89ceec26-hwnls"]
Nov 02 03:13:37.408913 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:37.408881    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89ceec26-hwnls"
Nov 02 03:13:37.416292 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:37.416259    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-ca6546/e2e-conflict-a-5eca459f-kserve-5744ffc845-xgm78" event={"ID":"c862a274-1d99-4f90-926e-4a72041456ab","Type":"ContainerStarted","Data":"5c584c396c8865dbcf4ed5cc3f60d9a19378c86111d39791bd2d38849e58ffaf"}
Nov 02 03:13:37.416414 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:37.416300    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-ca6546/e2e-conflict-a-5eca459f-kserve-5744ffc845-xgm78" event={"ID":"c862a274-1d99-4f90-926e-4a72041456ab","Type":"ContainerStarted","Data":"d1de3356e1478bf1a600eff01d8073d980aeb7f60a896dab0dea4595d9e0aa60"}
Nov 02 03:13:37.459942 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:37.459907    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89ceec26-hwnls"]
Nov 02 03:13:37.462445 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:37.462418    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 03:13:37.462618 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:37.462593    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 03:13:37.496539 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:37.496515    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9z5bk\""
Nov 02 03:13:37.496539 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:37.496518    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 03:13:37.560909 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:37.560834    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97d3e35e-8ce3-4a2e-8f1e-9370367b4a1a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89ceec26-hwnls\" (UID: \"97d3e35e-8ce3-4a2e-8f1e-9370367b4a1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89ceec26-hwnls"
Nov 02 03:13:37.561073 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:37.560967    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sx9g5\" (UniqueName: \"kubernetes.io/projected/97d3e35e-8ce3-4a2e-8f1e-9370367b4a1a-kube-api-access-sx9g5\") pod \"maas-api-revoke-sub-89ceec26-hwnls\" (UID: \"97d3e35e-8ce3-4a2e-8f1e-9370367b4a1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89ceec26-hwnls"
Nov 02 03:13:37.561125 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:37.561080    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97d3e35e-8ce3-4a2e-8f1e-9370367b4a1a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89ceec26-hwnls\" (UID: \"97d3e35e-8ce3-4a2e-8f1e-9370367b4a1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89ceec26-hwnls"
Nov 02 03:13:37.643282 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:37.643236    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-ca6546/e2e-conflict-a-5eca459f-kserve-5744ffc845-xgm78" podStartSLOduration=1.643222502 podStartE2EDuration="1.643222502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:13:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:13:37.641972156 +0000 UTC m=+1558.220579554" watchObservedRunningTime="2026-11-02 03:13:37.643222502 +0000 UTC m=+1558.221829882"
Nov 02 03:13:37.662556 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:37.662522    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sx9g5\" (UniqueName: \"kubernetes.io/projected/97d3e35e-8ce3-4a2e-8f1e-9370367b4a1a-kube-api-access-sx9g5\") pod \"maas-api-revoke-sub-89ceec26-hwnls\" (UID: \"97d3e35e-8ce3-4a2e-8f1e-9370367b4a1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89ceec26-hwnls"
Nov 02 03:13:37.662707 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:37.662600    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97d3e35e-8ce3-4a2e-8f1e-9370367b4a1a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89ceec26-hwnls\" (UID: \"97d3e35e-8ce3-4a2e-8f1e-9370367b4a1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89ceec26-hwnls"
Nov 02 03:13:37.662707 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:37.662653    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97d3e35e-8ce3-4a2e-8f1e-9370367b4a1a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89ceec26-hwnls\" (UID: \"97d3e35e-8ce3-4a2e-8f1e-9370367b4a1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89ceec26-hwnls"
Nov 02 03:13:37.663286 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:37.663266    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97d3e35e-8ce3-4a2e-8f1e-9370367b4a1a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89ceec26-hwnls\" (UID: \"97d3e35e-8ce3-4a2e-8f1e-9370367b4a1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89ceec26-hwnls"
Nov 02 03:13:37.664970 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:37.664945    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97d3e35e-8ce3-4a2e-8f1e-9370367b4a1a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89ceec26-hwnls\" (UID: \"97d3e35e-8ce3-4a2e-8f1e-9370367b4a1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89ceec26-hwnls"
Nov 02 03:13:37.720927 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:37.720893    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sx9g5\" (UniqueName: \"kubernetes.io/projected/97d3e35e-8ce3-4a2e-8f1e-9370367b4a1a-kube-api-access-sx9g5\") pod \"maas-api-revoke-sub-89ceec26-hwnls\" (UID: \"97d3e35e-8ce3-4a2e-8f1e-9370367b4a1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89ceec26-hwnls"
Nov 02 03:13:37.724722 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:37.724704    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89ceec26-hwnls"
Nov 02 03:13:37.897728 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:37.897657    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-ca6546/e2e-conflict-a-5eca459f-kserve-5744ffc845-xgm78"
Nov 02 03:13:38.109729 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:38.109463    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89ceec26-hwnls"]
Nov 02 03:13:38.112223 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:13:38.112193    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97d3e35e_8ce3_4a2e_8f1e_9370367b4a1a.slice/crio-981d6a321cafd087e45c6bf858bad529a0068ff1dc6c1806b1a202abfaf707b4 WatchSource:0}: Error finding container 981d6a321cafd087e45c6bf858bad529a0068ff1dc6c1806b1a202abfaf707b4: Status 404 returned error can't find the container with id 981d6a321cafd087e45c6bf858bad529a0068ff1dc6c1806b1a202abfaf707b4
Nov 02 03:13:38.423901 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:38.423830    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89ceec26-hwnls" event={"ID":"97d3e35e-8ce3-4a2e-8f1e-9370367b4a1a","Type":"ContainerStarted","Data":"981d6a321cafd087e45c6bf858bad529a0068ff1dc6c1806b1a202abfaf707b4"}
Nov 02 03:13:39.123978 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:39.123945    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-76bc6b7c84-sn57j"]
Nov 02 03:13:39.128068 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:39.128045    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-76bc6b7c84-sn57j"
Nov 02 03:13:39.181777 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:39.181744    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-9crjj\""
Nov 02 03:13:39.183416 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:39.183390    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-76bc6b7c84-sn57j"]
Nov 02 03:13:39.280110 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:39.280005    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a10bb5f4-5036-49fe-8e38-a00227e07a4a-routing-overlay-volume\") pod \"payload-processing-external-model-76bc6b7c84-sn57j\" (UID: \"a10bb5f4-5036-49fe-8e38-a00227e07a4a\") " pod="models-as-a-service/payload-processing-external-model-76bc6b7c84-sn57j"
Nov 02 03:13:39.280110 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:39.280066    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a10bb5f4-5036-49fe-8e38-a00227e07a4a-plugins-config-volume\") pod \"payload-processing-external-model-76bc6b7c84-sn57j\" (UID: \"a10bb5f4-5036-49fe-8e38-a00227e07a4a\") " pod="models-as-a-service/payload-processing-external-model-76bc6b7c84-sn57j"
Nov 02 03:13:39.280317 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:39.280170    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kxjl\" (UniqueName: \"kubernetes.io/projected/a10bb5f4-5036-49fe-8e38-a00227e07a4a-kube-api-access-5kxjl\") pod \"payload-processing-external-model-76bc6b7c84-sn57j\" (UID: \"a10bb5f4-5036-49fe-8e38-a00227e07a4a\") " pod="models-as-a-service/payload-processing-external-model-76bc6b7c84-sn57j"
Nov 02 03:13:39.280317 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:39.280211    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a10bb5f4-5036-49fe-8e38-a00227e07a4a-provider-credentials\") pod \"payload-processing-external-model-76bc6b7c84-sn57j\" (UID: \"a10bb5f4-5036-49fe-8e38-a00227e07a4a\") " pod="models-as-a-service/payload-processing-external-model-76bc6b7c84-sn57j"
Nov 02 03:13:39.381201 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:39.381165    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a10bb5f4-5036-49fe-8e38-a00227e07a4a-routing-overlay-volume\") pod \"payload-processing-external-model-76bc6b7c84-sn57j\" (UID: \"a10bb5f4-5036-49fe-8e38-a00227e07a4a\") " pod="models-as-a-service/payload-processing-external-model-76bc6b7c84-sn57j"
Nov 02 03:13:39.381201 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:39.381208    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a10bb5f4-5036-49fe-8e38-a00227e07a4a-plugins-config-volume\") pod \"payload-processing-external-model-76bc6b7c84-sn57j\" (UID: \"a10bb5f4-5036-49fe-8e38-a00227e07a4a\") " pod="models-as-a-service/payload-processing-external-model-76bc6b7c84-sn57j"
Nov 02 03:13:39.381395 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:39.381252    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kxjl\" (UniqueName: \"kubernetes.io/projected/a10bb5f4-5036-49fe-8e38-a00227e07a4a-kube-api-access-5kxjl\") pod \"payload-processing-external-model-76bc6b7c84-sn57j\" (UID: \"a10bb5f4-5036-49fe-8e38-a00227e07a4a\") " pod="models-as-a-service/payload-processing-external-model-76bc6b7c84-sn57j"
Nov 02 03:13:39.381395 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:39.381276    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a10bb5f4-5036-49fe-8e38-a00227e07a4a-provider-credentials\") pod \"payload-processing-external-model-76bc6b7c84-sn57j\" (UID: \"a10bb5f4-5036-49fe-8e38-a00227e07a4a\") " pod="models-as-a-service/payload-processing-external-model-76bc6b7c84-sn57j"
Nov 02 03:13:39.381839 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:39.381812    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a10bb5f4-5036-49fe-8e38-a00227e07a4a-routing-overlay-volume\") pod \"payload-processing-external-model-76bc6b7c84-sn57j\" (UID: \"a10bb5f4-5036-49fe-8e38-a00227e07a4a\") " pod="models-as-a-service/payload-processing-external-model-76bc6b7c84-sn57j"
Nov 02 03:13:39.382100 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:39.382079    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a10bb5f4-5036-49fe-8e38-a00227e07a4a-plugins-config-volume\") pod \"payload-processing-external-model-76bc6b7c84-sn57j\" (UID: \"a10bb5f4-5036-49fe-8e38-a00227e07a4a\") " pod="models-as-a-service/payload-processing-external-model-76bc6b7c84-sn57j"
Nov 02 03:13:39.383743 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:39.383722    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a10bb5f4-5036-49fe-8e38-a00227e07a4a-provider-credentials\") pod \"payload-processing-external-model-76bc6b7c84-sn57j\" (UID: \"a10bb5f4-5036-49fe-8e38-a00227e07a4a\") " pod="models-as-a-service/payload-processing-external-model-76bc6b7c84-sn57j"
Nov 02 03:13:39.431812 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:39.431781    4496 generic.go:356] "Generic (PLEG): container finished" podID="97d3e35e-8ce3-4a2e-8f1e-9370367b4a1a" containerID="01e2fddb4f0e48331e4632319437e0b1273743466a9685ebcdc4236be094f502" exitCode=0
Nov 02 03:13:39.431912 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:39.431866    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89ceec26-hwnls" event={"ID":"97d3e35e-8ce3-4a2e-8f1e-9370367b4a1a","Type":"ContainerDied","Data":"01e2fddb4f0e48331e4632319437e0b1273743466a9685ebcdc4236be094f502"}
Nov 02 03:13:39.438955 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:39.438926    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kxjl\" (UniqueName: \"kubernetes.io/projected/a10bb5f4-5036-49fe-8e38-a00227e07a4a-kube-api-access-5kxjl\") pod \"payload-processing-external-model-76bc6b7c84-sn57j\" (UID: \"a10bb5f4-5036-49fe-8e38-a00227e07a4a\") " pod="models-as-a-service/payload-processing-external-model-76bc6b7c84-sn57j"
Nov 02 03:13:39.442799 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:39.442779    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-76bc6b7c84-sn57j"
Nov 02 03:13:39.819366 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:13:39.819337    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda10bb5f4_5036_49fe_8e38_a00227e07a4a.slice/crio-b7e73168b12a027ea295d9dfc8cfe6187566dc7f39eee1c9a7a07ece8f94281c WatchSource:0}: Error finding container b7e73168b12a027ea295d9dfc8cfe6187566dc7f39eee1c9a7a07ece8f94281c: Status 404 returned error can't find the container with id b7e73168b12a027ea295d9dfc8cfe6187566dc7f39eee1c9a7a07ece8f94281c
Nov 02 03:13:39.819693 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:39.819668    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-76bc6b7c84-sn57j"]
Nov 02 03:13:40.441519 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:40.441479    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-76bc6b7c84-sn57j" event={"ID":"a10bb5f4-5036-49fe-8e38-a00227e07a4a","Type":"ContainerStarted","Data":"dabcf902d1f4f26cc23598499dbb632ddc375c5344aba7d12c5685d4d42a4bfb"}
Nov 02 03:13:40.441945 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:40.441780    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-76bc6b7c84-sn57j"
Nov 02 03:13:40.441945 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:40.441821    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-76bc6b7c84-sn57j" event={"ID":"a10bb5f4-5036-49fe-8e38-a00227e07a4a","Type":"ContainerStarted","Data":"b7e73168b12a027ea295d9dfc8cfe6187566dc7f39eee1c9a7a07ece8f94281c"}
Nov 02 03:13:40.556954 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:40.556893    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-76bc6b7c84-sn57j" podStartSLOduration=1.556874213 podStartE2EDuration="1.556874213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:13:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:13:40.556549453 +0000 UTC m=+1561.135156842" watchObservedRunningTime="2026-11-02 03:13:40.556874213 +0000 UTC m=+1561.135481594"
Nov 02 03:13:40.601470 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:40.601445    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89ceec26-hwnls"
Nov 02 03:13:40.696720 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:40.696639    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97d3e35e-8ce3-4a2e-8f1e-9370367b4a1a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97d3e35e-8ce3-4a2e-8f1e-9370367b4a1a-maas-api-openshift-ca\") pod \"97d3e35e-8ce3-4a2e-8f1e-9370367b4a1a\" (UID: \"97d3e35e-8ce3-4a2e-8f1e-9370367b4a1a\") "
Nov 02 03:13:40.696720 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:40.696706    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97d3e35e-8ce3-4a2e-8f1e-9370367b4a1a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97d3e35e-8ce3-4a2e-8f1e-9370367b4a1a-maas-api-xks-ca\") pod \"97d3e35e-8ce3-4a2e-8f1e-9370367b4a1a\" (UID: \"97d3e35e-8ce3-4a2e-8f1e-9370367b4a1a\") "
Nov 02 03:13:40.696941 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:40.696831    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97d3e35e-8ce3-4a2e-8f1e-9370367b4a1a-kube-api-access-sx9g5\" (UniqueName: \"kubernetes.io/projected/97d3e35e-8ce3-4a2e-8f1e-9370367b4a1a-kube-api-access-sx9g5\") pod \"97d3e35e-8ce3-4a2e-8f1e-9370367b4a1a\" (UID: \"97d3e35e-8ce3-4a2e-8f1e-9370367b4a1a\") "
Nov 02 03:13:40.696993 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:40.696967    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97d3e35e-8ce3-4a2e-8f1e-9370367b4a1a-maas-api-openshift-ca" pod "97d3e35e-8ce3-4a2e-8f1e-9370367b4a1a" (UID: "97d3e35e-8ce3-4a2e-8f1e-9370367b4a1a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:13:40.697242 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:40.697225    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97d3e35e-8ce3-4a2e-8f1e-9370367b4a1a-maas-api-openshift-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:13:40.698767 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:40.698741    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97d3e35e-8ce3-4a2e-8f1e-9370367b4a1a-maas-api-xks-ca" pod "97d3e35e-8ce3-4a2e-8f1e-9370367b4a1a" (UID: "97d3e35e-8ce3-4a2e-8f1e-9370367b4a1a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 03:13:40.698869 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:40.698784    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97d3e35e-8ce3-4a2e-8f1e-9370367b4a1a-kube-api-access-sx9g5" pod "97d3e35e-8ce3-4a2e-8f1e-9370367b4a1a" (UID: "97d3e35e-8ce3-4a2e-8f1e-9370367b4a1a"). InnerVolumeSpecName "kube-api-access-sx9g5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:13:40.798106 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:40.798071    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sx9g5\" (UniqueName: \"kubernetes.io/projected/97d3e35e-8ce3-4a2e-8f1e-9370367b4a1a-kube-api-access-sx9g5\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:13:40.798106 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:40.798100    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97d3e35e-8ce3-4a2e-8f1e-9370367b4a1a-maas-api-xks-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:13:41.451071 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:41.451042    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89ceec26-hwnls"
Nov 02 03:13:41.451546 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:41.451070    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89ceec26-hwnls" event={"ID":"97d3e35e-8ce3-4a2e-8f1e-9370367b4a1a","Type":"ContainerDied","Data":"981d6a321cafd087e45c6bf858bad529a0068ff1dc6c1806b1a202abfaf707b4"}
Nov 02 03:13:41.451546 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:41.451125    4496 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="981d6a321cafd087e45c6bf858bad529a0068ff1dc6c1806b1a202abfaf707b4"
Nov 02 03:13:46.897755 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:46.897716    4496 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-ca6546/e2e-conflict-a-5eca459f-kserve-5744ffc845-xgm78"
Nov 02 03:13:46.911238 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:46.911209    4496 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-ca6546/e2e-conflict-a-5eca459f-kserve-5744ffc845-xgm78"
Nov 02 03:13:47.508326 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:47.508287    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-ca6546/e2e-conflict-a-5eca459f-kserve-5744ffc845-xgm78"
Nov 02 03:13:50.860966 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:50.860929    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lpxkv"]
Nov 02 03:13:50.861527 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:50.861511    4496 cpu_manager.go:414] "containerMap: removing container" podUID="97d3e35e-8ce3-4a2e-8f1e-9370367b4a1a" containerName="revoke-keys"
Nov 02 03:13:50.861527 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:50.861527    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97d3e35e-8ce3-4a2e-8f1e-9370367b4a1a" containerName="revoke-keys"
Nov 02 03:13:50.861635 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:50.861625    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="97d3e35e-8ce3-4a2e-8f1e-9370367b4a1a" containerName="revoke-keys"
Nov 02 03:13:50.866486 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:50.866464    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lpxkv"
Nov 02 03:13:50.906713 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:50.906687    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f395e8e3-ec87-4a63-94b5-8b29e017c2e9-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-lpxkv\" (UID: \"f395e8e3-ec87-4a63-94b5-8b29e017c2e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lpxkv"
Nov 02 03:13:50.906855 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:50.906752    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f395e8e3-ec87-4a63-94b5-8b29e017c2e9-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-lpxkv\" (UID: \"f395e8e3-ec87-4a63-94b5-8b29e017c2e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lpxkv"
Nov 02 03:13:50.906855 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:50.906825    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f395e8e3-ec87-4a63-94b5-8b29e017c2e9-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-lpxkv\" (UID: \"f395e8e3-ec87-4a63-94b5-8b29e017c2e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lpxkv"
Nov 02 03:13:50.906943 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:50.906859    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f395e8e3-ec87-4a63-94b5-8b29e017c2e9-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-lpxkv\" (UID: \"f395e8e3-ec87-4a63-94b5-8b29e017c2e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lpxkv"
Nov 02 03:13:50.906984 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:50.906942    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f395e8e3-ec87-4a63-94b5-8b29e017c2e9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-lpxkv\" (UID: \"f395e8e3-ec87-4a63-94b5-8b29e017c2e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lpxkv"
Nov 02 03:13:50.906984 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:50.906970    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f395e8e3-ec87-4a63-94b5-8b29e017c2e9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-lpxkv\" (UID: \"f395e8e3-ec87-4a63-94b5-8b29e017c2e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lpxkv"
Nov 02 03:13:50.907123 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:50.907091    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f395e8e3-ec87-4a63-94b5-8b29e017c2e9-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-lpxkv\" (UID: \"f395e8e3-ec87-4a63-94b5-8b29e017c2e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lpxkv"
Nov 02 03:13:50.907183 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:50.907130    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jldt\" (UniqueName: \"kubernetes.io/projected/f395e8e3-ec87-4a63-94b5-8b29e017c2e9-kube-api-access-2jldt\") pod \"maas-default-gateway-openshift-default-58847f9fc5-lpxkv\" (UID: \"f395e8e3-ec87-4a63-94b5-8b29e017c2e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lpxkv"
Nov 02 03:13:50.907246 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:50.907232    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f395e8e3-ec87-4a63-94b5-8b29e017c2e9-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-lpxkv\" (UID: \"f395e8e3-ec87-4a63-94b5-8b29e017c2e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lpxkv"
Nov 02 03:13:51.007886 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:51.007852    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f395e8e3-ec87-4a63-94b5-8b29e017c2e9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-lpxkv\" (UID: \"f395e8e3-ec87-4a63-94b5-8b29e017c2e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lpxkv"
Nov 02 03:13:51.007886 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:51.007887    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f395e8e3-ec87-4a63-94b5-8b29e017c2e9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-lpxkv\" (UID: \"f395e8e3-ec87-4a63-94b5-8b29e017c2e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lpxkv"
Nov 02 03:13:51.008131 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:51.007925    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f395e8e3-ec87-4a63-94b5-8b29e017c2e9-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-lpxkv\" (UID: \"f395e8e3-ec87-4a63-94b5-8b29e017c2e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lpxkv"
Nov 02 03:13:51.008131 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:51.007943    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jldt\" (UniqueName: \"kubernetes.io/projected/f395e8e3-ec87-4a63-94b5-8b29e017c2e9-kube-api-access-2jldt\") pod \"maas-default-gateway-openshift-default-58847f9fc5-lpxkv\" (UID: \"f395e8e3-ec87-4a63-94b5-8b29e017c2e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lpxkv"
Nov 02 03:13:51.008131 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:51.007977    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f395e8e3-ec87-4a63-94b5-8b29e017c2e9-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-lpxkv\" (UID: \"f395e8e3-ec87-4a63-94b5-8b29e017c2e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lpxkv"
Nov 02 03:13:51.008131 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:51.008043    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f395e8e3-ec87-4a63-94b5-8b29e017c2e9-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-lpxkv\" (UID: \"f395e8e3-ec87-4a63-94b5-8b29e017c2e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lpxkv"
Nov 02 03:13:51.008131 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:51.008120    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f395e8e3-ec87-4a63-94b5-8b29e017c2e9-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-lpxkv\" (UID: \"f395e8e3-ec87-4a63-94b5-8b29e017c2e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lpxkv"
Nov 02 03:13:51.008405 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:51.008140    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f395e8e3-ec87-4a63-94b5-8b29e017c2e9-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-lpxkv\" (UID: \"f395e8e3-ec87-4a63-94b5-8b29e017c2e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lpxkv"
Nov 02 03:13:51.008405 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:51.008159    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f395e8e3-ec87-4a63-94b5-8b29e017c2e9-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-lpxkv\" (UID: \"f395e8e3-ec87-4a63-94b5-8b29e017c2e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lpxkv"
Nov 02 03:13:51.008515 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:51.008460    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f395e8e3-ec87-4a63-94b5-8b29e017c2e9-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-lpxkv\" (UID: \"f395e8e3-ec87-4a63-94b5-8b29e017c2e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lpxkv"
Nov 02 03:13:51.008569 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:51.008510    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f395e8e3-ec87-4a63-94b5-8b29e017c2e9-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-lpxkv\" (UID: \"f395e8e3-ec87-4a63-94b5-8b29e017c2e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lpxkv"
Nov 02 03:13:51.008569 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:51.008549    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f395e8e3-ec87-4a63-94b5-8b29e017c2e9-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-lpxkv\" (UID: \"f395e8e3-ec87-4a63-94b5-8b29e017c2e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lpxkv"
Nov 02 03:13:51.008821 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:51.008792    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f395e8e3-ec87-4a63-94b5-8b29e017c2e9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-lpxkv\" (UID: \"f395e8e3-ec87-4a63-94b5-8b29e017c2e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lpxkv"
Nov 02 03:13:51.008900 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:51.008804    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f395e8e3-ec87-4a63-94b5-8b29e017c2e9-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-lpxkv\" (UID: \"f395e8e3-ec87-4a63-94b5-8b29e017c2e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lpxkv"
Nov 02 03:13:51.010553 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:51.010531    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f395e8e3-ec87-4a63-94b5-8b29e017c2e9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-lpxkv\" (UID: \"f395e8e3-ec87-4a63-94b5-8b29e017c2e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lpxkv"
Nov 02 03:13:51.010691 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:51.010673    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f395e8e3-ec87-4a63-94b5-8b29e017c2e9-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-lpxkv\" (UID: \"f395e8e3-ec87-4a63-94b5-8b29e017c2e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lpxkv"
Nov 02 03:13:51.064792 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:51.064759    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lpxkv"]
Nov 02 03:13:51.069445 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:51.069412    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jldt\" (UniqueName: \"kubernetes.io/projected/f395e8e3-ec87-4a63-94b5-8b29e017c2e9-kube-api-access-2jldt\") pod \"maas-default-gateway-openshift-default-58847f9fc5-lpxkv\" (UID: \"f395e8e3-ec87-4a63-94b5-8b29e017c2e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lpxkv"
Nov 02 03:13:51.069607 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:51.069531    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f395e8e3-ec87-4a63-94b5-8b29e017c2e9-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-lpxkv\" (UID: \"f395e8e3-ec87-4a63-94b5-8b29e017c2e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lpxkv"
Nov 02 03:13:51.179411 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:51.179331    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lpxkv"
Nov 02 03:13:51.454915 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:51.454825    4496 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-76bc6b7c84-sn57j" podUID="a10bb5f4-5036-49fe-8e38-a00227e07a4a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 03:13:51.631891 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:51.624001    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lpxkv"]
Nov 02 03:13:51.633693 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:51.633654    4496 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 03:13:51.633816 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:51.633717    4496 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 03:13:51.633816 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:51.633774    4496 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 03:13:52.532111 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:52.532032    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lpxkv" event={"ID":"f395e8e3-ec87-4a63-94b5-8b29e017c2e9","Type":"ContainerStarted","Data":"9e9567b0a912755a5e5ddc41ca9c638ebe3b98bfbcdccaa34c95aaea52404ad2"}
Nov 02 03:13:52.532111 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:52.532085    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lpxkv" event={"ID":"f395e8e3-ec87-4a63-94b5-8b29e017c2e9","Type":"ContainerStarted","Data":"7340e0387975abf27063e436e13360754c3ed87f1264900c2d3e677bf239b12d"}
Nov 02 03:13:52.760041 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:52.759907    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lpxkv" podStartSLOduration=2.759886615 podStartE2EDuration="2.759886615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:13:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:13:52.704966235 +0000 UTC m=+1573.283573618" watchObservedRunningTime="2026-11-02 03:13:52.759886615 +0000 UTC m=+1573.338493999"
Nov 02 03:13:53.181739 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:53.181182    4496 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lpxkv"
Nov 02 03:13:53.182115 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:53.182078    4496 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-lpxkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Nov 02 03:13:53.182259 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:53.182143    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lpxkv" podUID="f395e8e3-ec87-4a63-94b5-8b29e017c2e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Nov 02 03:13:54.179812 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:54.179763    4496 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-lpxkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Nov 02 03:13:54.180336 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:54.179831    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lpxkv" podUID="f395e8e3-ec87-4a63-94b5-8b29e017c2e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Nov 02 03:13:55.179837 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:55.179805    4496 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-lpxkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Nov 02 03:13:55.180231 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:55.179861    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lpxkv" podUID="f395e8e3-ec87-4a63-94b5-8b29e017c2e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Nov 02 03:13:56.186038 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:56.185421    4496 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-lpxkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Nov 02 03:13:56.186038 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:56.185508    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lpxkv" podUID="f395e8e3-ec87-4a63-94b5-8b29e017c2e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Nov 02 03:13:57.179828 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:57.179793    4496 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-lpxkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Nov 02 03:13:57.180000 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:57.179841    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lpxkv" podUID="f395e8e3-ec87-4a63-94b5-8b29e017c2e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Nov 02 03:13:58.180618 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:58.180579    4496 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-lpxkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Nov 02 03:13:58.181182 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:58.180661    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lpxkv" podUID="f395e8e3-ec87-4a63-94b5-8b29e017c2e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Nov 02 03:13:59.179787 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:59.179747    4496 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-lpxkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Nov 02 03:13:59.180702 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:13:59.180646    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lpxkv" podUID="f395e8e3-ec87-4a63-94b5-8b29e017c2e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Nov 02 03:14:00.327192 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:00.326326    4496 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lpxkv"
Nov 02 03:14:00.327192 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:00.326793    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lpxkv"
Nov 02 03:14:00.327970 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:00.327947    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lpxkv"
Nov 02 03:14:01.452543 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:01.452500    4496 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-76bc6b7c84-sn57j" podUID="a10bb5f4-5036-49fe-8e38-a00227e07a4a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 03:14:03.527102 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:14:03.527063    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 03:14:05.966237 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:05.966202    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-277pb"]
Nov 02 03:14:05.970260 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:05.970242    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-277pb"
Nov 02 03:14:06.026594 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:06.026562    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-277pb"]
Nov 02 03:14:06.108505 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:06.108476    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3051f935-d87a-422e-9073-909b2c2768ac-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-277pb\" (UID: \"3051f935-d87a-422e-9073-909b2c2768ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-277pb"
Nov 02 03:14:06.108505 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:06.108509    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p62vw\" (UniqueName: \"kubernetes.io/projected/3051f935-d87a-422e-9073-909b2c2768ac-kube-api-access-p62vw\") pod \"maas-default-gateway-openshift-default-58847f9fc5-277pb\" (UID: \"3051f935-d87a-422e-9073-909b2c2768ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-277pb"
Nov 02 03:14:06.108703 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:06.108537    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3051f935-d87a-422e-9073-909b2c2768ac-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-277pb\" (UID: \"3051f935-d87a-422e-9073-909b2c2768ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-277pb"
Nov 02 03:14:06.108703 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:06.108600    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3051f935-d87a-422e-9073-909b2c2768ac-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-277pb\" (UID: \"3051f935-d87a-422e-9073-909b2c2768ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-277pb"
Nov 02 03:14:06.108776 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:06.108762    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3051f935-d87a-422e-9073-909b2c2768ac-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-277pb\" (UID: \"3051f935-d87a-422e-9073-909b2c2768ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-277pb"
Nov 02 03:14:06.108810 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:06.108796    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3051f935-d87a-422e-9073-909b2c2768ac-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-277pb\" (UID: \"3051f935-d87a-422e-9073-909b2c2768ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-277pb"
Nov 02 03:14:06.108895 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:06.108875    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3051f935-d87a-422e-9073-909b2c2768ac-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-277pb\" (UID: \"3051f935-d87a-422e-9073-909b2c2768ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-277pb"
Nov 02 03:14:06.108931 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:06.108917    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3051f935-d87a-422e-9073-909b2c2768ac-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-277pb\" (UID: \"3051f935-d87a-422e-9073-909b2c2768ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-277pb"
Nov 02 03:14:06.108966 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:06.108935    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3051f935-d87a-422e-9073-909b2c2768ac-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-277pb\" (UID: \"3051f935-d87a-422e-9073-909b2c2768ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-277pb"
Nov 02 03:14:06.210304 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:06.210273    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3051f935-d87a-422e-9073-909b2c2768ac-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-277pb\" (UID: \"3051f935-d87a-422e-9073-909b2c2768ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-277pb"
Nov 02 03:14:06.210304 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:06.210302    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p62vw\" (UniqueName: \"kubernetes.io/projected/3051f935-d87a-422e-9073-909b2c2768ac-kube-api-access-p62vw\") pod \"maas-default-gateway-openshift-default-58847f9fc5-277pb\" (UID: \"3051f935-d87a-422e-9073-909b2c2768ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-277pb"
Nov 02 03:14:06.210504 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:06.210327    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3051f935-d87a-422e-9073-909b2c2768ac-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-277pb\" (UID: \"3051f935-d87a-422e-9073-909b2c2768ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-277pb"
Nov 02 03:14:06.210504 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:06.210349    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3051f935-d87a-422e-9073-909b2c2768ac-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-277pb\" (UID: \"3051f935-d87a-422e-9073-909b2c2768ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-277pb"
Nov 02 03:14:06.210504 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:06.210415    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3051f935-d87a-422e-9073-909b2c2768ac-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-277pb\" (UID: \"3051f935-d87a-422e-9073-909b2c2768ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-277pb"
Nov 02 03:14:06.210504 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:06.210448    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3051f935-d87a-422e-9073-909b2c2768ac-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-277pb\" (UID: \"3051f935-d87a-422e-9073-909b2c2768ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-277pb"
Nov 02 03:14:06.210504 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:06.210494    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3051f935-d87a-422e-9073-909b2c2768ac-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-277pb\" (UID: \"3051f935-d87a-422e-9073-909b2c2768ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-277pb"
Nov 02 03:14:06.210759 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:06.210529    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3051f935-d87a-422e-9073-909b2c2768ac-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-277pb\" (UID: \"3051f935-d87a-422e-9073-909b2c2768ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-277pb"
Nov 02 03:14:06.210759 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:06.210554    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3051f935-d87a-422e-9073-909b2c2768ac-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-277pb\" (UID: \"3051f935-d87a-422e-9073-909b2c2768ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-277pb"
Nov 02 03:14:06.210759 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:06.210710    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3051f935-d87a-422e-9073-909b2c2768ac-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-277pb\" (UID: \"3051f935-d87a-422e-9073-909b2c2768ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-277pb"
Nov 02 03:14:06.210925 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:06.210874    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3051f935-d87a-422e-9073-909b2c2768ac-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-277pb\" (UID: \"3051f935-d87a-422e-9073-909b2c2768ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-277pb"
Nov 02 03:14:06.210925 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:06.210913    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3051f935-d87a-422e-9073-909b2c2768ac-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-277pb\" (UID: \"3051f935-d87a-422e-9073-909b2c2768ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-277pb"
Nov 02 03:14:06.211006 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:06.210927    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3051f935-d87a-422e-9073-909b2c2768ac-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-277pb\" (UID: \"3051f935-d87a-422e-9073-909b2c2768ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-277pb"
Nov 02 03:14:06.211330 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:06.211313    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3051f935-d87a-422e-9073-909b2c2768ac-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-277pb\" (UID: \"3051f935-d87a-422e-9073-909b2c2768ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-277pb"
Nov 02 03:14:06.212726 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:06.212699    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3051f935-d87a-422e-9073-909b2c2768ac-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-277pb\" (UID: \"3051f935-d87a-422e-9073-909b2c2768ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-277pb"
Nov 02 03:14:06.212839 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:06.212772    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3051f935-d87a-422e-9073-909b2c2768ac-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-277pb\" (UID: \"3051f935-d87a-422e-9073-909b2c2768ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-277pb"
Nov 02 03:14:06.267771 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:06.267742    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3051f935-d87a-422e-9073-909b2c2768ac-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-277pb\" (UID: \"3051f935-d87a-422e-9073-909b2c2768ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-277pb"
Nov 02 03:14:06.267997 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:06.267978    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p62vw\" (UniqueName: \"kubernetes.io/projected/3051f935-d87a-422e-9073-909b2c2768ac-kube-api-access-p62vw\") pod \"maas-default-gateway-openshift-default-58847f9fc5-277pb\" (UID: \"3051f935-d87a-422e-9073-909b2c2768ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-277pb"
Nov 02 03:14:06.283818 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:06.283794    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-277pb"
Nov 02 03:14:06.712629 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:06.712543    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-277pb"]
Nov 02 03:14:06.750178 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:14:06.750146    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3051f935_d87a_422e_9073_909b2c2768ac.slice/crio-a219aca5ef632fbebf00cc4f9491fd47b079cb3a912ebd68670f37e3375b821b WatchSource:0}: Error finding container a219aca5ef632fbebf00cc4f9491fd47b079cb3a912ebd68670f37e3375b821b: Status 404 returned error can't find the container with id a219aca5ef632fbebf00cc4f9491fd47b079cb3a912ebd68670f37e3375b821b
Nov 02 03:14:06.752568 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:06.752538    4496 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 03:14:06.752655 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:06.752600    4496 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 03:14:06.752655 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:06.752632    4496 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 03:14:07.667922 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:07.667889    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-277pb" event={"ID":"3051f935-d87a-422e-9073-909b2c2768ac","Type":"ContainerStarted","Data":"f508d79f63179aee6899d89526a45e19a8f3725b8c9ed7be843528ac401a5a4c"}
Nov 02 03:14:07.668382 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:07.668108    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-277pb" event={"ID":"3051f935-d87a-422e-9073-909b2c2768ac","Type":"ContainerStarted","Data":"a219aca5ef632fbebf00cc4f9491fd47b079cb3a912ebd68670f37e3375b821b"}
Nov 02 03:14:08.284628 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:08.284592    4496 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-277pb"
Nov 02 03:14:08.286607 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:08.286573    4496 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-277pb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Nov 02 03:14:08.286751 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:08.286624    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-277pb" podUID="3051f935-d87a-422e-9073-909b2c2768ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Nov 02 03:14:09.285196 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:09.285156    4496 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-277pb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Nov 02 03:14:09.285726 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:09.285218    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-277pb" podUID="3051f935-d87a-422e-9073-909b2c2768ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Nov 02 03:14:10.284377 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:10.284337    4496 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-277pb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Nov 02 03:14:10.284566 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:10.284411    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-277pb" podUID="3051f935-d87a-422e-9073-909b2c2768ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Nov 02 03:14:11.285056 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:11.285003    4496 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-277pb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Nov 02 03:14:11.285536 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:11.285095    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-277pb" podUID="3051f935-d87a-422e-9073-909b2c2768ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Nov 02 03:14:11.466135 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:11.466094    4496 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-76bc6b7c84-sn57j" podUID="a10bb5f4-5036-49fe-8e38-a00227e07a4a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 03:14:12.284373 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:12.284327    4496 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-277pb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Nov 02 03:14:12.284545 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:12.284400    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-277pb" podUID="3051f935-d87a-422e-9073-909b2c2768ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Nov 02 03:14:13.284989 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:13.284943    4496 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-277pb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Nov 02 03:14:13.285423 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:13.285050    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-277pb" podUID="3051f935-d87a-422e-9073-909b2c2768ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Nov 02 03:14:14.284758 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:14.284720    4496 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-277pb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Nov 02 03:14:14.284949 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:14.284798    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-277pb" podUID="3051f935-d87a-422e-9073-909b2c2768ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Nov 02 03:14:15.288675 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:15.288642    4496 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-277pb"
Nov 02 03:14:15.289116 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:15.289056    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-277pb"
Nov 02 03:14:15.289705 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:15.289675    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-277pb"
Nov 02 03:14:15.459508 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:15.459456    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-277pb" podStartSLOduration=10.459443128 podStartE2EDuration="10.459443128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:14:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:14:07.844094567 +0000 UTC m=+1588.422701949" watchObservedRunningTime="2026-11-02 03:14:15.459443128 +0000 UTC m=+1596.038050507"
Nov 02 03:14:21.456874 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:21.456835    4496 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-76bc6b7c84-sn57j" podUID="a10bb5f4-5036-49fe-8e38-a00227e07a4a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 03:14:24.076039 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:24.069037    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-5ac93f/distinct-w1-5ac93f-kserve-64697cb8c9-vj9sk"]
Nov 02 03:14:24.080807 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:24.080776    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5ac93f/distinct-w1-5ac93f-kserve-64697cb8c9-vj9sk"
Nov 02 03:14:24.120742 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:24.120699    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75a1c221-38e8-47cf-a8b5-305396cee703-model-cache\") pod \"distinct-w1-5ac93f-kserve-64697cb8c9-vj9sk\" (UID: \"75a1c221-38e8-47cf-a8b5-305396cee703\") " pod="e2e-models-e2e-worker-w1-5ac93f/distinct-w1-5ac93f-kserve-64697cb8c9-vj9sk"
Nov 02 03:14:24.120897 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:24.120748    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75a1c221-38e8-47cf-a8b5-305396cee703-tmp-dir\") pod \"distinct-w1-5ac93f-kserve-64697cb8c9-vj9sk\" (UID: \"75a1c221-38e8-47cf-a8b5-305396cee703\") " pod="e2e-models-e2e-worker-w1-5ac93f/distinct-w1-5ac93f-kserve-64697cb8c9-vj9sk"
Nov 02 03:14:24.120897 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:24.120814    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjggc\" (UniqueName: \"kubernetes.io/projected/75a1c221-38e8-47cf-a8b5-305396cee703-kube-api-access-cjggc\") pod \"distinct-w1-5ac93f-kserve-64697cb8c9-vj9sk\" (UID: \"75a1c221-38e8-47cf-a8b5-305396cee703\") " pod="e2e-models-e2e-worker-w1-5ac93f/distinct-w1-5ac93f-kserve-64697cb8c9-vj9sk"
Nov 02 03:14:24.120897 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:24.120856    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75a1c221-38e8-47cf-a8b5-305396cee703-home\") pod \"distinct-w1-5ac93f-kserve-64697cb8c9-vj9sk\" (UID: \"75a1c221-38e8-47cf-a8b5-305396cee703\") " pod="e2e-models-e2e-worker-w1-5ac93f/distinct-w1-5ac93f-kserve-64697cb8c9-vj9sk"
Nov 02 03:14:24.121004 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:24.120939    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75a1c221-38e8-47cf-a8b5-305396cee703-dshm\") pod \"distinct-w1-5ac93f-kserve-64697cb8c9-vj9sk\" (UID: \"75a1c221-38e8-47cf-a8b5-305396cee703\") " pod="e2e-models-e2e-worker-w1-5ac93f/distinct-w1-5ac93f-kserve-64697cb8c9-vj9sk"
Nov 02 03:14:24.121064 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:24.121052    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75a1c221-38e8-47cf-a8b5-305396cee703-tls-certs\") pod \"distinct-w1-5ac93f-kserve-64697cb8c9-vj9sk\" (UID: \"75a1c221-38e8-47cf-a8b5-305396cee703\") " pod="e2e-models-e2e-worker-w1-5ac93f/distinct-w1-5ac93f-kserve-64697cb8c9-vj9sk"
Nov 02 03:14:24.128802 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:24.128773    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-5ac93f/distinct-w1-5ac93f-kserve-64697cb8c9-vj9sk"]
Nov 02 03:14:24.173816 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:24.173781    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-5ac93f\"/\"openshift-service-ca.crt\""
Nov 02 03:14:24.173816 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:24.173818    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-5ac93f\"/\"kube-root-ca.crt\""
Nov 02 03:14:24.174083 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:24.173899    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-5ac93f\"/\"distinct-w1-5ac93f-kserve-self-signed-certs\""
Nov 02 03:14:24.184967 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:24.184944    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-5ac93f\"/\"default-dockercfg-4jt6g\""
Nov 02 03:14:24.221871 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:24.221846    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75a1c221-38e8-47cf-a8b5-305396cee703-model-cache\") pod \"distinct-w1-5ac93f-kserve-64697cb8c9-vj9sk\" (UID: \"75a1c221-38e8-47cf-a8b5-305396cee703\") " pod="e2e-models-e2e-worker-w1-5ac93f/distinct-w1-5ac93f-kserve-64697cb8c9-vj9sk"
Nov 02 03:14:24.222054 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:24.221875    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75a1c221-38e8-47cf-a8b5-305396cee703-tmp-dir\") pod \"distinct-w1-5ac93f-kserve-64697cb8c9-vj9sk\" (UID: \"75a1c221-38e8-47cf-a8b5-305396cee703\") " pod="e2e-models-e2e-worker-w1-5ac93f/distinct-w1-5ac93f-kserve-64697cb8c9-vj9sk"
Nov 02 03:14:24.222054 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:24.221914    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjggc\" (UniqueName: \"kubernetes.io/projected/75a1c221-38e8-47cf-a8b5-305396cee703-kube-api-access-cjggc\") pod \"distinct-w1-5ac93f-kserve-64697cb8c9-vj9sk\" (UID: \"75a1c221-38e8-47cf-a8b5-305396cee703\") " pod="e2e-models-e2e-worker-w1-5ac93f/distinct-w1-5ac93f-kserve-64697cb8c9-vj9sk"
Nov 02 03:14:24.222054 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:24.221943    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75a1c221-38e8-47cf-a8b5-305396cee703-home\") pod \"distinct-w1-5ac93f-kserve-64697cb8c9-vj9sk\" (UID: \"75a1c221-38e8-47cf-a8b5-305396cee703\") " pod="e2e-models-e2e-worker-w1-5ac93f/distinct-w1-5ac93f-kserve-64697cb8c9-vj9sk"
Nov 02 03:14:24.222266 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:24.222102    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75a1c221-38e8-47cf-a8b5-305396cee703-dshm\") pod \"distinct-w1-5ac93f-kserve-64697cb8c9-vj9sk\" (UID: \"75a1c221-38e8-47cf-a8b5-305396cee703\") " pod="e2e-models-e2e-worker-w1-5ac93f/distinct-w1-5ac93f-kserve-64697cb8c9-vj9sk"
Nov 02 03:14:24.222266 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:24.222211    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75a1c221-38e8-47cf-a8b5-305396cee703-tls-certs\") pod \"distinct-w1-5ac93f-kserve-64697cb8c9-vj9sk\" (UID: \"75a1c221-38e8-47cf-a8b5-305396cee703\") " pod="e2e-models-e2e-worker-w1-5ac93f/distinct-w1-5ac93f-kserve-64697cb8c9-vj9sk"
Nov 02 03:14:24.222402 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:24.222377    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75a1c221-38e8-47cf-a8b5-305396cee703-tmp-dir\") pod \"distinct-w1-5ac93f-kserve-64697cb8c9-vj9sk\" (UID: \"75a1c221-38e8-47cf-a8b5-305396cee703\") " pod="e2e-models-e2e-worker-w1-5ac93f/distinct-w1-5ac93f-kserve-64697cb8c9-vj9sk"
Nov 02 03:14:24.222457 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:24.222416    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75a1c221-38e8-47cf-a8b5-305396cee703-model-cache\") pod \"distinct-w1-5ac93f-kserve-64697cb8c9-vj9sk\" (UID: \"75a1c221-38e8-47cf-a8b5-305396cee703\") " pod="e2e-models-e2e-worker-w1-5ac93f/distinct-w1-5ac93f-kserve-64697cb8c9-vj9sk"
Nov 02 03:14:24.222516 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:24.222490    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75a1c221-38e8-47cf-a8b5-305396cee703-home\") pod \"distinct-w1-5ac93f-kserve-64697cb8c9-vj9sk\" (UID: \"75a1c221-38e8-47cf-a8b5-305396cee703\") " pod="e2e-models-e2e-worker-w1-5ac93f/distinct-w1-5ac93f-kserve-64697cb8c9-vj9sk"
Nov 02 03:14:24.224711 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:24.224687    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75a1c221-38e8-47cf-a8b5-305396cee703-dshm\") pod \"distinct-w1-5ac93f-kserve-64697cb8c9-vj9sk\" (UID: \"75a1c221-38e8-47cf-a8b5-305396cee703\") " pod="e2e-models-e2e-worker-w1-5ac93f/distinct-w1-5ac93f-kserve-64697cb8c9-vj9sk"
Nov 02 03:14:24.229880 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:24.229855    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75a1c221-38e8-47cf-a8b5-305396cee703-tls-certs\") pod \"distinct-w1-5ac93f-kserve-64697cb8c9-vj9sk\" (UID: \"75a1c221-38e8-47cf-a8b5-305396cee703\") " pod="e2e-models-e2e-worker-w1-5ac93f/distinct-w1-5ac93f-kserve-64697cb8c9-vj9sk"
Nov 02 03:14:24.279725 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:24.279687    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjggc\" (UniqueName: \"kubernetes.io/projected/75a1c221-38e8-47cf-a8b5-305396cee703-kube-api-access-cjggc\") pod \"distinct-w1-5ac93f-kserve-64697cb8c9-vj9sk\" (UID: \"75a1c221-38e8-47cf-a8b5-305396cee703\") " pod="e2e-models-e2e-worker-w1-5ac93f/distinct-w1-5ac93f-kserve-64697cb8c9-vj9sk"
Nov 02 03:14:24.394274 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:24.394203    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5ac93f/distinct-w1-5ac93f-kserve-64697cb8c9-vj9sk"
Nov 02 03:14:24.785340 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:24.785307    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-5ac93f/distinct-w1-5ac93f-kserve-64697cb8c9-vj9sk"]
Nov 02 03:14:24.786075 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:14:24.786049    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod75a1c221_38e8_47cf_a8b5_305396cee703.slice/crio-f914b8941a44be80c4b6d985232f9ff1a86ac8d66f6dd6d4311c2e2be80e21de WatchSource:0}: Error finding container f914b8941a44be80c4b6d985232f9ff1a86ac8d66f6dd6d4311c2e2be80e21de: Status 404 returned error can't find the container with id f914b8941a44be80c4b6d985232f9ff1a86ac8d66f6dd6d4311c2e2be80e21de
Nov 02 03:14:25.799114 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:25.799073    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5ac93f/distinct-w1-5ac93f-kserve-64697cb8c9-vj9sk" event={"ID":"75a1c221-38e8-47cf-a8b5-305396cee703","Type":"ContainerStarted","Data":"dcf0712d1c3794925117aea7b6bdd0f9d526e48868becfc5d506b9eedc61a1ae"}
Nov 02 03:14:25.799492 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:25.799129    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5ac93f/distinct-w1-5ac93f-kserve-64697cb8c9-vj9sk" event={"ID":"75a1c221-38e8-47cf-a8b5-305396cee703","Type":"ContainerStarted","Data":"f914b8941a44be80c4b6d985232f9ff1a86ac8d66f6dd6d4311c2e2be80e21de"}
Nov 02 03:14:25.969244 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:25.969190    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-5ac93f/distinct-w1-5ac93f-kserve-64697cb8c9-vj9sk" podStartSLOduration=1.9691770119999998 podStartE2EDuration="1.969177012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:14:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:14:25.966010974 +0000 UTC m=+1606.544618356" watchObservedRunningTime="2026-11-02 03:14:25.969177012 +0000 UTC m=+1606.547784393"
Nov 02 03:14:26.394956 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:26.394925    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5ac93f/distinct-w1-5ac93f-kserve-64697cb8c9-vj9sk"
Nov 02 03:14:31.452459 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:31.452421    4496 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-76bc6b7c84-sn57j" podUID="a10bb5f4-5036-49fe-8e38-a00227e07a4a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 03:14:32.571612 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:32.571565    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-76bc6b7c84-sn57j"]
Nov 02 03:14:32.573041 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:32.572989    4496 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-76bc6b7c84-sn57j" podUID="a10bb5f4-5036-49fe-8e38-a00227e07a4a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 03:14:32.606799 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:32.606767    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-cbb4b6c58-x6vr6"]
Nov 02 03:14:32.610333 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:32.610316    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-cbb4b6c58-x6vr6"
Nov 02 03:14:32.681850 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:32.681821    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-cbb4b6c58-x6vr6"]
Nov 02 03:14:32.722914 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:32.722877    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e2cc045-6a4c-4322-b1ec-e104057041af-plugins-config-volume\") pod \"payload-processing-external-model-cbb4b6c58-x6vr6\" (UID: \"4e2cc045-6a4c-4322-b1ec-e104057041af\") " pod="models-as-a-service/payload-processing-external-model-cbb4b6c58-x6vr6"
Nov 02 03:14:32.723099 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:32.722937    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wflln\" (UniqueName: \"kubernetes.io/projected/4e2cc045-6a4c-4322-b1ec-e104057041af-kube-api-access-wflln\") pod \"payload-processing-external-model-cbb4b6c58-x6vr6\" (UID: \"4e2cc045-6a4c-4322-b1ec-e104057041af\") " pod="models-as-a-service/payload-processing-external-model-cbb4b6c58-x6vr6"
Nov 02 03:14:32.723099 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:32.723089    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4e2cc045-6a4c-4322-b1ec-e104057041af-routing-overlay-volume\") pod \"payload-processing-external-model-cbb4b6c58-x6vr6\" (UID: \"4e2cc045-6a4c-4322-b1ec-e104057041af\") " pod="models-as-a-service/payload-processing-external-model-cbb4b6c58-x6vr6"
Nov 02 03:14:32.723181 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:32.723123    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4e2cc045-6a4c-4322-b1ec-e104057041af-provider-credentials\") pod \"payload-processing-external-model-cbb4b6c58-x6vr6\" (UID: \"4e2cc045-6a4c-4322-b1ec-e104057041af\") " pod="models-as-a-service/payload-processing-external-model-cbb4b6c58-x6vr6"
Nov 02 03:14:32.824847 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:32.824734    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4e2cc045-6a4c-4322-b1ec-e104057041af-routing-overlay-volume\") pod \"payload-processing-external-model-cbb4b6c58-x6vr6\" (UID: \"4e2cc045-6a4c-4322-b1ec-e104057041af\") " pod="models-as-a-service/payload-processing-external-model-cbb4b6c58-x6vr6"
Nov 02 03:14:32.824847 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:32.824804    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4e2cc045-6a4c-4322-b1ec-e104057041af-provider-credentials\") pod \"payload-processing-external-model-cbb4b6c58-x6vr6\" (UID: \"4e2cc045-6a4c-4322-b1ec-e104057041af\") " pod="models-as-a-service/payload-processing-external-model-cbb4b6c58-x6vr6"
Nov 02 03:14:32.825121 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:32.824883    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e2cc045-6a4c-4322-b1ec-e104057041af-plugins-config-volume\") pod \"payload-processing-external-model-cbb4b6c58-x6vr6\" (UID: \"4e2cc045-6a4c-4322-b1ec-e104057041af\") " pod="models-as-a-service/payload-processing-external-model-cbb4b6c58-x6vr6"
Nov 02 03:14:32.825121 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:32.824946    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wflln\" (UniqueName: \"kubernetes.io/projected/4e2cc045-6a4c-4322-b1ec-e104057041af-kube-api-access-wflln\") pod \"payload-processing-external-model-cbb4b6c58-x6vr6\" (UID: \"4e2cc045-6a4c-4322-b1ec-e104057041af\") " pod="models-as-a-service/payload-processing-external-model-cbb4b6c58-x6vr6"
Nov 02 03:14:32.825669 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:32.825604    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4e2cc045-6a4c-4322-b1ec-e104057041af-routing-overlay-volume\") pod \"payload-processing-external-model-cbb4b6c58-x6vr6\" (UID: \"4e2cc045-6a4c-4322-b1ec-e104057041af\") " pod="models-as-a-service/payload-processing-external-model-cbb4b6c58-x6vr6"
Nov 02 03:14:32.826110 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:32.826083    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e2cc045-6a4c-4322-b1ec-e104057041af-plugins-config-volume\") pod \"payload-processing-external-model-cbb4b6c58-x6vr6\" (UID: \"4e2cc045-6a4c-4322-b1ec-e104057041af\") " pod="models-as-a-service/payload-processing-external-model-cbb4b6c58-x6vr6"
Nov 02 03:14:32.827729 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:32.827702    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4e2cc045-6a4c-4322-b1ec-e104057041af-provider-credentials\") pod \"payload-processing-external-model-cbb4b6c58-x6vr6\" (UID: \"4e2cc045-6a4c-4322-b1ec-e104057041af\") " pod="models-as-a-service/payload-processing-external-model-cbb4b6c58-x6vr6"
Nov 02 03:14:32.882696 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:32.882660    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wflln\" (UniqueName: \"kubernetes.io/projected/4e2cc045-6a4c-4322-b1ec-e104057041af-kube-api-access-wflln\") pod \"payload-processing-external-model-cbb4b6c58-x6vr6\" (UID: \"4e2cc045-6a4c-4322-b1ec-e104057041af\") " pod="models-as-a-service/payload-processing-external-model-cbb4b6c58-x6vr6"
Nov 02 03:14:32.924706 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:32.924674    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-cbb4b6c58-x6vr6"
Nov 02 03:14:33.299252 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:14:33.299222    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e2cc045_6a4c_4322_b1ec_e104057041af.slice/crio-09361bd86491238fe12ccca5684f00a095d7244d94f00069d1cd4a033d9a5603 WatchSource:0}: Error finding container 09361bd86491238fe12ccca5684f00a095d7244d94f00069d1cd4a033d9a5603: Status 404 returned error can't find the container with id 09361bd86491238fe12ccca5684f00a095d7244d94f00069d1cd4a033d9a5603
Nov 02 03:14:33.299411 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:33.299339    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-cbb4b6c58-x6vr6"]
Nov 02 03:14:33.526442 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:14:33.526404    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 03:14:33.867742 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:33.867700    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-cbb4b6c58-x6vr6" event={"ID":"4e2cc045-6a4c-4322-b1ec-e104057041af","Type":"ContainerStarted","Data":"8f0dfe8a00190fb673118cfd50a4f9ca25c40fba091d43f471b93ba147cf13ba"}
Nov 02 03:14:33.868277 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:33.867796    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-cbb4b6c58-x6vr6" event={"ID":"4e2cc045-6a4c-4322-b1ec-e104057041af","Type":"ContainerStarted","Data":"09361bd86491238fe12ccca5684f00a095d7244d94f00069d1cd4a033d9a5603"}
Nov 02 03:14:33.868277 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:33.867826    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-cbb4b6c58-x6vr6"
Nov 02 03:14:34.394393 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:34.394353    4496 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-5ac93f/distinct-w1-5ac93f-kserve-64697cb8c9-vj9sk"
Nov 02 03:14:34.407701 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:34.407674    4496 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-5ac93f/distinct-w1-5ac93f-kserve-64697cb8c9-vj9sk"
Nov 02 03:14:34.528421 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:34.528368    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-cbb4b6c58-x6vr6" podStartSLOduration=2.528355105 podStartE2EDuration="2.528355105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:14:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:14:33.983948484 +0000 UTC m=+1614.562555863" watchObservedRunningTime="2026-11-02 03:14:34.528355105 +0000 UTC m=+1615.106962537"
Nov 02 03:14:34.890504 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:34.890471    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5ac93f/distinct-w1-5ac93f-kserve-64697cb8c9-vj9sk"
Nov 02 03:14:37.571962 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:37.571917    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-76bc6b7c84-sn57j" podUID="a10bb5f4-5036-49fe-8e38-a00227e07a4a" containerName="payload-processing" containerID="cri-o://dabcf902d1f4f26cc23598499dbb632ddc375c5344aba7d12c5685d4d42a4bfb" gracePeriod=25
Nov 02 03:14:37.851089 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:37.851066    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-76bc6b7c84-sn57j"
Nov 02 03:14:37.902470 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:37.902429    4496 generic.go:356] "Generic (PLEG): container finished" podID="a10bb5f4-5036-49fe-8e38-a00227e07a4a" containerID="dabcf902d1f4f26cc23598499dbb632ddc375c5344aba7d12c5685d4d42a4bfb" exitCode=0
Nov 02 03:14:37.902647 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:37.902530    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-76bc6b7c84-sn57j" event={"ID":"a10bb5f4-5036-49fe-8e38-a00227e07a4a","Type":"ContainerDied","Data":"dabcf902d1f4f26cc23598499dbb632ddc375c5344aba7d12c5685d4d42a4bfb"}
Nov 02 03:14:37.902647 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:37.902583    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-76bc6b7c84-sn57j" event={"ID":"a10bb5f4-5036-49fe-8e38-a00227e07a4a","Type":"ContainerDied","Data":"b7e73168b12a027ea295d9dfc8cfe6187566dc7f39eee1c9a7a07ece8f94281c"}
Nov 02 03:14:37.902647 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:37.902615    4496 scope.go:122] "RemoveContainer" containerID="dabcf902d1f4f26cc23598499dbb632ddc375c5344aba7d12c5685d4d42a4bfb"
Nov 02 03:14:37.902820 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:37.902771    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-76bc6b7c84-sn57j"
Nov 02 03:14:37.914640 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:37.914616    4496 scope.go:122] "RemoveContainer" containerID="dabcf902d1f4f26cc23598499dbb632ddc375c5344aba7d12c5685d4d42a4bfb"
Nov 02 03:14:37.914935 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:14:37.914913    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dabcf902d1f4f26cc23598499dbb632ddc375c5344aba7d12c5685d4d42a4bfb\": container with ID starting with dabcf902d1f4f26cc23598499dbb632ddc375c5344aba7d12c5685d4d42a4bfb not found: ID does not exist" containerID="dabcf902d1f4f26cc23598499dbb632ddc375c5344aba7d12c5685d4d42a4bfb"
Nov 02 03:14:37.915050 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:37.914943    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dabcf902d1f4f26cc23598499dbb632ddc375c5344aba7d12c5685d4d42a4bfb"} err="failed to get container status \"dabcf902d1f4f26cc23598499dbb632ddc375c5344aba7d12c5685d4d42a4bfb\": rpc error: code = NotFound desc = could not find container \"dabcf902d1f4f26cc23598499dbb632ddc375c5344aba7d12c5685d4d42a4bfb\": container with ID starting with dabcf902d1f4f26cc23598499dbb632ddc375c5344aba7d12c5685d4d42a4bfb not found: ID does not exist"
Nov 02 03:14:37.985631 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:37.985592    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a10bb5f4-5036-49fe-8e38-a00227e07a4a-kube-api-access-5kxjl\" (UniqueName: \"kubernetes.io/projected/a10bb5f4-5036-49fe-8e38-a00227e07a4a-kube-api-access-5kxjl\") pod \"a10bb5f4-5036-49fe-8e38-a00227e07a4a\" (UID: \"a10bb5f4-5036-49fe-8e38-a00227e07a4a\") "
Nov 02 03:14:37.985829 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:37.985662    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a10bb5f4-5036-49fe-8e38-a00227e07a4a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a10bb5f4-5036-49fe-8e38-a00227e07a4a-plugins-config-volume\") pod \"a10bb5f4-5036-49fe-8e38-a00227e07a4a\" (UID: \"a10bb5f4-5036-49fe-8e38-a00227e07a4a\") "
Nov 02 03:14:37.985829 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:37.985697    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a10bb5f4-5036-49fe-8e38-a00227e07a4a-provider-credentials\" (UniqueName: \"kubernetes.io/projected/a10bb5f4-5036-49fe-8e38-a00227e07a4a-provider-credentials\") pod \"a10bb5f4-5036-49fe-8e38-a00227e07a4a\" (UID: \"a10bb5f4-5036-49fe-8e38-a00227e07a4a\") "
Nov 02 03:14:37.985829 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:37.985749    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a10bb5f4-5036-49fe-8e38-a00227e07a4a-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a10bb5f4-5036-49fe-8e38-a00227e07a4a-routing-overlay-volume\") pod \"a10bb5f4-5036-49fe-8e38-a00227e07a4a\" (UID: \"a10bb5f4-5036-49fe-8e38-a00227e07a4a\") "
Nov 02 03:14:37.986564 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:37.986536    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a10bb5f4-5036-49fe-8e38-a00227e07a4a-plugins-config-volume" pod "a10bb5f4-5036-49fe-8e38-a00227e07a4a" (UID: "a10bb5f4-5036-49fe-8e38-a00227e07a4a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:14:37.986698 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:37.986661    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a10bb5f4-5036-49fe-8e38-a00227e07a4a-routing-overlay-volume" pod "a10bb5f4-5036-49fe-8e38-a00227e07a4a" (UID: "a10bb5f4-5036-49fe-8e38-a00227e07a4a"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:14:37.989531 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:37.989504    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a10bb5f4-5036-49fe-8e38-a00227e07a4a-kube-api-access-5kxjl" pod "a10bb5f4-5036-49fe-8e38-a00227e07a4a" (UID: "a10bb5f4-5036-49fe-8e38-a00227e07a4a"). InnerVolumeSpecName "kube-api-access-5kxjl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:14:37.989927 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:37.989902    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a10bb5f4-5036-49fe-8e38-a00227e07a4a-provider-credentials" pod "a10bb5f4-5036-49fe-8e38-a00227e07a4a" (UID: "a10bb5f4-5036-49fe-8e38-a00227e07a4a"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:14:38.087227 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:38.087137    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kxjl\" (UniqueName: \"kubernetes.io/projected/a10bb5f4-5036-49fe-8e38-a00227e07a4a-kube-api-access-5kxjl\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:14:38.087227 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:38.087165    4496 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a10bb5f4-5036-49fe-8e38-a00227e07a4a-plugins-config-volume\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:14:38.087227 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:38.087176    4496 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a10bb5f4-5036-49fe-8e38-a00227e07a4a-provider-credentials\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:14:38.087227 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:38.087185    4496 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a10bb5f4-5036-49fe-8e38-a00227e07a4a-routing-overlay-volume\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:14:38.367099 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:38.366077    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-76bc6b7c84-sn57j"]
Nov 02 03:14:38.389571 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:38.389531    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-76bc6b7c84-sn57j"]
Nov 02 03:14:39.617186 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:39.617141    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a10bb5f4-5036-49fe-8e38-a00227e07a4a" path="/var/lib/kubelet/pods/a10bb5f4-5036-49fe-8e38-a00227e07a4a/volumes"
Nov 02 03:14:44.701130 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:14:44.701091    4496 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-7d506ce743a8: secret "agc-cred-7d506ce743a8" not found
Nov 02 03:14:44.701130 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:14:44.701131    4496 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6c456c64b9-drg8b: secret "agc-cred-7d506ce743a8" not found
Nov 02 03:14:44.701703 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:14:44.701195    4496 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/07a0a43a-4232-467a-b7f4-ff4772b34e33-provider-credentials podName:07a0a43a-4232-467a-b7f4-ff4772b34e33 nodeName:}" failed. No retries permitted until 2026-11-02 03:14:45.201180072 +0000 UTC m=+1625.779787569 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/07a0a43a-4232-467a-b7f4-ff4772b34e33-provider-credentials") pod "payload-processing-external-model-6c456c64b9-drg8b" (UID: "07a0a43a-4232-467a-b7f4-ff4772b34e33") : secret "agc-cred-7d506ce743a8" not found
Nov 02 03:14:44.879490 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:44.879447    4496 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-cbb4b6c58-x6vr6" podUID="4e2cc045-6a4c-4322-b1ec-e104057041af" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 03:14:45.206294 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:14:45.206263    4496 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-7d506ce743a8: secret "agc-cred-7d506ce743a8" not found
Nov 02 03:14:45.206294 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:14:45.206292    4496 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6c456c64b9-drg8b: secret "agc-cred-7d506ce743a8" not found
Nov 02 03:14:45.206504 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:14:45.206353    4496 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/07a0a43a-4232-467a-b7f4-ff4772b34e33-provider-credentials podName:07a0a43a-4232-467a-b7f4-ff4772b34e33 nodeName:}" failed. No retries permitted until 2026-11-02 03:14:46.206336985 +0000 UTC m=+1626.784944347 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/07a0a43a-4232-467a-b7f4-ff4772b34e33-provider-credentials") pod "payload-processing-external-model-6c456c64b9-drg8b" (UID: "07a0a43a-4232-467a-b7f4-ff4772b34e33") : secret "agc-cred-7d506ce743a8" not found
Nov 02 03:14:46.219979 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:14:46.219569    4496 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-7d506ce743a8: secret "agc-cred-7d506ce743a8" not found
Nov 02 03:14:46.219979 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:14:46.219614    4496 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6c456c64b9-drg8b: secret "agc-cred-7d506ce743a8" not found
Nov 02 03:14:46.219979 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:14:46.219700    4496 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/07a0a43a-4232-467a-b7f4-ff4772b34e33-provider-credentials podName:07a0a43a-4232-467a-b7f4-ff4772b34e33 nodeName:}" failed. No retries permitted until 2026-11-02 03:14:48.21967922 +0000 UTC m=+1628.798286592 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/07a0a43a-4232-467a-b7f4-ff4772b34e33-provider-credentials") pod "payload-processing-external-model-6c456c64b9-drg8b" (UID: "07a0a43a-4232-467a-b7f4-ff4772b34e33") : secret "agc-cred-7d506ce743a8" not found
Nov 02 03:14:48.239593 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:14:48.239556    4496 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-7d506ce743a8: secret "agc-cred-7d506ce743a8" not found
Nov 02 03:14:48.239593 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:14:48.239589    4496 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6c456c64b9-drg8b: secret "agc-cred-7d506ce743a8" not found
Nov 02 03:14:48.240007 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:14:48.239646    4496 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/07a0a43a-4232-467a-b7f4-ff4772b34e33-provider-credentials podName:07a0a43a-4232-467a-b7f4-ff4772b34e33 nodeName:}" failed. No retries permitted until 2026-11-02 03:14:52.239631806 +0000 UTC m=+1632.818239177 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/07a0a43a-4232-467a-b7f4-ff4772b34e33-provider-credentials") pod "payload-processing-external-model-6c456c64b9-drg8b" (UID: "07a0a43a-4232-467a-b7f4-ff4772b34e33") : secret "agc-cred-7d506ce743a8" not found
Nov 02 03:14:52.286339 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:14:52.286302    4496 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-7d506ce743a8: secret "agc-cred-7d506ce743a8" not found
Nov 02 03:14:52.286339 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:14:52.286340    4496 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6c456c64b9-drg8b: secret "agc-cred-7d506ce743a8" not found
Nov 02 03:14:52.286783 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:14:52.286422    4496 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/07a0a43a-4232-467a-b7f4-ff4772b34e33-provider-credentials podName:07a0a43a-4232-467a-b7f4-ff4772b34e33 nodeName:}" failed. No retries permitted until 2026-11-02 03:15:00.286401515 +0000 UTC m=+1640.865008878 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/07a0a43a-4232-467a-b7f4-ff4772b34e33-provider-credentials") pod "payload-processing-external-model-6c456c64b9-drg8b" (UID: "07a0a43a-4232-467a-b7f4-ff4772b34e33") : secret "agc-cred-7d506ce743a8" not found
Nov 02 03:14:52.819435 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:52.819403    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-cbb4b6c58-x6vr6"]
Nov 02 03:14:52.820686 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:52.820663    4496 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-cbb4b6c58-x6vr6" podUID="4e2cc045-6a4c-4322-b1ec-e104057041af" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 03:14:52.849635 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:52.849603    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-777d6b586f-brq7x"]
Nov 02 03:14:52.850256 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:52.850236    4496 cpu_manager.go:414] "containerMap: removing container" podUID="a10bb5f4-5036-49fe-8e38-a00227e07a4a" containerName="payload-processing"
Nov 02 03:14:52.850256 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:52.850251    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a10bb5f4-5036-49fe-8e38-a00227e07a4a" containerName="payload-processing"
Nov 02 03:14:52.850420 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:52.850406    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="a10bb5f4-5036-49fe-8e38-a00227e07a4a" containerName="payload-processing"
Nov 02 03:14:52.855373 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:52.855351    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-777d6b586f-brq7x"
Nov 02 03:14:52.908218 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:52.908183    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-06e7ce0d87b6\""
Nov 02 03:14:52.908218 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:52.908187    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-08e8b2351e52\""
Nov 02 03:14:52.929611 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:52.929579    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-777d6b586f-brq7x"]
Nov 02 03:14:52.994892 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:52.994862    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/775f32d1-c694-4cdc-8a83-d82930ffaf37-plugins-config-volume\") pod \"payload-processing-external-model-777d6b586f-brq7x\" (UID: \"775f32d1-c694-4cdc-8a83-d82930ffaf37\") " pod="models-as-a-service/payload-processing-external-model-777d6b586f-brq7x"
Nov 02 03:14:52.994892 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:52.994900    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/775f32d1-c694-4cdc-8a83-d82930ffaf37-provider-credentials\") pod \"payload-processing-external-model-777d6b586f-brq7x\" (UID: \"775f32d1-c694-4cdc-8a83-d82930ffaf37\") " pod="models-as-a-service/payload-processing-external-model-777d6b586f-brq7x"
Nov 02 03:14:52.995152 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:52.995047    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/775f32d1-c694-4cdc-8a83-d82930ffaf37-routing-overlay-volume\") pod \"payload-processing-external-model-777d6b586f-brq7x\" (UID: \"775f32d1-c694-4cdc-8a83-d82930ffaf37\") " pod="models-as-a-service/payload-processing-external-model-777d6b586f-brq7x"
Nov 02 03:14:52.995214 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:52.995193    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvbkx\" (UniqueName: \"kubernetes.io/projected/775f32d1-c694-4cdc-8a83-d82930ffaf37-kube-api-access-mvbkx\") pod \"payload-processing-external-model-777d6b586f-brq7x\" (UID: \"775f32d1-c694-4cdc-8a83-d82930ffaf37\") " pod="models-as-a-service/payload-processing-external-model-777d6b586f-brq7x"
Nov 02 03:14:53.095954 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:53.095854    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/775f32d1-c694-4cdc-8a83-d82930ffaf37-plugins-config-volume\") pod \"payload-processing-external-model-777d6b586f-brq7x\" (UID: \"775f32d1-c694-4cdc-8a83-d82930ffaf37\") " pod="models-as-a-service/payload-processing-external-model-777d6b586f-brq7x"
Nov 02 03:14:53.095954 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:53.095886    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/775f32d1-c694-4cdc-8a83-d82930ffaf37-provider-credentials\") pod \"payload-processing-external-model-777d6b586f-brq7x\" (UID: \"775f32d1-c694-4cdc-8a83-d82930ffaf37\") " pod="models-as-a-service/payload-processing-external-model-777d6b586f-brq7x"
Nov 02 03:14:53.095954 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:53.095941    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/775f32d1-c694-4cdc-8a83-d82930ffaf37-routing-overlay-volume\") pod \"payload-processing-external-model-777d6b586f-brq7x\" (UID: \"775f32d1-c694-4cdc-8a83-d82930ffaf37\") " pod="models-as-a-service/payload-processing-external-model-777d6b586f-brq7x"
Nov 02 03:14:53.096269 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:53.096059    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvbkx\" (UniqueName: \"kubernetes.io/projected/775f32d1-c694-4cdc-8a83-d82930ffaf37-kube-api-access-mvbkx\") pod \"payload-processing-external-model-777d6b586f-brq7x\" (UID: \"775f32d1-c694-4cdc-8a83-d82930ffaf37\") " pod="models-as-a-service/payload-processing-external-model-777d6b586f-brq7x"
Nov 02 03:14:53.096755 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:53.096724    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/775f32d1-c694-4cdc-8a83-d82930ffaf37-routing-overlay-volume\") pod \"payload-processing-external-model-777d6b586f-brq7x\" (UID: \"775f32d1-c694-4cdc-8a83-d82930ffaf37\") " pod="models-as-a-service/payload-processing-external-model-777d6b586f-brq7x"
Nov 02 03:14:53.096867 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:53.096794    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/775f32d1-c694-4cdc-8a83-d82930ffaf37-plugins-config-volume\") pod \"payload-processing-external-model-777d6b586f-brq7x\" (UID: \"775f32d1-c694-4cdc-8a83-d82930ffaf37\") " pod="models-as-a-service/payload-processing-external-model-777d6b586f-brq7x"
Nov 02 03:14:53.098771 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:53.098749    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/775f32d1-c694-4cdc-8a83-d82930ffaf37-provider-credentials\") pod \"payload-processing-external-model-777d6b586f-brq7x\" (UID: \"775f32d1-c694-4cdc-8a83-d82930ffaf37\") " pod="models-as-a-service/payload-processing-external-model-777d6b586f-brq7x"
Nov 02 03:14:53.155304 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:53.155271    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvbkx\" (UniqueName: \"kubernetes.io/projected/775f32d1-c694-4cdc-8a83-d82930ffaf37-kube-api-access-mvbkx\") pod \"payload-processing-external-model-777d6b586f-brq7x\" (UID: \"775f32d1-c694-4cdc-8a83-d82930ffaf37\") " pod="models-as-a-service/payload-processing-external-model-777d6b586f-brq7x"
Nov 02 03:14:53.169777 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:53.169749    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-777d6b586f-brq7x"
Nov 02 03:14:53.548875 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:14:53.548832    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod775f32d1_c694_4cdc_8a83_d82930ffaf37.slice/crio-693d81bd97f75de38bbf8d1d578b99264a16d90c0b3162ef2116537bd42a186d WatchSource:0}: Error finding container 693d81bd97f75de38bbf8d1d578b99264a16d90c0b3162ef2116537bd42a186d: Status 404 returned error can't find the container with id 693d81bd97f75de38bbf8d1d578b99264a16d90c0b3162ef2116537bd42a186d
Nov 02 03:14:53.549337 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:53.549045    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-777d6b586f-brq7x"]
Nov 02 03:14:54.089323 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:54.089285    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-777d6b586f-brq7x" event={"ID":"775f32d1-c694-4cdc-8a83-d82930ffaf37","Type":"ContainerStarted","Data":"3cfe823620cb0594ab0260c8a31e8cc8743c7a50b49291679b36ffce70b6416e"}
Nov 02 03:14:54.089525 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:54.089338    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-777d6b586f-brq7x" event={"ID":"775f32d1-c694-4cdc-8a83-d82930ffaf37","Type":"ContainerStarted","Data":"693d81bd97f75de38bbf8d1d578b99264a16d90c0b3162ef2116537bd42a186d"}
Nov 02 03:14:54.089525 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:54.089382    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-777d6b586f-brq7x"
Nov 02 03:14:54.205531 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:54.205468    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-777d6b586f-brq7x" podStartSLOduration=2.20545546 podStartE2EDuration="2.20545546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:14:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:14:54.205069869 +0000 UTC m=+1634.783677248" watchObservedRunningTime="2026-11-02 03:14:54.20545546 +0000 UTC m=+1634.784062839"
Nov 02 03:14:57.820653 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:57.820609    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-cbb4b6c58-x6vr6" podUID="4e2cc045-6a4c-4322-b1ec-e104057041af" containerName="payload-processing" containerID="cri-o://8f0dfe8a00190fb673118cfd50a4f9ca25c40fba091d43f471b93ba147cf13ba" gracePeriod=25
Nov 02 03:14:58.088827 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:58.088765    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-cbb4b6c58-x6vr6"
Nov 02 03:14:58.121078 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:58.121046    4496 generic.go:356] "Generic (PLEG): container finished" podID="4e2cc045-6a4c-4322-b1ec-e104057041af" containerID="8f0dfe8a00190fb673118cfd50a4f9ca25c40fba091d43f471b93ba147cf13ba" exitCode=0
Nov 02 03:14:58.121253 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:58.121088    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-cbb4b6c58-x6vr6"
Nov 02 03:14:58.121253 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:58.121114    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-cbb4b6c58-x6vr6" event={"ID":"4e2cc045-6a4c-4322-b1ec-e104057041af","Type":"ContainerDied","Data":"8f0dfe8a00190fb673118cfd50a4f9ca25c40fba091d43f471b93ba147cf13ba"}
Nov 02 03:14:58.121253 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:58.121172    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-cbb4b6c58-x6vr6" event={"ID":"4e2cc045-6a4c-4322-b1ec-e104057041af","Type":"ContainerDied","Data":"09361bd86491238fe12ccca5684f00a095d7244d94f00069d1cd4a033d9a5603"}
Nov 02 03:14:58.121253 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:58.121195    4496 scope.go:122] "RemoveContainer" containerID="8f0dfe8a00190fb673118cfd50a4f9ca25c40fba091d43f471b93ba147cf13ba"
Nov 02 03:14:58.131145 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:58.131127    4496 scope.go:122] "RemoveContainer" containerID="8f0dfe8a00190fb673118cfd50a4f9ca25c40fba091d43f471b93ba147cf13ba"
Nov 02 03:14:58.131430 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:14:58.131408    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f0dfe8a00190fb673118cfd50a4f9ca25c40fba091d43f471b93ba147cf13ba\": container with ID starting with 8f0dfe8a00190fb673118cfd50a4f9ca25c40fba091d43f471b93ba147cf13ba not found: ID does not exist" containerID="8f0dfe8a00190fb673118cfd50a4f9ca25c40fba091d43f471b93ba147cf13ba"
Nov 02 03:14:58.131481 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:58.131439    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f0dfe8a00190fb673118cfd50a4f9ca25c40fba091d43f471b93ba147cf13ba"} err="failed to get container status \"8f0dfe8a00190fb673118cfd50a4f9ca25c40fba091d43f471b93ba147cf13ba\": rpc error: code = NotFound desc = could not find container \"8f0dfe8a00190fb673118cfd50a4f9ca25c40fba091d43f471b93ba147cf13ba\": container with ID starting with 8f0dfe8a00190fb673118cfd50a4f9ca25c40fba091d43f471b93ba147cf13ba not found: ID does not exist"
Nov 02 03:14:58.255008 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:58.254975    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e2cc045-6a4c-4322-b1ec-e104057041af-provider-credentials\" (UniqueName: \"kubernetes.io/projected/4e2cc045-6a4c-4322-b1ec-e104057041af-provider-credentials\") pod \"4e2cc045-6a4c-4322-b1ec-e104057041af\" (UID: \"4e2cc045-6a4c-4322-b1ec-e104057041af\") "
Nov 02 03:14:58.255008 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:58.255028    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e2cc045-6a4c-4322-b1ec-e104057041af-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4e2cc045-6a4c-4322-b1ec-e104057041af-routing-overlay-volume\") pod \"4e2cc045-6a4c-4322-b1ec-e104057041af\" (UID: \"4e2cc045-6a4c-4322-b1ec-e104057041af\") "
Nov 02 03:14:58.255284 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:58.255073    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e2cc045-6a4c-4322-b1ec-e104057041af-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e2cc045-6a4c-4322-b1ec-e104057041af-plugins-config-volume\") pod \"4e2cc045-6a4c-4322-b1ec-e104057041af\" (UID: \"4e2cc045-6a4c-4322-b1ec-e104057041af\") "
Nov 02 03:14:58.255343 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:58.255303    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e2cc045-6a4c-4322-b1ec-e104057041af-kube-api-access-wflln\" (UniqueName: \"kubernetes.io/projected/4e2cc045-6a4c-4322-b1ec-e104057041af-kube-api-access-wflln\") pod \"4e2cc045-6a4c-4322-b1ec-e104057041af\" (UID: \"4e2cc045-6a4c-4322-b1ec-e104057041af\") "
Nov 02 03:14:58.255466 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:58.255440    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e2cc045-6a4c-4322-b1ec-e104057041af-routing-overlay-volume" pod "4e2cc045-6a4c-4322-b1ec-e104057041af" (UID: "4e2cc045-6a4c-4322-b1ec-e104057041af"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:14:58.255525 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:58.255480    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e2cc045-6a4c-4322-b1ec-e104057041af-plugins-config-volume" pod "4e2cc045-6a4c-4322-b1ec-e104057041af" (UID: "4e2cc045-6a4c-4322-b1ec-e104057041af"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:14:58.255914 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:58.255878    4496 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4e2cc045-6a4c-4322-b1ec-e104057041af-routing-overlay-volume\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:14:58.256062 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:58.255915    4496 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e2cc045-6a4c-4322-b1ec-e104057041af-plugins-config-volume\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:14:58.257404 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:58.257375    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e2cc045-6a4c-4322-b1ec-e104057041af-kube-api-access-wflln" pod "4e2cc045-6a4c-4322-b1ec-e104057041af" (UID: "4e2cc045-6a4c-4322-b1ec-e104057041af"). InnerVolumeSpecName "kube-api-access-wflln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:14:58.257519 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:58.257472    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e2cc045-6a4c-4322-b1ec-e104057041af-provider-credentials" pod "4e2cc045-6a4c-4322-b1ec-e104057041af" (UID: "4e2cc045-6a4c-4322-b1ec-e104057041af"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:14:58.357282 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:58.357202    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wflln\" (UniqueName: \"kubernetes.io/projected/4e2cc045-6a4c-4322-b1ec-e104057041af-kube-api-access-wflln\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:14:58.357282 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:58.357229    4496 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4e2cc045-6a4c-4322-b1ec-e104057041af-provider-credentials\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:14:58.554507 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:58.554470    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-cbb4b6c58-x6vr6"]
Nov 02 03:14:58.559727 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:58.559699    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-cbb4b6c58-x6vr6"]
Nov 02 03:14:59.577007 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:14:59.576971    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e2cc045-6a4c-4322-b1ec-e104057041af" path="/var/lib/kubelet/pods/4e2cc045-6a4c-4322-b1ec-e104057041af/volumes"
Nov 02 03:15:00.169913 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:00.169854    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-858236-29893155-8f9z6"]
Nov 02 03:15:00.169913 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:00.170802    4496 cpu_manager.go:414] "containerMap: removing container" podUID="4e2cc045-6a4c-4322-b1ec-e104057041af" containerName="payload-processing"
Nov 02 03:15:00.169913 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:00.170824    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e2cc045-6a4c-4322-b1ec-e104057041af" containerName="payload-processing"
Nov 02 03:15:00.169913 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:00.170971    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e2cc045-6a4c-4322-b1ec-e104057041af" containerName="payload-processing"
Nov 02 03:15:00.176176 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:00.176156    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ca6546-29893155-cgltk"]
Nov 02 03:15:00.176321 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:00.176293    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-858236-29893155-8f9z6"
Nov 02 03:15:00.180827 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:00.180804    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893155-7zcl8"]
Nov 02 03:15:00.180975 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:00.180910    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ca6546-29893155-cgltk"
Nov 02 03:15:00.185216 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:00.184962    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893155-7zcl8"
Nov 02 03:15:00.220331 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:00.220301    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac93f-29893155-69hl6"]
Nov 02 03:15:00.224559 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:00.224532    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac93f-29893155-69hl6"
Nov 02 03:15:00.229969 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:00.229938    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 03:15:00.229969 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:00.229947    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 03:15:00.310040 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:00.309074    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893155-7zcl8"]
Nov 02 03:15:00.310040 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:00.309157    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ca6546-29893155-cgltk"]
Nov 02 03:15:00.315776 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:00.315730    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c457ead-4e87-4c5c-9082-4978277f7043-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-5ac93f-29893155-69hl6\" (UID: \"9c457ead-4e87-4c5c-9082-4978277f7043\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac93f-29893155-69hl6"
Nov 02 03:15:00.315959 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:00.315790    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vb7rj\" (UniqueName: \"kubernetes.io/projected/0e9a33a4-c8f9-4f47-9d62-dde798ecd364-kube-api-access-vb7rj\") pod \"maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893155-7zcl8\" (UID: \"0e9a33a4-c8f9-4f47-9d62-dde798ecd364\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893155-7zcl8"
Nov 02 03:15:00.315959 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:00.315827    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4t7g\" (UniqueName: \"kubernetes.io/projected/a4216a4f-039c-4954-80d4-513838584e46-kube-api-access-p4t7g\") pod \"maas-api-key-cleanup-e2e-worker-w2-ca6546-29893155-cgltk\" (UID: \"a4216a4f-039c-4954-80d4-513838584e46\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ca6546-29893155-cgltk"
Nov 02 03:15:00.315959 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:00.315889    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f94649fd-ac69-42db-bdf8-01b2befa5ca9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-858236-29893155-8f9z6\" (UID: \"f94649fd-ac69-42db-bdf8-01b2befa5ca9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-858236-29893155-8f9z6"
Nov 02 03:15:00.315959 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:00.315938    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4216a4f-039c-4954-80d4-513838584e46-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-ca6546-29893155-cgltk\" (UID: \"a4216a4f-039c-4954-80d4-513838584e46\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ca6546-29893155-cgltk"
Nov 02 03:15:00.316205 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:00.315965    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7h5j\" (UniqueName: \"kubernetes.io/projected/9c457ead-4e87-4c5c-9082-4978277f7043-kube-api-access-k7h5j\") pod \"maas-api-key-cleanup-e2e-worker-w1-5ac93f-29893155-69hl6\" (UID: \"9c457ead-4e87-4c5c-9082-4978277f7043\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac93f-29893155-69hl6"
Nov 02 03:15:00.316205 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:00.316041    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t25wl\" (UniqueName: \"kubernetes.io/projected/f94649fd-ac69-42db-bdf8-01b2befa5ca9-kube-api-access-t25wl\") pod \"maas-api-key-cleanup-e2e-worker-w3-858236-29893155-8f9z6\" (UID: \"f94649fd-ac69-42db-bdf8-01b2befa5ca9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-858236-29893155-8f9z6"
Nov 02 03:15:00.316205 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:00.316128    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e9a33a4-c8f9-4f47-9d62-dde798ecd364-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893155-7zcl8\" (UID: \"0e9a33a4-c8f9-4f47-9d62-dde798ecd364\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893155-7zcl8"
Nov 02 03:15:00.316384 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:15:00.316363    4496 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-7d506ce743a8: secret "agc-cred-7d506ce743a8" not found
Nov 02 03:15:00.316437 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:15:00.316389    4496 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6c456c64b9-drg8b: secret "agc-cred-7d506ce743a8" not found
Nov 02 03:15:00.316437 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:15:00.316433    4496 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/07a0a43a-4232-467a-b7f4-ff4772b34e33-provider-credentials podName:07a0a43a-4232-467a-b7f4-ff4772b34e33 nodeName:}" failed. No retries permitted until 2026-11-02 03:15:16.316417904 +0000 UTC m=+1656.895025262 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/07a0a43a-4232-467a-b7f4-ff4772b34e33-provider-credentials") pod "payload-processing-external-model-6c456c64b9-drg8b" (UID: "07a0a43a-4232-467a-b7f4-ff4772b34e33") : secret "agc-cred-7d506ce743a8" not found
Nov 02 03:15:00.324160 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:00.324129    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9z5bk\""
Nov 02 03:15:00.379571 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:00.379535    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-858236-29893155-8f9z6"]
Nov 02 03:15:00.417713 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:00.417677    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c457ead-4e87-4c5c-9082-4978277f7043-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-5ac93f-29893155-69hl6\" (UID: \"9c457ead-4e87-4c5c-9082-4978277f7043\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac93f-29893155-69hl6"
Nov 02 03:15:00.417910 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:00.417731    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb7rj\" (UniqueName: \"kubernetes.io/projected/0e9a33a4-c8f9-4f47-9d62-dde798ecd364-kube-api-access-vb7rj\") pod \"maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893155-7zcl8\" (UID: \"0e9a33a4-c8f9-4f47-9d62-dde798ecd364\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893155-7zcl8"
Nov 02 03:15:00.417910 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:00.417755    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4t7g\" (UniqueName: \"kubernetes.io/projected/a4216a4f-039c-4954-80d4-513838584e46-kube-api-access-p4t7g\") pod \"maas-api-key-cleanup-e2e-worker-w2-ca6546-29893155-cgltk\" (UID: \"a4216a4f-039c-4954-80d4-513838584e46\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ca6546-29893155-cgltk"
Nov 02 03:15:00.417910 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:00.417798    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f94649fd-ac69-42db-bdf8-01b2befa5ca9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-858236-29893155-8f9z6\" (UID: \"f94649fd-ac69-42db-bdf8-01b2befa5ca9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-858236-29893155-8f9z6"
Nov 02 03:15:00.417910 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:00.417842    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4216a4f-039c-4954-80d4-513838584e46-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-ca6546-29893155-cgltk\" (UID: \"a4216a4f-039c-4954-80d4-513838584e46\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ca6546-29893155-cgltk"
Nov 02 03:15:00.417910 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:00.417867    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7h5j\" (UniqueName: \"kubernetes.io/projected/9c457ead-4e87-4c5c-9082-4978277f7043-kube-api-access-k7h5j\") pod \"maas-api-key-cleanup-e2e-worker-w1-5ac93f-29893155-69hl6\" (UID: \"9c457ead-4e87-4c5c-9082-4978277f7043\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac93f-29893155-69hl6"
Nov 02 03:15:00.417910 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:00.417909    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t25wl\" (UniqueName: \"kubernetes.io/projected/f94649fd-ac69-42db-bdf8-01b2befa5ca9-kube-api-access-t25wl\") pod \"maas-api-key-cleanup-e2e-worker-w3-858236-29893155-8f9z6\" (UID: \"f94649fd-ac69-42db-bdf8-01b2befa5ca9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-858236-29893155-8f9z6"
Nov 02 03:15:00.418296 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:00.417985    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e9a33a4-c8f9-4f47-9d62-dde798ecd364-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893155-7zcl8\" (UID: \"0e9a33a4-c8f9-4f47-9d62-dde798ecd364\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893155-7zcl8"
Nov 02 03:15:00.418576 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:00.418544    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c457ead-4e87-4c5c-9082-4978277f7043-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-5ac93f-29893155-69hl6\" (UID: \"9c457ead-4e87-4c5c-9082-4978277f7043\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac93f-29893155-69hl6"
Nov 02 03:15:00.418784 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:00.418753    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e9a33a4-c8f9-4f47-9d62-dde798ecd364-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893155-7zcl8\" (UID: \"0e9a33a4-c8f9-4f47-9d62-dde798ecd364\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893155-7zcl8"
Nov 02 03:15:00.418868 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:00.418758    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4216a4f-039c-4954-80d4-513838584e46-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-ca6546-29893155-cgltk\" (UID: \"a4216a4f-039c-4954-80d4-513838584e46\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ca6546-29893155-cgltk"
Nov 02 03:15:00.419372 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:00.419348    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f94649fd-ac69-42db-bdf8-01b2befa5ca9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-858236-29893155-8f9z6\" (UID: \"f94649fd-ac69-42db-bdf8-01b2befa5ca9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-858236-29893155-8f9z6"
Nov 02 03:15:00.475815 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:00.475777    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vb7rj\" (UniqueName: \"kubernetes.io/projected/0e9a33a4-c8f9-4f47-9d62-dde798ecd364-kube-api-access-vb7rj\") pod \"maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893155-7zcl8\" (UID: \"0e9a33a4-c8f9-4f47-9d62-dde798ecd364\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893155-7zcl8"
Nov 02 03:15:00.476066 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:00.476008    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7h5j\" (UniqueName: \"kubernetes.io/projected/9c457ead-4e87-4c5c-9082-4978277f7043-kube-api-access-k7h5j\") pod \"maas-api-key-cleanup-e2e-worker-w1-5ac93f-29893155-69hl6\" (UID: \"9c457ead-4e87-4c5c-9082-4978277f7043\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac93f-29893155-69hl6"
Nov 02 03:15:00.477170 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:00.477147    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4t7g\" (UniqueName: \"kubernetes.io/projected/a4216a4f-039c-4954-80d4-513838584e46-kube-api-access-p4t7g\") pod \"maas-api-key-cleanup-e2e-worker-w2-ca6546-29893155-cgltk\" (UID: \"a4216a4f-039c-4954-80d4-513838584e46\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ca6546-29893155-cgltk"
Nov 02 03:15:00.477490 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:00.477459    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t25wl\" (UniqueName: \"kubernetes.io/projected/f94649fd-ac69-42db-bdf8-01b2befa5ca9-kube-api-access-t25wl\") pod \"maas-api-key-cleanup-e2e-worker-w3-858236-29893155-8f9z6\" (UID: \"f94649fd-ac69-42db-bdf8-01b2befa5ca9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-858236-29893155-8f9z6"
Nov 02 03:15:00.493919 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:00.493890    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-858236-29893155-8f9z6"
Nov 02 03:15:00.500475 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:00.500454    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ca6546-29893155-cgltk"
Nov 02 03:15:00.507568 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:00.507422    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893155-7zcl8"
Nov 02 03:15:00.507568 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:00.507468    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac93f-29893155-69hl6"]
Nov 02 03:15:00.573313 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:00.560679    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac93f-29893155-69hl6"
Nov 02 03:15:01.309348 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:01.308563    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ca6546-29893155-cgltk"]
Nov 02 03:15:01.356754 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:15:01.346798    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf94649fd_ac69_42db_bdf8_01b2befa5ca9.slice/crio-b897600349d280db8f8c3be25c31d0f2a1d04221d8cbaa927e523d3c956ad5b9 WatchSource:0}: Error finding container b897600349d280db8f8c3be25c31d0f2a1d04221d8cbaa927e523d3c956ad5b9: Status 404 returned error can't find the container with id b897600349d280db8f8c3be25c31d0f2a1d04221d8cbaa927e523d3c956ad5b9
Nov 02 03:15:01.360147 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:01.360068    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893155-7zcl8"]
Nov 02 03:15:01.365900 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:01.364378    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-858236-29893155-8f9z6"]
Nov 02 03:15:01.409156 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:01.407223    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac93f-29893155-69hl6"]
Nov 02 03:15:02.202039 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:02.201762    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-858236-29893155-8f9z6" event={"ID":"f94649fd-ac69-42db-bdf8-01b2befa5ca9","Type":"ContainerStarted","Data":"dc4b641905f4c0acf8cffe69bf59bcbbf8c287c778079fac1066a5f70f07f9f2"}
Nov 02 03:15:02.202039 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:02.201836    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-858236-29893155-8f9z6" event={"ID":"f94649fd-ac69-42db-bdf8-01b2befa5ca9","Type":"ContainerStarted","Data":"b897600349d280db8f8c3be25c31d0f2a1d04221d8cbaa927e523d3c956ad5b9"}
Nov 02 03:15:02.207036 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:02.205213    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893155-7zcl8" event={"ID":"0e9a33a4-c8f9-4f47-9d62-dde798ecd364","Type":"ContainerStarted","Data":"8039cebae499add1892f293cf6e3f83f2cee8c426adea78141b3138c7c15e5a4"}
Nov 02 03:15:02.207036 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:02.205271    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893155-7zcl8" event={"ID":"0e9a33a4-c8f9-4f47-9d62-dde798ecd364","Type":"ContainerStarted","Data":"67ce4ae9e66b42424cc5728fa3b3b651c70d21fc60fd0d49051c036dbdedbf15"}
Nov 02 03:15:02.224365 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:02.224329    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ca6546-29893155-cgltk" event={"ID":"a4216a4f-039c-4954-80d4-513838584e46","Type":"ContainerStarted","Data":"479cbaad159ebc83a777f9575991cfc45c32a3720dab467c6cfdf1be8e2961ad"}
Nov 02 03:15:02.224526 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:02.224390    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ca6546-29893155-cgltk" event={"ID":"a4216a4f-039c-4954-80d4-513838584e46","Type":"ContainerStarted","Data":"e8ba1a7bb580778995b3cba7d15d92ebac36697ee7f9d50d7fdf731b171debbb"}
Nov 02 03:15:02.226971 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:02.226942    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac93f-29893155-69hl6" event={"ID":"9c457ead-4e87-4c5c-9082-4978277f7043","Type":"ContainerStarted","Data":"a256d92afca1c0d18f53b30bd1f0fa9b225dd91f33ee6f4c0313c88040e08720"}
Nov 02 03:15:02.227101 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:02.226986    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac93f-29893155-69hl6" event={"ID":"9c457ead-4e87-4c5c-9082-4978277f7043","Type":"ContainerStarted","Data":"8a995bf7a7be679362775d1e65865d95b533bf5f7c9a830efbc250abf1dea27a"}
Nov 02 03:15:02.331734 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:02.331680    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-858236-29893155-8f9z6" podStartSLOduration=2.331658162 podStartE2EDuration="2.331658162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:15:02.32992405 +0000 UTC m=+1642.908531429" watchObservedRunningTime="2026-11-02 03:15:02.331658162 +0000 UTC m=+1642.910265542"
Nov 02 03:15:02.485098 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:02.474846    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac93f-29893155-69hl6" podStartSLOduration=2.474822752 podStartE2EDuration="2.474822752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:15:02.453859786 +0000 UTC m=+1643.032467168" watchObservedRunningTime="2026-11-02 03:15:02.474822752 +0000 UTC m=+1643.053430133"
Nov 02 03:15:02.584050 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:02.577659    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893155-7zcl8" podStartSLOduration=2.577629214 podStartE2EDuration="2.577629214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:15:02.576680122 +0000 UTC m=+1643.155287506" watchObservedRunningTime="2026-11-02 03:15:02.577629214 +0000 UTC m=+1643.156236598"
Nov 02 03:15:03.248083 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:03.243505    4496 generic.go:356] "Generic (PLEG): container finished" podID="0e9a33a4-c8f9-4f47-9d62-dde798ecd364" containerID="8039cebae499add1892f293cf6e3f83f2cee8c426adea78141b3138c7c15e5a4" exitCode=60
Nov 02 03:15:03.248083 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:03.243725    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893155-7zcl8" event={"ID":"0e9a33a4-c8f9-4f47-9d62-dde798ecd364","Type":"ContainerDied","Data":"8039cebae499add1892f293cf6e3f83f2cee8c426adea78141b3138c7c15e5a4"}
Nov 02 03:15:03.249040 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:03.248808    4496 scope.go:122] "RemoveContainer" containerID="8039cebae499add1892f293cf6e3f83f2cee8c426adea78141b3138c7c15e5a4"
Nov 02 03:15:03.275154 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:03.273590    4496 generic.go:356] "Generic (PLEG): container finished" podID="a4216a4f-039c-4954-80d4-513838584e46" containerID="479cbaad159ebc83a777f9575991cfc45c32a3720dab467c6cfdf1be8e2961ad" exitCode=60
Nov 02 03:15:03.275154 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:03.273732    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ca6546-29893155-cgltk" event={"ID":"a4216a4f-039c-4954-80d4-513838584e46","Type":"ContainerDied","Data":"479cbaad159ebc83a777f9575991cfc45c32a3720dab467c6cfdf1be8e2961ad"}
Nov 02 03:15:03.275154 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:03.274573    4496 scope.go:122] "RemoveContainer" containerID="479cbaad159ebc83a777f9575991cfc45c32a3720dab467c6cfdf1be8e2961ad"
Nov 02 03:15:03.286272 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:03.285319    4496 generic.go:356] "Generic (PLEG): container finished" podID="9c457ead-4e87-4c5c-9082-4978277f7043" containerID="a256d92afca1c0d18f53b30bd1f0fa9b225dd91f33ee6f4c0313c88040e08720" exitCode=60
Nov 02 03:15:03.286272 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:03.285437    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac93f-29893155-69hl6" event={"ID":"9c457ead-4e87-4c5c-9082-4978277f7043","Type":"ContainerDied","Data":"a256d92afca1c0d18f53b30bd1f0fa9b225dd91f33ee6f4c0313c88040e08720"}
Nov 02 03:15:03.286272 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:03.285873    4496 scope.go:122] "RemoveContainer" containerID="a256d92afca1c0d18f53b30bd1f0fa9b225dd91f33ee6f4c0313c88040e08720"
Nov 02 03:15:03.316151 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:03.315398    4496 generic.go:356] "Generic (PLEG): container finished" podID="f94649fd-ac69-42db-bdf8-01b2befa5ca9" containerID="dc4b641905f4c0acf8cffe69bf59bcbbf8c287c778079fac1066a5f70f07f9f2" exitCode=60
Nov 02 03:15:03.316151 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:03.315473    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-858236-29893155-8f9z6" event={"ID":"f94649fd-ac69-42db-bdf8-01b2befa5ca9","Type":"ContainerDied","Data":"dc4b641905f4c0acf8cffe69bf59bcbbf8c287c778079fac1066a5f70f07f9f2"}
Nov 02 03:15:03.316151 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:03.315821    4496 scope.go:122] "RemoveContainer" containerID="dc4b641905f4c0acf8cffe69bf59bcbbf8c287c778079fac1066a5f70f07f9f2"
Nov 02 03:15:03.379040 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:03.367735    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ca6546-29893155-cgltk" podStartSLOduration=3.367716832 podStartE2EDuration="3.367716832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:15:02.726333165 +0000 UTC m=+1643.304940546" watchObservedRunningTime="2026-11-02 03:15:03.367716832 +0000 UTC m=+1643.946324212"
Nov 02 03:15:03.715060 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:15:03.712384    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 03:15:04.324333 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:04.324292    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ca6546-29893155-cgltk" event={"ID":"a4216a4f-039c-4954-80d4-513838584e46","Type":"ContainerStarted","Data":"14513a37f261089447c96c906c537ec13782d785b6628cdad89c0bc36acfcdb7"}
Nov 02 03:15:04.326208 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:04.326184    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac93f-29893155-69hl6" event={"ID":"9c457ead-4e87-4c5c-9082-4978277f7043","Type":"ContainerStarted","Data":"f4ece4633cb57518d9649fa945983a49b966ef5903a2ba5d710f501ddda99245"}
Nov 02 03:15:04.328122 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:04.328100    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-858236-29893155-8f9z6" event={"ID":"f94649fd-ac69-42db-bdf8-01b2befa5ca9","Type":"ContainerStarted","Data":"b4a750354757d5e7fc71497df779fae2d66d1da6d1c79fbc4725541dd7fe3dc4"}
Nov 02 03:15:04.329924 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:04.329902    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893155-7zcl8" event={"ID":"0e9a33a4-c8f9-4f47-9d62-dde798ecd364","Type":"ContainerStarted","Data":"368e0310317004fef64394ad11a95b1badc1852be87030a61f7ae61d27b26028"}
Nov 02 03:15:04.671792 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:04.671751    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-s9dfr"]
Nov 02 03:15:04.676990 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:04.676963    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-s9dfr"
Nov 02 03:15:04.721437 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:04.721404    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e667364-fba5-4584-9251-773ae8e1f2e7-istio-data\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-s9dfr\" (UID: \"9e667364-fba5-4584-9251-773ae8e1f2e7\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-s9dfr"
Nov 02 03:15:04.721437 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:04.721437    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e667364-fba5-4584-9251-773ae8e1f2e7-istio-token\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-s9dfr\" (UID: \"9e667364-fba5-4584-9251-773ae8e1f2e7\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-s9dfr"
Nov 02 03:15:04.721617 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:04.721521    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e667364-fba5-4584-9251-773ae8e1f2e7-istiod-ca-cert\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-s9dfr\" (UID: \"9e667364-fba5-4584-9251-773ae8e1f2e7\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-s9dfr"
Nov 02 03:15:04.721617 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:04.721557    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e667364-fba5-4584-9251-773ae8e1f2e7-istio-podinfo\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-s9dfr\" (UID: \"9e667364-fba5-4584-9251-773ae8e1f2e7\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-s9dfr"
Nov 02 03:15:04.721617 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:04.721607    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e667364-fba5-4584-9251-773ae8e1f2e7-istio-envoy\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-s9dfr\" (UID: \"9e667364-fba5-4584-9251-773ae8e1f2e7\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-s9dfr"
Nov 02 03:15:04.721731 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:04.721705    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e667364-fba5-4584-9251-773ae8e1f2e7-credential-socket\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-s9dfr\" (UID: \"9e667364-fba5-4584-9251-773ae8e1f2e7\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-s9dfr"
Nov 02 03:15:04.721763 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:04.721748    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e667364-fba5-4584-9251-773ae8e1f2e7-workload-socket\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-s9dfr\" (UID: \"9e667364-fba5-4584-9251-773ae8e1f2e7\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-s9dfr"
Nov 02 03:15:04.721803 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:04.721775    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e667364-fba5-4584-9251-773ae8e1f2e7-workload-certs\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-s9dfr\" (UID: \"9e667364-fba5-4584-9251-773ae8e1f2e7\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-s9dfr"
Nov 02 03:15:04.721868 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:04.721854    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmh7s\" (UniqueName: \"kubernetes.io/projected/9e667364-fba5-4584-9251-773ae8e1f2e7-kube-api-access-cmh7s\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-s9dfr\" (UID: \"9e667364-fba5-4584-9251-773ae8e1f2e7\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-s9dfr"
Nov 02 03:15:04.730112 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:04.730088    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-5ac93f-openshift-default-dockercfg-cjnld\""
Nov 02 03:15:04.731534 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:04.731512    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-s9dfr"]
Nov 02 03:15:04.822972 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:04.822938    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmh7s\" (UniqueName: \"kubernetes.io/projected/9e667364-fba5-4584-9251-773ae8e1f2e7-kube-api-access-cmh7s\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-s9dfr\" (UID: \"9e667364-fba5-4584-9251-773ae8e1f2e7\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-s9dfr"
Nov 02 03:15:04.823162 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:04.822991    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e667364-fba5-4584-9251-773ae8e1f2e7-istio-data\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-s9dfr\" (UID: \"9e667364-fba5-4584-9251-773ae8e1f2e7\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-s9dfr"
Nov 02 03:15:04.823162 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:04.823009    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e667364-fba5-4584-9251-773ae8e1f2e7-istio-token\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-s9dfr\" (UID: \"9e667364-fba5-4584-9251-773ae8e1f2e7\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-s9dfr"
Nov 02 03:15:04.823162 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:04.823069    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e667364-fba5-4584-9251-773ae8e1f2e7-istiod-ca-cert\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-s9dfr\" (UID: \"9e667364-fba5-4584-9251-773ae8e1f2e7\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-s9dfr"
Nov 02 03:15:04.823162 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:04.823099    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e667364-fba5-4584-9251-773ae8e1f2e7-istio-podinfo\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-s9dfr\" (UID: \"9e667364-fba5-4584-9251-773ae8e1f2e7\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-s9dfr"
Nov 02 03:15:04.823162 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:04.823150    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e667364-fba5-4584-9251-773ae8e1f2e7-istio-envoy\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-s9dfr\" (UID: \"9e667364-fba5-4584-9251-773ae8e1f2e7\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-s9dfr"
Nov 02 03:15:04.823382 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:04.823187    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e667364-fba5-4584-9251-773ae8e1f2e7-credential-socket\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-s9dfr\" (UID: \"9e667364-fba5-4584-9251-773ae8e1f2e7\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-s9dfr"
Nov 02 03:15:04.823382 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:04.823233    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e667364-fba5-4584-9251-773ae8e1f2e7-workload-socket\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-s9dfr\" (UID: \"9e667364-fba5-4584-9251-773ae8e1f2e7\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-s9dfr"
Nov 02 03:15:04.823382 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:04.823267    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e667364-fba5-4584-9251-773ae8e1f2e7-workload-certs\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-s9dfr\" (UID: \"9e667364-fba5-4584-9251-773ae8e1f2e7\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-s9dfr"
Nov 02 03:15:04.823579 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:04.823553    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e667364-fba5-4584-9251-773ae8e1f2e7-istio-data\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-s9dfr\" (UID: \"9e667364-fba5-4584-9251-773ae8e1f2e7\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-s9dfr"
Nov 02 03:15:04.823646 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:04.823602    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e667364-fba5-4584-9251-773ae8e1f2e7-credential-socket\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-s9dfr\" (UID: \"9e667364-fba5-4584-9251-773ae8e1f2e7\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-s9dfr"
Nov 02 03:15:04.823757 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:04.823730    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e667364-fba5-4584-9251-773ae8e1f2e7-workload-socket\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-s9dfr\" (UID: \"9e667364-fba5-4584-9251-773ae8e1f2e7\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-s9dfr"
Nov 02 03:15:04.823809 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:04.823787    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e667364-fba5-4584-9251-773ae8e1f2e7-workload-certs\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-s9dfr\" (UID: \"9e667364-fba5-4584-9251-773ae8e1f2e7\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-s9dfr"
Nov 02 03:15:04.823848 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:04.823811    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e667364-fba5-4584-9251-773ae8e1f2e7-istiod-ca-cert\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-s9dfr\" (UID: \"9e667364-fba5-4584-9251-773ae8e1f2e7\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-s9dfr"
Nov 02 03:15:04.825572 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:04.825548    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e667364-fba5-4584-9251-773ae8e1f2e7-istio-envoy\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-s9dfr\" (UID: \"9e667364-fba5-4584-9251-773ae8e1f2e7\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-s9dfr"
Nov 02 03:15:04.825826 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:04.825798    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e667364-fba5-4584-9251-773ae8e1f2e7-istio-podinfo\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-s9dfr\" (UID: \"9e667364-fba5-4584-9251-773ae8e1f2e7\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-s9dfr"
Nov 02 03:15:04.881075 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:04.880989    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e667364-fba5-4584-9251-773ae8e1f2e7-istio-token\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-s9dfr\" (UID: \"9e667364-fba5-4584-9251-773ae8e1f2e7\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-s9dfr"
Nov 02 03:15:04.881217 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:04.881138    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmh7s\" (UniqueName: \"kubernetes.io/projected/9e667364-fba5-4584-9251-773ae8e1f2e7-kube-api-access-cmh7s\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-s9dfr\" (UID: \"9e667364-fba5-4584-9251-773ae8e1f2e7\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-s9dfr"
Nov 02 03:15:04.991609 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:04.991570    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-s9dfr"
Nov 02 03:15:05.102670 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:05.102638    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-777d6b586f-brq7x"
Nov 02 03:15:05.251338 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:05.251303    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6c456c64b9-drg8b"]
Nov 02 03:15:05.339096 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:05.339059    4496 generic.go:356] "Generic (PLEG): container finished" podID="0e9a33a4-c8f9-4f47-9d62-dde798ecd364" containerID="368e0310317004fef64394ad11a95b1badc1852be87030a61f7ae61d27b26028" exitCode=60
Nov 02 03:15:05.339256 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:05.339150    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893155-7zcl8" event={"ID":"0e9a33a4-c8f9-4f47-9d62-dde798ecd364","Type":"ContainerDied","Data":"368e0310317004fef64394ad11a95b1badc1852be87030a61f7ae61d27b26028"}
Nov 02 03:15:05.339256 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:05.339224    4496 scope.go:122] "RemoveContainer" containerID="8039cebae499add1892f293cf6e3f83f2cee8c426adea78141b3138c7c15e5a4"
Nov 02 03:15:05.339412 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:05.339394    4496 scope.go:122] "RemoveContainer" containerID="368e0310317004fef64394ad11a95b1badc1852be87030a61f7ae61d27b26028"
Nov 02 03:15:05.339741 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:15:05.339709    4496 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893155-7zcl8_odh-ai-gateway-infra(0e9a33a4-c8f9-4f47-9d62-dde798ecd364)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893155-7zcl8" podUID="0e9a33a4-c8f9-4f47-9d62-dde798ecd364"
Nov 02 03:15:05.341497 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:05.341472    4496 generic.go:356] "Generic (PLEG): container finished" podID="a4216a4f-039c-4954-80d4-513838584e46" containerID="14513a37f261089447c96c906c537ec13782d785b6628cdad89c0bc36acfcdb7" exitCode=60
Nov 02 03:15:05.341596 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:05.341529    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ca6546-29893155-cgltk" event={"ID":"a4216a4f-039c-4954-80d4-513838584e46","Type":"ContainerDied","Data":"14513a37f261089447c96c906c537ec13782d785b6628cdad89c0bc36acfcdb7"}
Nov 02 03:15:05.341821 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:05.341804    4496 scope.go:122] "RemoveContainer" containerID="14513a37f261089447c96c906c537ec13782d785b6628cdad89c0bc36acfcdb7"
Nov 02 03:15:05.342118 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:15:05.342087    4496 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-ca6546-29893155-cgltk_odh-ai-gateway-infra(a4216a4f-039c-4954-80d4-513838584e46)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ca6546-29893155-cgltk" podUID="a4216a4f-039c-4954-80d4-513838584e46"
Nov 02 03:15:05.343710 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:05.343692    4496 generic.go:356] "Generic (PLEG): container finished" podID="9c457ead-4e87-4c5c-9082-4978277f7043" containerID="f4ece4633cb57518d9649fa945983a49b966ef5903a2ba5d710f501ddda99245" exitCode=60
Nov 02 03:15:05.343826 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:05.343768    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac93f-29893155-69hl6" event={"ID":"9c457ead-4e87-4c5c-9082-4978277f7043","Type":"ContainerDied","Data":"f4ece4633cb57518d9649fa945983a49b966ef5903a2ba5d710f501ddda99245"}
Nov 02 03:15:05.343985 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:05.343967    4496 scope.go:122] "RemoveContainer" containerID="f4ece4633cb57518d9649fa945983a49b966ef5903a2ba5d710f501ddda99245"
Nov 02 03:15:05.344253 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:15:05.344226    4496 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-5ac93f-29893155-69hl6_odh-ai-gateway-infra(9c457ead-4e87-4c5c-9082-4978277f7043)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac93f-29893155-69hl6" podUID="9c457ead-4e87-4c5c-9082-4978277f7043"
Nov 02 03:15:05.345935 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:05.345906    4496 generic.go:356] "Generic (PLEG): container finished" podID="f94649fd-ac69-42db-bdf8-01b2befa5ca9" containerID="b4a750354757d5e7fc71497df779fae2d66d1da6d1c79fbc4725541dd7fe3dc4" exitCode=60
Nov 02 03:15:05.346062 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:05.345967    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-858236-29893155-8f9z6" event={"ID":"f94649fd-ac69-42db-bdf8-01b2befa5ca9","Type":"ContainerDied","Data":"b4a750354757d5e7fc71497df779fae2d66d1da6d1c79fbc4725541dd7fe3dc4"}
Nov 02 03:15:05.346235 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:05.346221    4496 scope.go:122] "RemoveContainer" containerID="b4a750354757d5e7fc71497df779fae2d66d1da6d1c79fbc4725541dd7fe3dc4"
Nov 02 03:15:05.346436 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:15:05.346418    4496 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-858236-29893155-8f9z6_odh-ai-gateway-infra(f94649fd-ac69-42db-bdf8-01b2befa5ca9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-858236-29893155-8f9z6" podUID="f94649fd-ac69-42db-bdf8-01b2befa5ca9"
Nov 02 03:15:05.358047 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:05.358027    4496 scope.go:122] "RemoveContainer" containerID="479cbaad159ebc83a777f9575991cfc45c32a3720dab467c6cfdf1be8e2961ad"
Nov 02 03:15:05.374114 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:05.374090    4496 scope.go:122] "RemoveContainer" containerID="a256d92afca1c0d18f53b30bd1f0fa9b225dd91f33ee6f4c0313c88040e08720"
Nov 02 03:15:05.388094 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:05.388077    4496 scope.go:122] "RemoveContainer" containerID="dc4b641905f4c0acf8cffe69bf59bcbbf8c287c778079fac1066a5f70f07f9f2"
Nov 02 03:15:05.425348 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:15:05.425320    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e667364_fba5_4584_9251_773ae8e1f2e7.slice/crio-c78caab70c6503ba801cc1c8f4529294fdd326aff887a7cbcbc4139b38dd5f28 WatchSource:0}: Error finding container c78caab70c6503ba801cc1c8f4529294fdd326aff887a7cbcbc4139b38dd5f28: Status 404 returned error can't find the container with id c78caab70c6503ba801cc1c8f4529294fdd326aff887a7cbcbc4139b38dd5f28
Nov 02 03:15:05.426114 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:05.426090    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-s9dfr"]
Nov 02 03:15:05.427721 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:05.427693    4496 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 03:15:05.427782 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:05.427753    4496 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 03:15:05.427826 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:05.427784    4496 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 03:15:06.371412 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:06.371381    4496 scope.go:122] "RemoveContainer" containerID="368e0310317004fef64394ad11a95b1badc1852be87030a61f7ae61d27b26028"
Nov 02 03:15:06.371918 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:15:06.371655    4496 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893155-7zcl8_odh-ai-gateway-infra(0e9a33a4-c8f9-4f47-9d62-dde798ecd364)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893155-7zcl8" podUID="0e9a33a4-c8f9-4f47-9d62-dde798ecd364"
Nov 02 03:15:06.373202 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:06.373178    4496 scope.go:122] "RemoveContainer" containerID="14513a37f261089447c96c906c537ec13782d785b6628cdad89c0bc36acfcdb7"
Nov 02 03:15:06.373455 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:15:06.373433    4496 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-ca6546-29893155-cgltk_odh-ai-gateway-infra(a4216a4f-039c-4954-80d4-513838584e46)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ca6546-29893155-cgltk" podUID="a4216a4f-039c-4954-80d4-513838584e46"
Nov 02 03:15:06.374717 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:06.374689    4496 scope.go:122] "RemoveContainer" containerID="f4ece4633cb57518d9649fa945983a49b966ef5903a2ba5d710f501ddda99245"
Nov 02 03:15:06.374944 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:15:06.374919    4496 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-5ac93f-29893155-69hl6_odh-ai-gateway-infra(9c457ead-4e87-4c5c-9082-4978277f7043)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac93f-29893155-69hl6" podUID="9c457ead-4e87-4c5c-9082-4978277f7043"
Nov 02 03:15:06.376093 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:06.376066    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-s9dfr" event={"ID":"9e667364-fba5-4584-9251-773ae8e1f2e7","Type":"ContainerStarted","Data":"31e1aecc835a3ae846a1bb2868bd48d05de57994a7f4b3a5de746e497bc23141"}
Nov 02 03:15:06.376184 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:06.376135    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-s9dfr" event={"ID":"9e667364-fba5-4584-9251-773ae8e1f2e7","Type":"ContainerStarted","Data":"c78caab70c6503ba801cc1c8f4529294fdd326aff887a7cbcbc4139b38dd5f28"}
Nov 02 03:15:06.377839 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:06.377822    4496 scope.go:122] "RemoveContainer" containerID="b4a750354757d5e7fc71497df779fae2d66d1da6d1c79fbc4725541dd7fe3dc4"
Nov 02 03:15:06.378080 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:15:06.378032    4496 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-858236-29893155-8f9z6_odh-ai-gateway-infra(f94649fd-ac69-42db-bdf8-01b2befa5ca9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-858236-29893155-8f9z6" podUID="f94649fd-ac69-42db-bdf8-01b2befa5ca9"
Nov 02 03:15:06.726536 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:06.726487    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-s9dfr" podStartSLOduration=2.726471581 podStartE2EDuration="2.726471581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:15:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:15:06.723146351 +0000 UTC m=+1647.301753732" watchObservedRunningTime="2026-11-02 03:15:06.726471581 +0000 UTC m=+1647.305078961"
Nov 02 03:15:06.992793 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:06.992709    4496 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-s9dfr"
Nov 02 03:15:06.994293 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:06.994272    4496 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-s9dfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Nov 02 03:15:06.994359 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:06.994331    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-s9dfr" podUID="9e667364-fba5-4584-9251-773ae8e1f2e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Nov 02 03:15:07.992905 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:07.992873    4496 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-s9dfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Nov 02 03:15:07.993314 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:07.992935    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-s9dfr" podUID="9e667364-fba5-4584-9251-773ae8e1f2e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Nov 02 03:15:08.992214 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:08.992179    4496 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-s9dfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Nov 02 03:15:08.992383 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:08.992232    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-s9dfr" podUID="9e667364-fba5-4584-9251-773ae8e1f2e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Nov 02 03:15:09.992732 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:09.992699    4496 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-s9dfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Nov 02 03:15:09.993251 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:09.993223    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-s9dfr" podUID="9e667364-fba5-4584-9251-773ae8e1f2e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Nov 02 03:15:10.252203 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:10.252078    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-6c456c64b9-drg8b" podUID="07a0a43a-4232-467a-b7f4-ff4772b34e33" containerName="payload-processing" containerID="cri-o://f69e2511634fff187da3d8b04aae287a979d0a672f38986d9567eb3a7804bbea" gracePeriod=25
Nov 02 03:15:10.408802 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:10.408774    4496 generic.go:356] "Generic (PLEG): container finished" podID="07a0a43a-4232-467a-b7f4-ff4772b34e33" containerID="f69e2511634fff187da3d8b04aae287a979d0a672f38986d9567eb3a7804bbea" exitCode=0
Nov 02 03:15:10.409043 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:10.408848    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6c456c64b9-drg8b" event={"ID":"07a0a43a-4232-467a-b7f4-ff4772b34e33","Type":"ContainerDied","Data":"f69e2511634fff187da3d8b04aae287a979d0a672f38986d9567eb3a7804bbea"}
Nov 02 03:15:10.596534 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:10.596511    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6c456c64b9-drg8b"
Nov 02 03:15:10.710959 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:10.710927    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07a0a43a-4232-467a-b7f4-ff4772b34e33-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/07a0a43a-4232-467a-b7f4-ff4772b34e33-routing-overlay-volume\") pod \"07a0a43a-4232-467a-b7f4-ff4772b34e33\" (UID: \"07a0a43a-4232-467a-b7f4-ff4772b34e33\") "
Nov 02 03:15:10.711154 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:10.711049    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07a0a43a-4232-467a-b7f4-ff4772b34e33-kube-api-access-ldccv\" (UniqueName: \"kubernetes.io/projected/07a0a43a-4232-467a-b7f4-ff4772b34e33-kube-api-access-ldccv\") pod \"07a0a43a-4232-467a-b7f4-ff4772b34e33\" (UID: \"07a0a43a-4232-467a-b7f4-ff4772b34e33\") "
Nov 02 03:15:10.711154 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:10.711112    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07a0a43a-4232-467a-b7f4-ff4772b34e33-provider-credentials\" (UniqueName: \"kubernetes.io/projected/07a0a43a-4232-467a-b7f4-ff4772b34e33-provider-credentials\") pod \"07a0a43a-4232-467a-b7f4-ff4772b34e33\" (UID: \"07a0a43a-4232-467a-b7f4-ff4772b34e33\") "
Nov 02 03:15:10.711251 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:10.711169    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07a0a43a-4232-467a-b7f4-ff4772b34e33-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07a0a43a-4232-467a-b7f4-ff4772b34e33-plugins-config-volume\") pod \"07a0a43a-4232-467a-b7f4-ff4772b34e33\" (UID: \"07a0a43a-4232-467a-b7f4-ff4772b34e33\") "
Nov 02 03:15:10.711433 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:10.711404    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07a0a43a-4232-467a-b7f4-ff4772b34e33-routing-overlay-volume" pod "07a0a43a-4232-467a-b7f4-ff4772b34e33" (UID: "07a0a43a-4232-467a-b7f4-ff4772b34e33"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:15:10.711498 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:10.711476    4496 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/07a0a43a-4232-467a-b7f4-ff4772b34e33-routing-overlay-volume\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:15:10.712221 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:10.712190    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07a0a43a-4232-467a-b7f4-ff4772b34e33-plugins-config-volume" pod "07a0a43a-4232-467a-b7f4-ff4772b34e33" (UID: "07a0a43a-4232-467a-b7f4-ff4772b34e33"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:15:10.713581 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:10.713558    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07a0a43a-4232-467a-b7f4-ff4772b34e33-provider-credentials" pod "07a0a43a-4232-467a-b7f4-ff4772b34e33" (UID: "07a0a43a-4232-467a-b7f4-ff4772b34e33"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:15:10.713668 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:10.713649    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07a0a43a-4232-467a-b7f4-ff4772b34e33-kube-api-access-ldccv" pod "07a0a43a-4232-467a-b7f4-ff4772b34e33" (UID: "07a0a43a-4232-467a-b7f4-ff4772b34e33"). InnerVolumeSpecName "kube-api-access-ldccv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:15:10.812643 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:10.812603    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldccv\" (UniqueName: \"kubernetes.io/projected/07a0a43a-4232-467a-b7f4-ff4772b34e33-kube-api-access-ldccv\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:15:10.812643 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:10.812635    4496 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/07a0a43a-4232-467a-b7f4-ff4772b34e33-provider-credentials\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:15:10.812643 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:10.812646    4496 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07a0a43a-4232-467a-b7f4-ff4772b34e33-plugins-config-volume\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:15:10.992250 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:10.992213    4496 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-s9dfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Nov 02 03:15:10.992391 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:10.992289    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-s9dfr" podUID="9e667364-fba5-4584-9251-773ae8e1f2e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Nov 02 03:15:11.417864 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:11.417780    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6c456c64b9-drg8b"
Nov 02 03:15:11.417864 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:11.417815    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6c456c64b9-drg8b" event={"ID":"07a0a43a-4232-467a-b7f4-ff4772b34e33","Type":"ContainerDied","Data":"72adb85ccaea8e8b75014ec69f4ea280d27a37ffefd375b26d8a43519d18bb65"}
Nov 02 03:15:11.418373 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:11.417882    4496 scope.go:122] "RemoveContainer" containerID="f69e2511634fff187da3d8b04aae287a979d0a672f38986d9567eb3a7804bbea"
Nov 02 03:15:11.587765 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:11.587723    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6c456c64b9-drg8b"]
Nov 02 03:15:11.591063 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:11.591035    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-6c456c64b9-drg8b"]
Nov 02 03:15:11.992209 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:11.992172    4496 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-s9dfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Nov 02 03:15:11.992393 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:11.992236    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-s9dfr" podUID="9e667364-fba5-4584-9251-773ae8e1f2e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Nov 02 03:15:12.996419 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:12.996391    4496 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-s9dfr"
Nov 02 03:15:12.996808 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:12.996788    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-s9dfr"
Nov 02 03:15:12.997462 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:12.997436    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-s9dfr"
Nov 02 03:15:13.575091 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:13.575052    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07a0a43a-4232-467a-b7f4-ff4772b34e33" path="/var/lib/kubelet/pods/07a0a43a-4232-467a-b7f4-ff4772b34e33/volumes"
Nov 02 03:15:16.566108 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:16.566076    4496 scope.go:122] "RemoveContainer" containerID="b4a750354757d5e7fc71497df779fae2d66d1da6d1c79fbc4725541dd7fe3dc4"
Nov 02 03:15:16.566591 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:16.566152    4496 scope.go:122] "RemoveContainer" containerID="368e0310317004fef64394ad11a95b1badc1852be87030a61f7ae61d27b26028"
Nov 02 03:15:16.566591 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:16.566249    4496 scope.go:122] "RemoveContainer" containerID="14513a37f261089447c96c906c537ec13782d785b6628cdad89c0bc36acfcdb7"
Nov 02 03:15:17.468343 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:17.468290    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ca6546-29893155-cgltk" event={"ID":"a4216a4f-039c-4954-80d4-513838584e46","Type":"ContainerStarted","Data":"e686aca4a45cc190df6fd3ffce7e76a492af668c45b62068dce8d77f3926ef1c"}
Nov 02 03:15:17.470431 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:17.470406    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-858236-29893155-8f9z6" event={"ID":"f94649fd-ac69-42db-bdf8-01b2befa5ca9","Type":"ContainerStarted","Data":"e9773771922022ed89d2d7f7d4b78debad0d747a0c7750512e0504310c5b313c"}
Nov 02 03:15:17.472513 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:17.472488    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893155-7zcl8" event={"ID":"0e9a33a4-c8f9-4f47-9d62-dde798ecd364","Type":"ContainerStarted","Data":"4b22ad5ba0e28818ee5abe8aefa353d870a62388b32351493c3c4b0b0be97ac1"}
Nov 02 03:15:18.481567 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:18.481531    4496 generic.go:356] "Generic (PLEG): container finished" podID="a4216a4f-039c-4954-80d4-513838584e46" containerID="e686aca4a45cc190df6fd3ffce7e76a492af668c45b62068dce8d77f3926ef1c" exitCode=60
Nov 02 03:15:18.482066 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:18.481607    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ca6546-29893155-cgltk" event={"ID":"a4216a4f-039c-4954-80d4-513838584e46","Type":"ContainerDied","Data":"e686aca4a45cc190df6fd3ffce7e76a492af668c45b62068dce8d77f3926ef1c"}
Nov 02 03:15:18.482066 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:18.481665    4496 scope.go:122] "RemoveContainer" containerID="14513a37f261089447c96c906c537ec13782d785b6628cdad89c0bc36acfcdb7"
Nov 02 03:15:18.482066 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:18.482056    4496 scope.go:122] "RemoveContainer" containerID="e686aca4a45cc190df6fd3ffce7e76a492af668c45b62068dce8d77f3926ef1c"
Nov 02 03:15:18.482405 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:15:18.482375    4496 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-ca6546-29893155-cgltk_odh-ai-gateway-infra(a4216a4f-039c-4954-80d4-513838584e46)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ca6546-29893155-cgltk" podUID="a4216a4f-039c-4954-80d4-513838584e46"
Nov 02 03:15:18.483845 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:18.483825    4496 generic.go:356] "Generic (PLEG): container finished" podID="f94649fd-ac69-42db-bdf8-01b2befa5ca9" containerID="e9773771922022ed89d2d7f7d4b78debad0d747a0c7750512e0504310c5b313c" exitCode=60
Nov 02 03:15:18.483933 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:18.483899    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-858236-29893155-8f9z6" event={"ID":"f94649fd-ac69-42db-bdf8-01b2befa5ca9","Type":"ContainerDied","Data":"e9773771922022ed89d2d7f7d4b78debad0d747a0c7750512e0504310c5b313c"}
Nov 02 03:15:18.484298 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:18.484279    4496 scope.go:122] "RemoveContainer" containerID="e9773771922022ed89d2d7f7d4b78debad0d747a0c7750512e0504310c5b313c"
Nov 02 03:15:18.484643 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:15:18.484619    4496 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-858236-29893155-8f9z6_odh-ai-gateway-infra(f94649fd-ac69-42db-bdf8-01b2befa5ca9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-858236-29893155-8f9z6" podUID="f94649fd-ac69-42db-bdf8-01b2befa5ca9"
Nov 02 03:15:18.485922 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:18.485904    4496 generic.go:356] "Generic (PLEG): container finished" podID="0e9a33a4-c8f9-4f47-9d62-dde798ecd364" containerID="4b22ad5ba0e28818ee5abe8aefa353d870a62388b32351493c3c4b0b0be97ac1" exitCode=60
Nov 02 03:15:18.486036 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:18.485970    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893155-7zcl8" event={"ID":"0e9a33a4-c8f9-4f47-9d62-dde798ecd364","Type":"ContainerDied","Data":"4b22ad5ba0e28818ee5abe8aefa353d870a62388b32351493c3c4b0b0be97ac1"}
Nov 02 03:15:18.486293 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:18.486277    4496 scope.go:122] "RemoveContainer" containerID="4b22ad5ba0e28818ee5abe8aefa353d870a62388b32351493c3c4b0b0be97ac1"
Nov 02 03:15:18.486524 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:15:18.486508    4496 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893155-7zcl8_odh-ai-gateway-infra(0e9a33a4-c8f9-4f47-9d62-dde798ecd364)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893155-7zcl8" podUID="0e9a33a4-c8f9-4f47-9d62-dde798ecd364"
Nov 02 03:15:18.494673 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:18.494655    4496 scope.go:122] "RemoveContainer" containerID="b4a750354757d5e7fc71497df779fae2d66d1da6d1c79fbc4725541dd7fe3dc4"
Nov 02 03:15:18.508685 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:18.508666    4496 scope.go:122] "RemoveContainer" containerID="368e0310317004fef64394ad11a95b1badc1852be87030a61f7ae61d27b26028"
Nov 02 03:15:18.566291 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:18.566273    4496 scope.go:122] "RemoveContainer" containerID="f4ece4633cb57518d9649fa945983a49b966ef5903a2ba5d710f501ddda99245"
Nov 02 03:15:18.597424 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:18.597388    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ca6546-29893155-cgltk"]
Nov 02 03:15:18.708487 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:18.708456    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893155-7zcl8"]
Nov 02 03:15:18.823845 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:18.823780    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-858236-29893155-8f9z6"]
Nov 02 03:15:19.507040 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:19.502575    4496 generic.go:356] "Generic (PLEG): container finished" podID="9c457ead-4e87-4c5c-9082-4978277f7043" containerID="76ddb55ad6654107576d535444c1781f887da23bad7658573ab797fab4efab36" exitCode=60
Nov 02 03:15:19.507040 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:19.502695    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac93f-29893155-69hl6" event={"ID":"9c457ead-4e87-4c5c-9082-4978277f7043","Type":"ContainerDied","Data":"76ddb55ad6654107576d535444c1781f887da23bad7658573ab797fab4efab36"}
Nov 02 03:15:19.507040 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:19.502757    4496 scope.go:122] "RemoveContainer" containerID="f4ece4633cb57518d9649fa945983a49b966ef5903a2ba5d710f501ddda99245"
Nov 02 03:15:19.507040 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:19.503269    4496 scope.go:122] "RemoveContainer" containerID="76ddb55ad6654107576d535444c1781f887da23bad7658573ab797fab4efab36"
Nov 02 03:15:19.507040 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:15:19.503605    4496 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-5ac93f-29893155-69hl6_odh-ai-gateway-infra(9c457ead-4e87-4c5c-9082-4978277f7043)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac93f-29893155-69hl6" podUID="9c457ead-4e87-4c5c-9082-4978277f7043"
Nov 02 03:15:20.273041 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:20.269320    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893155-7zcl8"
Nov 02 03:15:20.342107 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:20.340160    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-858236-29893155-8f9z6"
Nov 02 03:15:20.355619 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:20.355198    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ca6546-29893155-cgltk"
Nov 02 03:15:20.373035 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:20.370451    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f94649fd-ac69-42db-bdf8-01b2befa5ca9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f94649fd-ac69-42db-bdf8-01b2befa5ca9-maas-api-service-ca\") pod \"f94649fd-ac69-42db-bdf8-01b2befa5ca9\" (UID: \"f94649fd-ac69-42db-bdf8-01b2befa5ca9\") "
Nov 02 03:15:20.373035 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:20.370522    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e9a33a4-c8f9-4f47-9d62-dde798ecd364-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e9a33a4-c8f9-4f47-9d62-dde798ecd364-maas-api-service-ca\") pod \"0e9a33a4-c8f9-4f47-9d62-dde798ecd364\" (UID: \"0e9a33a4-c8f9-4f47-9d62-dde798ecd364\") "
Nov 02 03:15:20.373035 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:20.370624    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e9a33a4-c8f9-4f47-9d62-dde798ecd364-kube-api-access-vb7rj\" (UniqueName: \"kubernetes.io/projected/0e9a33a4-c8f9-4f47-9d62-dde798ecd364-kube-api-access-vb7rj\") pod \"0e9a33a4-c8f9-4f47-9d62-dde798ecd364\" (UID: \"0e9a33a4-c8f9-4f47-9d62-dde798ecd364\") "
Nov 02 03:15:20.373035 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:20.370714    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f94649fd-ac69-42db-bdf8-01b2befa5ca9-kube-api-access-t25wl\" (UniqueName: \"kubernetes.io/projected/f94649fd-ac69-42db-bdf8-01b2befa5ca9-kube-api-access-t25wl\") pod \"f94649fd-ac69-42db-bdf8-01b2befa5ca9\" (UID: \"f94649fd-ac69-42db-bdf8-01b2befa5ca9\") "
Nov 02 03:15:20.383079 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:20.375489    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f94649fd-ac69-42db-bdf8-01b2befa5ca9-maas-api-service-ca" pod "f94649fd-ac69-42db-bdf8-01b2befa5ca9" (UID: "f94649fd-ac69-42db-bdf8-01b2befa5ca9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:15:20.394035 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:20.392215    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e9a33a4-c8f9-4f47-9d62-dde798ecd364-maas-api-service-ca" pod "0e9a33a4-c8f9-4f47-9d62-dde798ecd364" (UID: "0e9a33a4-c8f9-4f47-9d62-dde798ecd364"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:15:20.410557 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:20.410499    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e9a33a4-c8f9-4f47-9d62-dde798ecd364-kube-api-access-vb7rj" pod "0e9a33a4-c8f9-4f47-9d62-dde798ecd364" (UID: "0e9a33a4-c8f9-4f47-9d62-dde798ecd364"). InnerVolumeSpecName "kube-api-access-vb7rj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:15:20.417201 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:20.417171    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f94649fd-ac69-42db-bdf8-01b2befa5ca9-kube-api-access-t25wl" pod "f94649fd-ac69-42db-bdf8-01b2befa5ca9" (UID: "f94649fd-ac69-42db-bdf8-01b2befa5ca9"). InnerVolumeSpecName "kube-api-access-t25wl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:15:20.513038 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:20.512386    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4216a4f-039c-4954-80d4-513838584e46-kube-api-access-p4t7g\" (UniqueName: \"kubernetes.io/projected/a4216a4f-039c-4954-80d4-513838584e46-kube-api-access-p4t7g\") pod \"a4216a4f-039c-4954-80d4-513838584e46\" (UID: \"a4216a4f-039c-4954-80d4-513838584e46\") "
Nov 02 03:15:20.513038 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:20.512486    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4216a4f-039c-4954-80d4-513838584e46-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4216a4f-039c-4954-80d4-513838584e46-maas-api-service-ca\") pod \"a4216a4f-039c-4954-80d4-513838584e46\" (UID: \"a4216a4f-039c-4954-80d4-513838584e46\") "
Nov 02 03:15:20.513599 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:20.513048    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f94649fd-ac69-42db-bdf8-01b2befa5ca9-maas-api-service-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:15:20.513599 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:20.513074    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e9a33a4-c8f9-4f47-9d62-dde798ecd364-maas-api-service-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:15:20.513599 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:20.513091    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vb7rj\" (UniqueName: \"kubernetes.io/projected/0e9a33a4-c8f9-4f47-9d62-dde798ecd364-kube-api-access-vb7rj\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:15:20.513599 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:20.513106    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t25wl\" (UniqueName: \"kubernetes.io/projected/f94649fd-ac69-42db-bdf8-01b2befa5ca9-kube-api-access-t25wl\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:15:20.519535 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:20.519492    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4216a4f-039c-4954-80d4-513838584e46-maas-api-service-ca" pod "a4216a4f-039c-4954-80d4-513838584e46" (UID: "a4216a4f-039c-4954-80d4-513838584e46"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:15:20.525061 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:20.521912    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4216a4f-039c-4954-80d4-513838584e46-kube-api-access-p4t7g" pod "a4216a4f-039c-4954-80d4-513838584e46" (UID: "a4216a4f-039c-4954-80d4-513838584e46"). InnerVolumeSpecName "kube-api-access-p4t7g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:15:20.596098 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:20.583693    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-858236-29893155-8f9z6" event={"ID":"f94649fd-ac69-42db-bdf8-01b2befa5ca9","Type":"ContainerDied","Data":"b897600349d280db8f8c3be25c31d0f2a1d04221d8cbaa927e523d3c956ad5b9"}
Nov 02 03:15:20.596098 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:20.583797    4496 scope.go:122] "RemoveContainer" containerID="e9773771922022ed89d2d7f7d4b78debad0d747a0c7750512e0504310c5b313c"
Nov 02 03:15:20.596098 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:20.583915    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-858236-29893155-8f9z6"
Nov 02 03:15:20.610663 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:20.610546    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893155-7zcl8" event={"ID":"0e9a33a4-c8f9-4f47-9d62-dde798ecd364","Type":"ContainerDied","Data":"67ce4ae9e66b42424cc5728fa3b3b651c70d21fc60fd0d49051c036dbdedbf15"}
Nov 02 03:15:20.610896 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:20.610864    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893155-7zcl8"
Nov 02 03:15:20.623158 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:20.623135    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4t7g\" (UniqueName: \"kubernetes.io/projected/a4216a4f-039c-4954-80d4-513838584e46-kube-api-access-p4t7g\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:15:20.623301 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:20.623287    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4216a4f-039c-4954-80d4-513838584e46-maas-api-service-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:15:20.628692 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:20.628668    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ca6546-29893155-cgltk" event={"ID":"a4216a4f-039c-4954-80d4-513838584e46","Type":"ContainerDied","Data":"e8ba1a7bb580778995b3cba7d15d92ebac36697ee7f9d50d7fdf731b171debbb"}
Nov 02 03:15:20.628909 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:20.628851    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ca6546-29893155-cgltk"
Nov 02 03:15:20.659510 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:20.633705    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac93f-29893155-69hl6"]
Nov 02 03:15:20.725054 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:20.722559    4496 scope.go:122] "RemoveContainer" containerID="4b22ad5ba0e28818ee5abe8aefa353d870a62388b32351493c3c4b0b0be97ac1"
Nov 02 03:15:20.788055 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:20.773185    4496 scope.go:122] "RemoveContainer" containerID="e686aca4a45cc190df6fd3ffce7e76a492af668c45b62068dce8d77f3926ef1c"
Nov 02 03:15:21.023336 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:21.023293    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893155-7zcl8"]
Nov 02 03:15:21.031045 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:21.028768    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893155-7zcl8"]
Nov 02 03:15:21.147039 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:21.143780    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ca6546-29893155-cgltk"]
Nov 02 03:15:21.171087 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:21.168027    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ca6546-29893155-cgltk"]
Nov 02 03:15:21.320683 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:21.320559    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-858236-29893155-8f9z6"]
Nov 02 03:15:21.323469 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:21.323390    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac93f-29893155-69hl6"
Nov 02 03:15:21.328583 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:21.328562    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-858236-29893155-8f9z6"]
Nov 02 03:15:21.426514 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:21.425278    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c457ead-4e87-4c5c-9082-4978277f7043-kube-api-access-k7h5j\" (UniqueName: \"kubernetes.io/projected/9c457ead-4e87-4c5c-9082-4978277f7043-kube-api-access-k7h5j\") pod \"9c457ead-4e87-4c5c-9082-4978277f7043\" (UID: \"9c457ead-4e87-4c5c-9082-4978277f7043\") "
Nov 02 03:15:21.426514 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:21.425506    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c457ead-4e87-4c5c-9082-4978277f7043-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c457ead-4e87-4c5c-9082-4978277f7043-maas-api-service-ca\") pod \"9c457ead-4e87-4c5c-9082-4978277f7043\" (UID: \"9c457ead-4e87-4c5c-9082-4978277f7043\") "
Nov 02 03:15:21.428604 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:21.428490    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c457ead-4e87-4c5c-9082-4978277f7043-maas-api-service-ca" pod "9c457ead-4e87-4c5c-9082-4978277f7043" (UID: "9c457ead-4e87-4c5c-9082-4978277f7043"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:15:21.466434 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:21.454222    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c457ead-4e87-4c5c-9082-4978277f7043-kube-api-access-k7h5j" pod "9c457ead-4e87-4c5c-9082-4978277f7043" (UID: "9c457ead-4e87-4c5c-9082-4978277f7043"). InnerVolumeSpecName "kube-api-access-k7h5j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:15:21.535039 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:21.528990    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7h5j\" (UniqueName: \"kubernetes.io/projected/9c457ead-4e87-4c5c-9082-4978277f7043-kube-api-access-k7h5j\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:15:21.535039 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:21.529047    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c457ead-4e87-4c5c-9082-4978277f7043-maas-api-service-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:15:21.589797 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:21.589682    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e9a33a4-c8f9-4f47-9d62-dde798ecd364" path="/var/lib/kubelet/pods/0e9a33a4-c8f9-4f47-9d62-dde798ecd364/volumes"
Nov 02 03:15:21.590376 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:21.590351    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4216a4f-039c-4954-80d4-513838584e46" path="/var/lib/kubelet/pods/a4216a4f-039c-4954-80d4-513838584e46/volumes"
Nov 02 03:15:21.590927 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:21.590907    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f94649fd-ac69-42db-bdf8-01b2befa5ca9" path="/var/lib/kubelet/pods/f94649fd-ac69-42db-bdf8-01b2befa5ca9/volumes"
Nov 02 03:15:21.638926 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:21.638889    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac93f-29893155-69hl6"
Nov 02 03:15:21.638926 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:21.638900    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac93f-29893155-69hl6" event={"ID":"9c457ead-4e87-4c5c-9082-4978277f7043","Type":"ContainerDied","Data":"8a995bf7a7be679362775d1e65865d95b533bf5f7c9a830efbc250abf1dea27a"}
Nov 02 03:15:21.639135 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:21.638968    4496 scope.go:122] "RemoveContainer" containerID="76ddb55ad6654107576d535444c1781f887da23bad7658573ab797fab4efab36"
Nov 02 03:15:21.806290 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:21.806258    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac93f-29893155-69hl6"]
Nov 02 03:15:21.809393 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:21.809371    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac93f-29893155-69hl6"]
Nov 02 03:15:23.574099 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:23.573981    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c457ead-4e87-4c5c-9082-4978277f7043" path="/var/lib/kubelet/pods/9c457ead-4e87-4c5c-9082-4978277f7043/volumes"
Nov 02 03:15:30.974832 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:30.974792    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-qtwvf"]
Nov 02 03:15:30.975574 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:30.975552    4496 cpu_manager.go:414] "containerMap: removing container" podUID="07a0a43a-4232-467a-b7f4-ff4772b34e33" containerName="payload-processing"
Nov 02 03:15:30.975670 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:30.975576    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07a0a43a-4232-467a-b7f4-ff4772b34e33" containerName="payload-processing"
Nov 02 03:15:30.975670 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:30.975598    4496 cpu_manager.go:414] "containerMap: removing container" podUID="9c457ead-4e87-4c5c-9082-4978277f7043" containerName="cleanup"
Nov 02 03:15:30.975670 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:30.975605    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c457ead-4e87-4c5c-9082-4978277f7043" containerName="cleanup"
Nov 02 03:15:30.975670 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:30.975615    4496 cpu_manager.go:414] "containerMap: removing container" podUID="f94649fd-ac69-42db-bdf8-01b2befa5ca9" containerName="cleanup"
Nov 02 03:15:30.975670 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:30.975623    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f94649fd-ac69-42db-bdf8-01b2befa5ca9" containerName="cleanup"
Nov 02 03:15:30.975670 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:30.975633    4496 cpu_manager.go:414] "containerMap: removing container" podUID="a4216a4f-039c-4954-80d4-513838584e46" containerName="cleanup"
Nov 02 03:15:30.975670 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:30.975641    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4216a4f-039c-4954-80d4-513838584e46" containerName="cleanup"
Nov 02 03:15:30.975670 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:30.975658    4496 cpu_manager.go:414] "containerMap: removing container" podUID="a4216a4f-039c-4954-80d4-513838584e46" containerName="cleanup"
Nov 02 03:15:30.975670 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:30.975665    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4216a4f-039c-4954-80d4-513838584e46" containerName="cleanup"
Nov 02 03:15:30.976134 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:30.975683    4496 cpu_manager.go:414] "containerMap: removing container" podUID="0e9a33a4-c8f9-4f47-9d62-dde798ecd364" containerName="cleanup"
Nov 02 03:15:30.976134 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:30.975692    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e9a33a4-c8f9-4f47-9d62-dde798ecd364" containerName="cleanup"
Nov 02 03:15:30.976134 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:30.975713    4496 cpu_manager.go:414] "containerMap: removing container" podUID="0e9a33a4-c8f9-4f47-9d62-dde798ecd364" containerName="cleanup"
Nov 02 03:15:30.976134 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:30.975721    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e9a33a4-c8f9-4f47-9d62-dde798ecd364" containerName="cleanup"
Nov 02 03:15:30.976134 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:30.975732    4496 cpu_manager.go:414] "containerMap: removing container" podUID="9c457ead-4e87-4c5c-9082-4978277f7043" containerName="cleanup"
Nov 02 03:15:30.976134 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:30.975739    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c457ead-4e87-4c5c-9082-4978277f7043" containerName="cleanup"
Nov 02 03:15:30.976134 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:30.975749    4496 cpu_manager.go:414] "containerMap: removing container" podUID="f94649fd-ac69-42db-bdf8-01b2befa5ca9" containerName="cleanup"
Nov 02 03:15:30.976134 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:30.975756    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f94649fd-ac69-42db-bdf8-01b2befa5ca9" containerName="cleanup"
Nov 02 03:15:30.976134 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:30.975767    4496 cpu_manager.go:414] "containerMap: removing container" podUID="0e9a33a4-c8f9-4f47-9d62-dde798ecd364" containerName="cleanup"
Nov 02 03:15:30.976134 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:30.975774    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e9a33a4-c8f9-4f47-9d62-dde798ecd364" containerName="cleanup"
Nov 02 03:15:30.976134 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:30.975933    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="07a0a43a-4232-467a-b7f4-ff4772b34e33" containerName="payload-processing"
Nov 02 03:15:30.976134 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:30.975948    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c457ead-4e87-4c5c-9082-4978277f7043" containerName="cleanup"
Nov 02 03:15:30.976134 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:30.975958    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="f94649fd-ac69-42db-bdf8-01b2befa5ca9" containerName="cleanup"
Nov 02 03:15:30.976134 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:30.975968    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4216a4f-039c-4954-80d4-513838584e46" containerName="cleanup"
Nov 02 03:15:30.976134 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:30.975979    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4216a4f-039c-4954-80d4-513838584e46" containerName="cleanup"
Nov 02 03:15:30.976134 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:30.975988    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e9a33a4-c8f9-4f47-9d62-dde798ecd364" containerName="cleanup"
Nov 02 03:15:30.976134 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:30.975999    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e9a33a4-c8f9-4f47-9d62-dde798ecd364" containerName="cleanup"
Nov 02 03:15:30.976134 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:30.976008    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="f94649fd-ac69-42db-bdf8-01b2befa5ca9" containerName="cleanup"
Nov 02 03:15:30.976134 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:30.976034    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="f94649fd-ac69-42db-bdf8-01b2befa5ca9" containerName="cleanup"
Nov 02 03:15:30.976134 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:30.976043    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e9a33a4-c8f9-4f47-9d62-dde798ecd364" containerName="cleanup"
Nov 02 03:15:30.976134 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:30.976058    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c457ead-4e87-4c5c-9082-4978277f7043" containerName="cleanup"
Nov 02 03:15:30.981333 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:30.981312    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-qtwvf"
Nov 02 03:15:31.151138 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:31.151096    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4975cf07-dbfe-4cd1-b857-37773a64e47c-istio-data\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-qtwvf\" (UID: \"4975cf07-dbfe-4cd1-b857-37773a64e47c\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-qtwvf"
Nov 02 03:15:31.151138 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:31.151136    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4975cf07-dbfe-4cd1-b857-37773a64e47c-istio-podinfo\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-qtwvf\" (UID: \"4975cf07-dbfe-4cd1-b857-37773a64e47c\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-qtwvf"
Nov 02 03:15:31.151363 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:31.151282    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4975cf07-dbfe-4cd1-b857-37773a64e47c-istio-envoy\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-qtwvf\" (UID: \"4975cf07-dbfe-4cd1-b857-37773a64e47c\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-qtwvf"
Nov 02 03:15:31.151363 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:31.151327    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4975cf07-dbfe-4cd1-b857-37773a64e47c-credential-socket\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-qtwvf\" (UID: \"4975cf07-dbfe-4cd1-b857-37773a64e47c\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-qtwvf"
Nov 02 03:15:31.151490 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:31.151385    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4975cf07-dbfe-4cd1-b857-37773a64e47c-workload-certs\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-qtwvf\" (UID: \"4975cf07-dbfe-4cd1-b857-37773a64e47c\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-qtwvf"
Nov 02 03:15:31.151490 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:31.151418    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4975cf07-dbfe-4cd1-b857-37773a64e47c-workload-socket\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-qtwvf\" (UID: \"4975cf07-dbfe-4cd1-b857-37773a64e47c\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-qtwvf"
Nov 02 03:15:31.151586 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:31.151497    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4975cf07-dbfe-4cd1-b857-37773a64e47c-istio-token\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-qtwvf\" (UID: \"4975cf07-dbfe-4cd1-b857-37773a64e47c\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-qtwvf"
Nov 02 03:15:31.151586 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:31.151529    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4975cf07-dbfe-4cd1-b857-37773a64e47c-istiod-ca-cert\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-qtwvf\" (UID: \"4975cf07-dbfe-4cd1-b857-37773a64e47c\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-qtwvf"
Nov 02 03:15:31.151678 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:31.151603    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vk5hh\" (UniqueName: \"kubernetes.io/projected/4975cf07-dbfe-4cd1-b857-37773a64e47c-kube-api-access-vk5hh\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-qtwvf\" (UID: \"4975cf07-dbfe-4cd1-b857-37773a64e47c\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-qtwvf"
Nov 02 03:15:31.185629 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:31.185587    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-qtwvf"]
Nov 02 03:15:31.252687 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:31.252641    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4975cf07-dbfe-4cd1-b857-37773a64e47c-istio-envoy\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-qtwvf\" (UID: \"4975cf07-dbfe-4cd1-b857-37773a64e47c\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-qtwvf"
Nov 02 03:15:31.252852 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:31.252710    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4975cf07-dbfe-4cd1-b857-37773a64e47c-credential-socket\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-qtwvf\" (UID: \"4975cf07-dbfe-4cd1-b857-37773a64e47c\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-qtwvf"
Nov 02 03:15:31.252852 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:31.252748    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4975cf07-dbfe-4cd1-b857-37773a64e47c-workload-certs\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-qtwvf\" (UID: \"4975cf07-dbfe-4cd1-b857-37773a64e47c\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-qtwvf"
Nov 02 03:15:31.252852 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:31.252798    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4975cf07-dbfe-4cd1-b857-37773a64e47c-workload-socket\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-qtwvf\" (UID: \"4975cf07-dbfe-4cd1-b857-37773a64e47c\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-qtwvf"
Nov 02 03:15:31.252852 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:31.252846    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4975cf07-dbfe-4cd1-b857-37773a64e47c-istio-token\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-qtwvf\" (UID: \"4975cf07-dbfe-4cd1-b857-37773a64e47c\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-qtwvf"
Nov 02 03:15:31.253068 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:31.252879    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4975cf07-dbfe-4cd1-b857-37773a64e47c-istiod-ca-cert\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-qtwvf\" (UID: \"4975cf07-dbfe-4cd1-b857-37773a64e47c\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-qtwvf"
Nov 02 03:15:31.253068 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:31.252948    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vk5hh\" (UniqueName: \"kubernetes.io/projected/4975cf07-dbfe-4cd1-b857-37773a64e47c-kube-api-access-vk5hh\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-qtwvf\" (UID: \"4975cf07-dbfe-4cd1-b857-37773a64e47c\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-qtwvf"
Nov 02 03:15:31.253068 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:31.253045    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4975cf07-dbfe-4cd1-b857-37773a64e47c-istio-data\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-qtwvf\" (UID: \"4975cf07-dbfe-4cd1-b857-37773a64e47c\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-qtwvf"
Nov 02 03:15:31.253237 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:31.253071    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4975cf07-dbfe-4cd1-b857-37773a64e47c-istio-podinfo\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-qtwvf\" (UID: \"4975cf07-dbfe-4cd1-b857-37773a64e47c\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-qtwvf"
Nov 02 03:15:31.253237 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:31.253158    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4975cf07-dbfe-4cd1-b857-37773a64e47c-credential-socket\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-qtwvf\" (UID: \"4975cf07-dbfe-4cd1-b857-37773a64e47c\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-qtwvf"
Nov 02 03:15:31.253237 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:31.253176    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4975cf07-dbfe-4cd1-b857-37773a64e47c-workload-certs\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-qtwvf\" (UID: \"4975cf07-dbfe-4cd1-b857-37773a64e47c\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-qtwvf"
Nov 02 03:15:31.253397 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:31.253229    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4975cf07-dbfe-4cd1-b857-37773a64e47c-workload-socket\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-qtwvf\" (UID: \"4975cf07-dbfe-4cd1-b857-37773a64e47c\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-qtwvf"
Nov 02 03:15:31.253547 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:31.253520    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4975cf07-dbfe-4cd1-b857-37773a64e47c-istio-data\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-qtwvf\" (UID: \"4975cf07-dbfe-4cd1-b857-37773a64e47c\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-qtwvf"
Nov 02 03:15:31.253779 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:31.253756    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4975cf07-dbfe-4cd1-b857-37773a64e47c-istiod-ca-cert\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-qtwvf\" (UID: \"4975cf07-dbfe-4cd1-b857-37773a64e47c\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-qtwvf"
Nov 02 03:15:31.255171 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:31.255145    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4975cf07-dbfe-4cd1-b857-37773a64e47c-istio-envoy\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-qtwvf\" (UID: \"4975cf07-dbfe-4cd1-b857-37773a64e47c\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-qtwvf"
Nov 02 03:15:31.255672 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:31.255650    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4975cf07-dbfe-4cd1-b857-37773a64e47c-istio-podinfo\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-qtwvf\" (UID: \"4975cf07-dbfe-4cd1-b857-37773a64e47c\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-qtwvf"
Nov 02 03:15:31.310731 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:31.310696    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4975cf07-dbfe-4cd1-b857-37773a64e47c-istio-token\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-qtwvf\" (UID: \"4975cf07-dbfe-4cd1-b857-37773a64e47c\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-qtwvf"
Nov 02 03:15:31.310873 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:31.310797    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vk5hh\" (UniqueName: \"kubernetes.io/projected/4975cf07-dbfe-4cd1-b857-37773a64e47c-kube-api-access-vk5hh\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-qtwvf\" (UID: \"4975cf07-dbfe-4cd1-b857-37773a64e47c\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-qtwvf"
Nov 02 03:15:31.595352 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:31.595259    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-qtwvf"
Nov 02 03:15:32.047033 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:32.044708    4496 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 03:15:32.047033 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:32.044795    4496 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 03:15:32.047033 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:32.044835    4496 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 03:15:32.056639 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:32.053287    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-qtwvf"]
Nov 02 03:15:32.748063 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:32.747901    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-qtwvf" event={"ID":"4975cf07-dbfe-4cd1-b857-37773a64e47c","Type":"ContainerStarted","Data":"a0db0477225506bce774eec4f294e517234bb9c687d857d6eef8d902f5831c18"}
Nov 02 03:15:32.748063 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:32.747969    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-qtwvf" event={"ID":"4975cf07-dbfe-4cd1-b857-37773a64e47c","Type":"ContainerStarted","Data":"1f65c19dba51cfd3e9beb31c7f5053d57695508a4e7552e7c8c2afd937947b8b"}
Nov 02 03:15:32.917152 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:32.917097    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-qtwvf" podStartSLOduration=2.917083146 podStartE2EDuration="2.917083146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:15:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:15:32.865691341 +0000 UTC m=+1673.444298721" watchObservedRunningTime="2026-11-02 03:15:32.917083146 +0000 UTC m=+1673.495690525"
Nov 02 03:15:33.525923 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:15:33.525884    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 03:15:33.596192 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:33.596162    4496 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-qtwvf"
Nov 02 03:15:33.597976 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:33.597946    4496 patch_prober.go:29] interesting pod/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-qtwvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body=
Nov 02 03:15:33.598120 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:33.597993    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-qtwvf" podUID="4975cf07-dbfe-4cd1-b857-37773a64e47c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused"
Nov 02 03:15:34.596118 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:34.596078    4496 patch_prober.go:29] interesting pod/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-qtwvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body=
Nov 02 03:15:34.596598 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:34.596135    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-qtwvf" podUID="4975cf07-dbfe-4cd1-b857-37773a64e47c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused"
Nov 02 03:15:35.596610 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:35.596571    4496 patch_prober.go:29] interesting pod/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-qtwvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body=
Nov 02 03:15:35.597041 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:35.596619    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-qtwvf" podUID="4975cf07-dbfe-4cd1-b857-37773a64e47c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused"
Nov 02 03:15:36.596340 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:36.596304    4496 patch_prober.go:29] interesting pod/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-qtwvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body=
Nov 02 03:15:36.596518 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:36.596351    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-qtwvf" podUID="4975cf07-dbfe-4cd1-b857-37773a64e47c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused"
Nov 02 03:15:37.596144 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:37.596110    4496 patch_prober.go:29] interesting pod/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-qtwvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body=
Nov 02 03:15:37.596527 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:37.596175    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-qtwvf" podUID="4975cf07-dbfe-4cd1-b857-37773a64e47c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused"
Nov 02 03:15:38.595721 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:38.595687    4496 patch_prober.go:29] interesting pod/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-qtwvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body=
Nov 02 03:15:38.595957 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:38.595735    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-qtwvf" podUID="4975cf07-dbfe-4cd1-b857-37773a64e47c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused"
Nov 02 03:15:39.601124 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:39.601093    4496 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-qtwvf"
Nov 02 03:15:39.601540 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:39.601471    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-qtwvf"
Nov 02 03:15:39.602295 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:39.602271    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-qtwvf"
Nov 02 03:15:39.984654 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:39.984622    4496 scope.go:122] "RemoveContainer" containerID="bc066aa7a3a37cc3379cb4487d3db5ae6b32747d7f20041a3216da434dae43eb"
Nov 02 03:15:39.994618 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:39.994595    4496 scope.go:122] "RemoveContainer" containerID="160ed1f6c3554e1ee4f2718e30785c7beb547a10a313e472f63fca38b33563ed"
Nov 02 03:15:40.008433 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:40.008409    4496 scope.go:122] "RemoveContainer" containerID="9f36b9bb5119eb47bb21369a13ab71640e278abb370d0c5366c43d324a931ee0"
Nov 02 03:15:40.017463 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:40.017441    4496 scope.go:122] "RemoveContainer" containerID="bee095c3a6f485380270328d114cd455f861554ecfe5f44670887fc7ae54bb62"
Nov 02 03:15:40.026355 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:40.026329    4496 scope.go:122] "RemoveContainer" containerID="93916a427110f0f0a5a2db3bd7b792560c2ea4a786e9cf96f4236115e12baf77"
Nov 02 03:15:48.841984 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:48.841947    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5130b62f-5lzt2"]
Nov 02 03:15:48.842838 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:48.842815    4496 cpu_manager.go:414] "containerMap: removing container" podUID="f94649fd-ac69-42db-bdf8-01b2befa5ca9" containerName="cleanup"
Nov 02 03:15:48.842881 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:48.842841    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f94649fd-ac69-42db-bdf8-01b2befa5ca9" containerName="cleanup"
Nov 02 03:15:48.842881 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:48.842862    4496 cpu_manager.go:414] "containerMap: removing container" podUID="a4216a4f-039c-4954-80d4-513838584e46" containerName="cleanup"
Nov 02 03:15:48.842881 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:48.842871    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4216a4f-039c-4954-80d4-513838584e46" containerName="cleanup"
Nov 02 03:15:48.842972 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:48.842902    4496 cpu_manager.go:414] "containerMap: removing container" podUID="9c457ead-4e87-4c5c-9082-4978277f7043" containerName="cleanup"
Nov 02 03:15:48.842972 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:48.842912    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c457ead-4e87-4c5c-9082-4978277f7043" containerName="cleanup"
Nov 02 03:15:48.843118 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:48.843092    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c457ead-4e87-4c5c-9082-4978277f7043" containerName="cleanup"
Nov 02 03:15:48.843163 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:48.843128    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4216a4f-039c-4954-80d4-513838584e46" containerName="cleanup"
Nov 02 03:15:48.849279 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:48.849255    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5130b62f-5lzt2"
Nov 02 03:15:48.874453 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:48.874422    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0b5742c-f6a7-4fa5-8fa6-e7aaa32175bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5130b62f-5lzt2\" (UID: \"f0b5742c-f6a7-4fa5-8fa6-e7aaa32175bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5130b62f-5lzt2"
Nov 02 03:15:48.874591 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:48.874541    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhtp5\" (UniqueName: \"kubernetes.io/projected/f0b5742c-f6a7-4fa5-8fa6-e7aaa32175bb-kube-api-access-zhtp5\") pod \"maas-api-revoke-sub-5130b62f-5lzt2\" (UID: \"f0b5742c-f6a7-4fa5-8fa6-e7aaa32175bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5130b62f-5lzt2"
Nov 02 03:15:48.874640 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:48.874612    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0b5742c-f6a7-4fa5-8fa6-e7aaa32175bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5130b62f-5lzt2\" (UID: \"f0b5742c-f6a7-4fa5-8fa6-e7aaa32175bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5130b62f-5lzt2"
Nov 02 03:15:48.900677 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:48.900650    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5130b62f-5lzt2"]
Nov 02 03:15:48.946710 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:48.946681    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 03:15:48.946879 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:48.946707    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 03:15:48.946879 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:48.946863    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 03:15:48.953317 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:48.953285    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9z5bk\""
Nov 02 03:15:48.975641 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:48.975615    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhtp5\" (UniqueName: \"kubernetes.io/projected/f0b5742c-f6a7-4fa5-8fa6-e7aaa32175bb-kube-api-access-zhtp5\") pod \"maas-api-revoke-sub-5130b62f-5lzt2\" (UID: \"f0b5742c-f6a7-4fa5-8fa6-e7aaa32175bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5130b62f-5lzt2"
Nov 02 03:15:48.975779 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:48.975698    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0b5742c-f6a7-4fa5-8fa6-e7aaa32175bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5130b62f-5lzt2\" (UID: \"f0b5742c-f6a7-4fa5-8fa6-e7aaa32175bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5130b62f-5lzt2"
Nov 02 03:15:48.975842 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:48.975820    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0b5742c-f6a7-4fa5-8fa6-e7aaa32175bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5130b62f-5lzt2\" (UID: \"f0b5742c-f6a7-4fa5-8fa6-e7aaa32175bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5130b62f-5lzt2"
Nov 02 03:15:48.976455 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:48.976433    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0b5742c-f6a7-4fa5-8fa6-e7aaa32175bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5130b62f-5lzt2\" (UID: \"f0b5742c-f6a7-4fa5-8fa6-e7aaa32175bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5130b62f-5lzt2"
Nov 02 03:15:48.978074 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:48.978055    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0b5742c-f6a7-4fa5-8fa6-e7aaa32175bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5130b62f-5lzt2\" (UID: \"f0b5742c-f6a7-4fa5-8fa6-e7aaa32175bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5130b62f-5lzt2"
Nov 02 03:15:49.033421 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:49.033391    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhtp5\" (UniqueName: \"kubernetes.io/projected/f0b5742c-f6a7-4fa5-8fa6-e7aaa32175bb-kube-api-access-zhtp5\") pod \"maas-api-revoke-sub-5130b62f-5lzt2\" (UID: \"f0b5742c-f6a7-4fa5-8fa6-e7aaa32175bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5130b62f-5lzt2"
Nov 02 03:15:49.164139 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:49.164058    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5130b62f-5lzt2"
Nov 02 03:15:49.556557 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:15:49.556521    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0b5742c_f6a7_4fa5_8fa6_e7aaa32175bb.slice/crio-0a7993c543df530512c0e7099dd7feb6dac1b2e4d651559d687780eed5282460 WatchSource:0}: Error finding container 0a7993c543df530512c0e7099dd7feb6dac1b2e4d651559d687780eed5282460: Status 404 returned error can't find the container with id 0a7993c543df530512c0e7099dd7feb6dac1b2e4d651559d687780eed5282460
Nov 02 03:15:49.556749 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:49.556724    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5130b62f-5lzt2"]
Nov 02 03:15:49.900288 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:49.900245    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5130b62f-5lzt2" event={"ID":"f0b5742c-f6a7-4fa5-8fa6-e7aaa32175bb","Type":"ContainerStarted","Data":"0a7993c543df530512c0e7099dd7feb6dac1b2e4d651559d687780eed5282460"}
Nov 02 03:15:50.910555 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:50.910516    4496 generic.go:356] "Generic (PLEG): container finished" podID="f0b5742c-f6a7-4fa5-8fa6-e7aaa32175bb" containerID="877ad8569a3b7a138dce60e9c8e75867599f15bf6a97d261920987830fe9d751" exitCode=0
Nov 02 03:15:50.911132 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:50.910730    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5130b62f-5lzt2" event={"ID":"f0b5742c-f6a7-4fa5-8fa6-e7aaa32175bb","Type":"ContainerDied","Data":"877ad8569a3b7a138dce60e9c8e75867599f15bf6a97d261920987830fe9d751"}
Nov 02 03:15:51.710444 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:51.710408    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-nmzqt"]
Nov 02 03:15:51.714376 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:51.714361    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-nmzqt"
Nov 02 03:15:51.767694 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:51.767670    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-1f6e39-openshift-default-dockercfg-qc8xh\""
Nov 02 03:15:51.804337 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:51.804312    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b9ba5f1-2280-4916-81d7-231b805e959e-istio-token\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-nmzqt\" (UID: \"9b9ba5f1-2280-4916-81d7-231b805e959e\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-nmzqt"
Nov 02 03:15:51.804480 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:51.804382    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b9ba5f1-2280-4916-81d7-231b805e959e-workload-socket\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-nmzqt\" (UID: \"9b9ba5f1-2280-4916-81d7-231b805e959e\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-nmzqt"
Nov 02 03:15:51.804480 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:51.804462    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b9ba5f1-2280-4916-81d7-231b805e959e-workload-certs\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-nmzqt\" (UID: \"9b9ba5f1-2280-4916-81d7-231b805e959e\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-nmzqt"
Nov 02 03:15:51.804578 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:51.804517    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ts4t5\" (UniqueName: \"kubernetes.io/projected/9b9ba5f1-2280-4916-81d7-231b805e959e-kube-api-access-ts4t5\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-nmzqt\" (UID: \"9b9ba5f1-2280-4916-81d7-231b805e959e\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-nmzqt"
Nov 02 03:15:51.804639 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:51.804610    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b9ba5f1-2280-4916-81d7-231b805e959e-istio-envoy\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-nmzqt\" (UID: \"9b9ba5f1-2280-4916-81d7-231b805e959e\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-nmzqt"
Nov 02 03:15:51.804715 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:51.804696    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b9ba5f1-2280-4916-81d7-231b805e959e-istio-data\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-nmzqt\" (UID: \"9b9ba5f1-2280-4916-81d7-231b805e959e\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-nmzqt"
Nov 02 03:15:51.804766 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:51.804753    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b9ba5f1-2280-4916-81d7-231b805e959e-credential-socket\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-nmzqt\" (UID: \"9b9ba5f1-2280-4916-81d7-231b805e959e\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-nmzqt"
Nov 02 03:15:51.804803 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:51.804782    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b9ba5f1-2280-4916-81d7-231b805e959e-istiod-ca-cert\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-nmzqt\" (UID: \"9b9ba5f1-2280-4916-81d7-231b805e959e\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-nmzqt"
Nov 02 03:15:51.804803 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:51.804799    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b9ba5f1-2280-4916-81d7-231b805e959e-istio-podinfo\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-nmzqt\" (UID: \"9b9ba5f1-2280-4916-81d7-231b805e959e\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-nmzqt"
Nov 02 03:15:51.824775 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:51.824744    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-nmzqt"]
Nov 02 03:15:51.905384 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:51.905353    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b9ba5f1-2280-4916-81d7-231b805e959e-istio-envoy\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-nmzqt\" (UID: \"9b9ba5f1-2280-4916-81d7-231b805e959e\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-nmzqt"
Nov 02 03:15:51.905571 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:51.905413    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b9ba5f1-2280-4916-81d7-231b805e959e-istio-data\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-nmzqt\" (UID: \"9b9ba5f1-2280-4916-81d7-231b805e959e\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-nmzqt"
Nov 02 03:15:51.905571 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:51.905447    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b9ba5f1-2280-4916-81d7-231b805e959e-credential-socket\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-nmzqt\" (UID: \"9b9ba5f1-2280-4916-81d7-231b805e959e\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-nmzqt"
Nov 02 03:15:51.905571 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:51.905494    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b9ba5f1-2280-4916-81d7-231b805e959e-istiod-ca-cert\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-nmzqt\" (UID: \"9b9ba5f1-2280-4916-81d7-231b805e959e\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-nmzqt"
Nov 02 03:15:51.905571 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:51.905521    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b9ba5f1-2280-4916-81d7-231b805e959e-istio-podinfo\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-nmzqt\" (UID: \"9b9ba5f1-2280-4916-81d7-231b805e959e\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-nmzqt"
Nov 02 03:15:51.905571 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:51.905551    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b9ba5f1-2280-4916-81d7-231b805e959e-istio-token\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-nmzqt\" (UID: \"9b9ba5f1-2280-4916-81d7-231b805e959e\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-nmzqt"
Nov 02 03:15:51.905852 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:51.905659    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b9ba5f1-2280-4916-81d7-231b805e959e-workload-socket\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-nmzqt\" (UID: \"9b9ba5f1-2280-4916-81d7-231b805e959e\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-nmzqt"
Nov 02 03:15:51.905852 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:51.905728    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b9ba5f1-2280-4916-81d7-231b805e959e-workload-certs\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-nmzqt\" (UID: \"9b9ba5f1-2280-4916-81d7-231b805e959e\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-nmzqt"
Nov 02 03:15:51.905852 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:51.905820    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ts4t5\" (UniqueName: \"kubernetes.io/projected/9b9ba5f1-2280-4916-81d7-231b805e959e-kube-api-access-ts4t5\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-nmzqt\" (UID: \"9b9ba5f1-2280-4916-81d7-231b805e959e\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-nmzqt"
Nov 02 03:15:51.906008 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:51.905938    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b9ba5f1-2280-4916-81d7-231b805e959e-istio-data\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-nmzqt\" (UID: \"9b9ba5f1-2280-4916-81d7-231b805e959e\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-nmzqt"
Nov 02 03:15:51.906138 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:51.906116    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b9ba5f1-2280-4916-81d7-231b805e959e-credential-socket\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-nmzqt\" (UID: \"9b9ba5f1-2280-4916-81d7-231b805e959e\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-nmzqt"
Nov 02 03:15:51.906232 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:51.906208    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b9ba5f1-2280-4916-81d7-231b805e959e-workload-socket\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-nmzqt\" (UID: \"9b9ba5f1-2280-4916-81d7-231b805e959e\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-nmzqt"
Nov 02 03:15:51.906274 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:51.906231    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b9ba5f1-2280-4916-81d7-231b805e959e-workload-certs\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-nmzqt\" (UID: \"9b9ba5f1-2280-4916-81d7-231b805e959e\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-nmzqt"
Nov 02 03:15:51.906420 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:51.906395    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b9ba5f1-2280-4916-81d7-231b805e959e-istiod-ca-cert\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-nmzqt\" (UID: \"9b9ba5f1-2280-4916-81d7-231b805e959e\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-nmzqt"
Nov 02 03:15:51.907700 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:51.907679    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b9ba5f1-2280-4916-81d7-231b805e959e-istio-envoy\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-nmzqt\" (UID: \"9b9ba5f1-2280-4916-81d7-231b805e959e\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-nmzqt"
Nov 02 03:15:51.908120 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:51.908104    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b9ba5f1-2280-4916-81d7-231b805e959e-istio-podinfo\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-nmzqt\" (UID: \"9b9ba5f1-2280-4916-81d7-231b805e959e\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-nmzqt"
Nov 02 03:15:51.962972 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:51.962891    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b9ba5f1-2280-4916-81d7-231b805e959e-istio-token\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-nmzqt\" (UID: \"9b9ba5f1-2280-4916-81d7-231b805e959e\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-nmzqt"
Nov 02 03:15:51.963390 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:51.963032    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ts4t5\" (UniqueName: \"kubernetes.io/projected/9b9ba5f1-2280-4916-81d7-231b805e959e-kube-api-access-ts4t5\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-nmzqt\" (UID: \"9b9ba5f1-2280-4916-81d7-231b805e959e\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-nmzqt"
Nov 02 03:15:52.028057 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:52.027997    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-nmzqt"
Nov 02 03:15:52.064363 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:52.064335    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5130b62f-5lzt2"
Nov 02 03:15:52.107779 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:52.107755    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0b5742c-f6a7-4fa5-8fa6-e7aaa32175bb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0b5742c-f6a7-4fa5-8fa6-e7aaa32175bb-maas-api-xks-ca\") pod \"f0b5742c-f6a7-4fa5-8fa6-e7aaa32175bb\" (UID: \"f0b5742c-f6a7-4fa5-8fa6-e7aaa32175bb\") "
Nov 02 03:15:52.107913 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:52.107887    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0b5742c-f6a7-4fa5-8fa6-e7aaa32175bb-kube-api-access-zhtp5\" (UniqueName: \"kubernetes.io/projected/f0b5742c-f6a7-4fa5-8fa6-e7aaa32175bb-kube-api-access-zhtp5\") pod \"f0b5742c-f6a7-4fa5-8fa6-e7aaa32175bb\" (UID: \"f0b5742c-f6a7-4fa5-8fa6-e7aaa32175bb\") "
Nov 02 03:15:52.107982 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:52.107956    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0b5742c-f6a7-4fa5-8fa6-e7aaa32175bb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0b5742c-f6a7-4fa5-8fa6-e7aaa32175bb-maas-api-openshift-ca\") pod \"f0b5742c-f6a7-4fa5-8fa6-e7aaa32175bb\" (UID: \"f0b5742c-f6a7-4fa5-8fa6-e7aaa32175bb\") "
Nov 02 03:15:52.108390 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:52.108360    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0b5742c-f6a7-4fa5-8fa6-e7aaa32175bb-maas-api-openshift-ca" pod "f0b5742c-f6a7-4fa5-8fa6-e7aaa32175bb" (UID: "f0b5742c-f6a7-4fa5-8fa6-e7aaa32175bb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:15:52.110853 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:52.110824    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0b5742c-f6a7-4fa5-8fa6-e7aaa32175bb-maas-api-xks-ca" pod "f0b5742c-f6a7-4fa5-8fa6-e7aaa32175bb" (UID: "f0b5742c-f6a7-4fa5-8fa6-e7aaa32175bb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 03:15:52.110942 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:52.110887    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0b5742c-f6a7-4fa5-8fa6-e7aaa32175bb-kube-api-access-zhtp5" pod "f0b5742c-f6a7-4fa5-8fa6-e7aaa32175bb" (UID: "f0b5742c-f6a7-4fa5-8fa6-e7aaa32175bb"). InnerVolumeSpecName "kube-api-access-zhtp5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:15:52.209961 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:52.209926    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0b5742c-f6a7-4fa5-8fa6-e7aaa32175bb-maas-api-openshift-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:15:52.209961 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:52.209964    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0b5742c-f6a7-4fa5-8fa6-e7aaa32175bb-maas-api-xks-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:15:52.210169 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:52.209982    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhtp5\" (UniqueName: \"kubernetes.io/projected/f0b5742c-f6a7-4fa5-8fa6-e7aaa32175bb-kube-api-access-zhtp5\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:15:52.460711 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:52.460684    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-nmzqt"]
Nov 02 03:15:52.461718 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:15:52.461690    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b9ba5f1_2280_4916_81d7_231b805e959e.slice/crio-dcb74053a57dc3f16abc881aac6fc309a39c967a2c8fd6d53cb448fca8958050 WatchSource:0}: Error finding container dcb74053a57dc3f16abc881aac6fc309a39c967a2c8fd6d53cb448fca8958050: Status 404 returned error can't find the container with id dcb74053a57dc3f16abc881aac6fc309a39c967a2c8fd6d53cb448fca8958050
Nov 02 03:15:52.464260 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:52.464225    4496 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 03:15:52.464353 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:52.464336    4496 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 03:15:52.464413 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:52.464386    4496 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 03:15:52.928944 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:52.928918    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5130b62f-5lzt2"
Nov 02 03:15:52.929106 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:52.928946    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5130b62f-5lzt2" event={"ID":"f0b5742c-f6a7-4fa5-8fa6-e7aaa32175bb","Type":"ContainerDied","Data":"0a7993c543df530512c0e7099dd7feb6dac1b2e4d651559d687780eed5282460"}
Nov 02 03:15:52.929106 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:52.929009    4496 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a7993c543df530512c0e7099dd7feb6dac1b2e4d651559d687780eed5282460"
Nov 02 03:15:52.930750 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:52.930727    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-nmzqt" event={"ID":"9b9ba5f1-2280-4916-81d7-231b805e959e","Type":"ContainerStarted","Data":"244e379c2d48153953ac24746d19b3947af2d45cee8f5d73a391c6723770c63b"}
Nov 02 03:15:52.930855 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:52.930766    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-nmzqt" event={"ID":"9b9ba5f1-2280-4916-81d7-231b805e959e","Type":"ContainerStarted","Data":"dcb74053a57dc3f16abc881aac6fc309a39c967a2c8fd6d53cb448fca8958050"}
Nov 02 03:15:53.028564 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:53.028533    4496 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-nmzqt"
Nov 02 03:15:53.030103 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:53.030076    4496 patch_prober.go:29] interesting pod/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-nmzqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body=
Nov 02 03:15:53.030201 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:53.030127    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-nmzqt" podUID="9b9ba5f1-2280-4916-81d7-231b805e959e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused"
Nov 02 03:15:53.058488 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:53.058422    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-nmzqt" podStartSLOduration=2.058401232 podStartE2EDuration="2.058401232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:15:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:15:53.050421305 +0000 UTC m=+1693.629028690" watchObservedRunningTime="2026-11-02 03:15:53.058401232 +0000 UTC m=+1693.637008613"
Nov 02 03:15:54.029026 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:54.028985    4496 patch_prober.go:29] interesting pod/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-nmzqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body=
Nov 02 03:15:54.029414 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:54.029059    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-nmzqt" podUID="9b9ba5f1-2280-4916-81d7-231b805e959e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused"
Nov 02 03:15:55.028682 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:55.028645    4496 patch_prober.go:29] interesting pod/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-nmzqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body=
Nov 02 03:15:55.028860 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:55.028700    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-nmzqt" podUID="9b9ba5f1-2280-4916-81d7-231b805e959e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused"
Nov 02 03:15:56.029357 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:56.029320    4496 patch_prober.go:29] interesting pod/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-nmzqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body=
Nov 02 03:15:56.029776 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:56.029379    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-nmzqt" podUID="9b9ba5f1-2280-4916-81d7-231b805e959e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused"
Nov 02 03:15:57.028384 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:57.028349    4496 patch_prober.go:29] interesting pod/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-nmzqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body=
Nov 02 03:15:57.028555 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:57.028397    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-nmzqt" podUID="9b9ba5f1-2280-4916-81d7-231b805e959e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused"
Nov 02 03:15:58.028847 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:58.028807    4496 patch_prober.go:29] interesting pod/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-nmzqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body=
Nov 02 03:15:58.029359 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:58.028862    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-nmzqt" podUID="9b9ba5f1-2280-4916-81d7-231b805e959e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused"
Nov 02 03:15:59.029123 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:59.029082    4496 patch_prober.go:29] interesting pod/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-nmzqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body=
Nov 02 03:15:59.029617 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:15:59.029133    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-nmzqt" podUID="9b9ba5f1-2280-4916-81d7-231b805e959e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused"
Nov 02 03:16:00.033464 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:00.033434    4496 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-nmzqt"
Nov 02 03:16:00.033932 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:00.033908    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-nmzqt"
Nov 02 03:16:00.034715 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:00.034692    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-nmzqt"
Nov 02 03:16:02.385135 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:02.385095    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-cbb4b6c58-87h44"]
Nov 02 03:16:02.386758 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:02.386730    4496 cpu_manager.go:414] "containerMap: removing container" podUID="f0b5742c-f6a7-4fa5-8fa6-e7aaa32175bb" containerName="revoke-keys"
Nov 02 03:16:02.386891 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:02.386878    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0b5742c-f6a7-4fa5-8fa6-e7aaa32175bb" containerName="revoke-keys"
Nov 02 03:16:02.387239 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:02.387222    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0b5742c-f6a7-4fa5-8fa6-e7aaa32175bb" containerName="revoke-keys"
Nov 02 03:16:02.391753 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:02.391731    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-cbb4b6c58-87h44"
Nov 02 03:16:02.440504 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:02.440468    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-cbb4b6c58-87h44"]
Nov 02 03:16:02.445411 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:02.445365    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ml52z\" (UniqueName: \"kubernetes.io/projected/3d90f14a-7e19-49df-8da8-ed96e12d2b96-kube-api-access-ml52z\") pod \"payload-processing-external-model-cbb4b6c58-87h44\" (UID: \"3d90f14a-7e19-49df-8da8-ed96e12d2b96\") " pod="models-as-a-service/payload-processing-external-model-cbb4b6c58-87h44"
Nov 02 03:16:02.445613 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:02.445548    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3d90f14a-7e19-49df-8da8-ed96e12d2b96-provider-credentials\") pod \"payload-processing-external-model-cbb4b6c58-87h44\" (UID: \"3d90f14a-7e19-49df-8da8-ed96e12d2b96\") " pod="models-as-a-service/payload-processing-external-model-cbb4b6c58-87h44"
Nov 02 03:16:02.445791 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:02.445762    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3d90f14a-7e19-49df-8da8-ed96e12d2b96-routing-overlay-volume\") pod \"payload-processing-external-model-cbb4b6c58-87h44\" (UID: \"3d90f14a-7e19-49df-8da8-ed96e12d2b96\") " pod="models-as-a-service/payload-processing-external-model-cbb4b6c58-87h44"
Nov 02 03:16:02.445860 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:02.445833    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d90f14a-7e19-49df-8da8-ed96e12d2b96-plugins-config-volume\") pod \"payload-processing-external-model-cbb4b6c58-87h44\" (UID: \"3d90f14a-7e19-49df-8da8-ed96e12d2b96\") " pod="models-as-a-service/payload-processing-external-model-cbb4b6c58-87h44"
Nov 02 03:16:02.546949 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:02.546906    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3d90f14a-7e19-49df-8da8-ed96e12d2b96-routing-overlay-volume\") pod \"payload-processing-external-model-cbb4b6c58-87h44\" (UID: \"3d90f14a-7e19-49df-8da8-ed96e12d2b96\") " pod="models-as-a-service/payload-processing-external-model-cbb4b6c58-87h44"
Nov 02 03:16:02.547590 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:02.546989    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d90f14a-7e19-49df-8da8-ed96e12d2b96-plugins-config-volume\") pod \"payload-processing-external-model-cbb4b6c58-87h44\" (UID: \"3d90f14a-7e19-49df-8da8-ed96e12d2b96\") " pod="models-as-a-service/payload-processing-external-model-cbb4b6c58-87h44"
Nov 02 03:16:02.547590 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:02.547232    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ml52z\" (UniqueName: \"kubernetes.io/projected/3d90f14a-7e19-49df-8da8-ed96e12d2b96-kube-api-access-ml52z\") pod \"payload-processing-external-model-cbb4b6c58-87h44\" (UID: \"3d90f14a-7e19-49df-8da8-ed96e12d2b96\") " pod="models-as-a-service/payload-processing-external-model-cbb4b6c58-87h44"
Nov 02 03:16:02.547590 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:02.547322    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3d90f14a-7e19-49df-8da8-ed96e12d2b96-provider-credentials\") pod \"payload-processing-external-model-cbb4b6c58-87h44\" (UID: \"3d90f14a-7e19-49df-8da8-ed96e12d2b96\") " pod="models-as-a-service/payload-processing-external-model-cbb4b6c58-87h44"
Nov 02 03:16:02.547871 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:02.547844    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3d90f14a-7e19-49df-8da8-ed96e12d2b96-routing-overlay-volume\") pod \"payload-processing-external-model-cbb4b6c58-87h44\" (UID: \"3d90f14a-7e19-49df-8da8-ed96e12d2b96\") " pod="models-as-a-service/payload-processing-external-model-cbb4b6c58-87h44"
Nov 02 03:16:02.548239 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:02.548192    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d90f14a-7e19-49df-8da8-ed96e12d2b96-plugins-config-volume\") pod \"payload-processing-external-model-cbb4b6c58-87h44\" (UID: \"3d90f14a-7e19-49df-8da8-ed96e12d2b96\") " pod="models-as-a-service/payload-processing-external-model-cbb4b6c58-87h44"
Nov 02 03:16:02.550201 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:02.550175    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3d90f14a-7e19-49df-8da8-ed96e12d2b96-provider-credentials\") pod \"payload-processing-external-model-cbb4b6c58-87h44\" (UID: \"3d90f14a-7e19-49df-8da8-ed96e12d2b96\") " pod="models-as-a-service/payload-processing-external-model-cbb4b6c58-87h44"
Nov 02 03:16:02.604640 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:02.604606    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ml52z\" (UniqueName: \"kubernetes.io/projected/3d90f14a-7e19-49df-8da8-ed96e12d2b96-kube-api-access-ml52z\") pod \"payload-processing-external-model-cbb4b6c58-87h44\" (UID: \"3d90f14a-7e19-49df-8da8-ed96e12d2b96\") " pod="models-as-a-service/payload-processing-external-model-cbb4b6c58-87h44"
Nov 02 03:16:02.707752 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:02.707665    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-cbb4b6c58-87h44"
Nov 02 03:16:02.727576 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:02.727544    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a20dfb7-7s952"]
Nov 02 03:16:02.737898 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:02.733265    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a20dfb7-7s952"
Nov 02 03:16:02.802092 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:02.789705    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a20dfb7-7s952"]
Nov 02 03:16:02.802092 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:02.790097    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 03:16:02.802092 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:02.790418    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 03:16:02.802092 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:02.790570    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9z5bk\""
Nov 02 03:16:02.802092 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:02.790728    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 03:16:02.850386 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:02.850113    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a16f45f-6782-4818-b31d-a341e55d8017-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a20dfb7-7s952\" (UID: \"4a16f45f-6782-4818-b31d-a341e55d8017\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a20dfb7-7s952"
Nov 02 03:16:02.850386 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:02.850182    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a16f45f-6782-4818-b31d-a341e55d8017-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a20dfb7-7s952\" (UID: \"4a16f45f-6782-4818-b31d-a341e55d8017\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a20dfb7-7s952"
Nov 02 03:16:02.850386 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:02.850368    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nf86\" (UniqueName: \"kubernetes.io/projected/4a16f45f-6782-4818-b31d-a341e55d8017-kube-api-access-6nf86\") pod \"maas-api-revoke-sub-8a20dfb7-7s952\" (UID: \"4a16f45f-6782-4818-b31d-a341e55d8017\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a20dfb7-7s952"
Nov 02 03:16:02.952347 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:02.952302    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a16f45f-6782-4818-b31d-a341e55d8017-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a20dfb7-7s952\" (UID: \"4a16f45f-6782-4818-b31d-a341e55d8017\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a20dfb7-7s952"
Nov 02 03:16:02.952578 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:02.952390    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a16f45f-6782-4818-b31d-a341e55d8017-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a20dfb7-7s952\" (UID: \"4a16f45f-6782-4818-b31d-a341e55d8017\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a20dfb7-7s952"
Nov 02 03:16:02.952578 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:02.952444    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nf86\" (UniqueName: \"kubernetes.io/projected/4a16f45f-6782-4818-b31d-a341e55d8017-kube-api-access-6nf86\") pod \"maas-api-revoke-sub-8a20dfb7-7s952\" (UID: \"4a16f45f-6782-4818-b31d-a341e55d8017\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a20dfb7-7s952"
Nov 02 03:16:02.953252 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:02.953220    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a16f45f-6782-4818-b31d-a341e55d8017-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a20dfb7-7s952\" (UID: \"4a16f45f-6782-4818-b31d-a341e55d8017\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a20dfb7-7s952"
Nov 02 03:16:02.955169 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:02.955138    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a16f45f-6782-4818-b31d-a341e55d8017-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a20dfb7-7s952\" (UID: \"4a16f45f-6782-4818-b31d-a341e55d8017\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a20dfb7-7s952"
Nov 02 03:16:03.011659 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:03.011627    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nf86\" (UniqueName: \"kubernetes.io/projected/4a16f45f-6782-4818-b31d-a341e55d8017-kube-api-access-6nf86\") pod \"maas-api-revoke-sub-8a20dfb7-7s952\" (UID: \"4a16f45f-6782-4818-b31d-a341e55d8017\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a20dfb7-7s952"
Nov 02 03:16:03.014549 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:03.014519    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a40aafd9-8rzqh"]
Nov 02 03:16:03.019141 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:03.019116    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a40aafd9-8rzqh"
Nov 02 03:16:03.061029 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:03.059669    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a20dfb7-7s952"
Nov 02 03:16:03.071926 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:03.062036    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b51ad1e-c78a-44b8-bad0-870b9e5ec1bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a40aafd9-8rzqh\" (UID: \"6b51ad1e-c78a-44b8-bad0-870b9e5ec1bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a40aafd9-8rzqh"
Nov 02 03:16:03.071926 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:03.062245    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbnh6\" (UniqueName: \"kubernetes.io/projected/6b51ad1e-c78a-44b8-bad0-870b9e5ec1bf-kube-api-access-sbnh6\") pod \"maas-api-revoke-sub-a40aafd9-8rzqh\" (UID: \"6b51ad1e-c78a-44b8-bad0-870b9e5ec1bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a40aafd9-8rzqh"
Nov 02 03:16:03.071926 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:03.062307    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b51ad1e-c78a-44b8-bad0-870b9e5ec1bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a40aafd9-8rzqh\" (UID: \"6b51ad1e-c78a-44b8-bad0-870b9e5ec1bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a40aafd9-8rzqh"
Nov 02 03:16:03.111356 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:03.106585    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a40aafd9-8rzqh"]
Nov 02 03:16:03.111356 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:03.110651    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-cbb4b6c58-87h44"]
Nov 02 03:16:03.112273 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:16:03.112244    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d90f14a_7e19_49df_8da8_ed96e12d2b96.slice/crio-1e5bbc0a21ebe84a4be6e9ee55389135eb34c56a310f8d979fc23ed2c66a4f52 WatchSource:0}: Error finding container 1e5bbc0a21ebe84a4be6e9ee55389135eb34c56a310f8d979fc23ed2c66a4f52: Status 404 returned error can't find the container with id 1e5bbc0a21ebe84a4be6e9ee55389135eb34c56a310f8d979fc23ed2c66a4f52
Nov 02 03:16:03.164233 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:03.163553    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbnh6\" (UniqueName: \"kubernetes.io/projected/6b51ad1e-c78a-44b8-bad0-870b9e5ec1bf-kube-api-access-sbnh6\") pod \"maas-api-revoke-sub-a40aafd9-8rzqh\" (UID: \"6b51ad1e-c78a-44b8-bad0-870b9e5ec1bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a40aafd9-8rzqh"
Nov 02 03:16:03.164233 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:03.163640    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b51ad1e-c78a-44b8-bad0-870b9e5ec1bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a40aafd9-8rzqh\" (UID: \"6b51ad1e-c78a-44b8-bad0-870b9e5ec1bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a40aafd9-8rzqh"
Nov 02 03:16:03.164233 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:03.163705    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b51ad1e-c78a-44b8-bad0-870b9e5ec1bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a40aafd9-8rzqh\" (UID: \"6b51ad1e-c78a-44b8-bad0-870b9e5ec1bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a40aafd9-8rzqh"
Nov 02 03:16:03.165597 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:03.165535    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b51ad1e-c78a-44b8-bad0-870b9e5ec1bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a40aafd9-8rzqh\" (UID: \"6b51ad1e-c78a-44b8-bad0-870b9e5ec1bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a40aafd9-8rzqh"
Nov 02 03:16:03.168497 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:03.168470    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b51ad1e-c78a-44b8-bad0-870b9e5ec1bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a40aafd9-8rzqh\" (UID: \"6b51ad1e-c78a-44b8-bad0-870b9e5ec1bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a40aafd9-8rzqh"
Nov 02 03:16:03.221846 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:03.221816    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbnh6\" (UniqueName: \"kubernetes.io/projected/6b51ad1e-c78a-44b8-bad0-870b9e5ec1bf-kube-api-access-sbnh6\") pod \"maas-api-revoke-sub-a40aafd9-8rzqh\" (UID: \"6b51ad1e-c78a-44b8-bad0-870b9e5ec1bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a40aafd9-8rzqh"
Nov 02 03:16:03.393375 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:03.393287    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a40aafd9-8rzqh"
Nov 02 03:16:03.502226 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:03.502118    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a20dfb7-7s952"]
Nov 02 03:16:03.505250 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:16:03.505216    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a16f45f_6782_4818_b31d_a341e55d8017.slice/crio-66104dfff6a5b74c4c7699a3b5178d99811d5152480899dd1327058cd5e63502 WatchSource:0}: Error finding container 66104dfff6a5b74c4c7699a3b5178d99811d5152480899dd1327058cd5e63502: Status 404 returned error can't find the container with id 66104dfff6a5b74c4c7699a3b5178d99811d5152480899dd1327058cd5e63502
Nov 02 03:16:03.539763 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:16:03.539682    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 03:16:03.784089 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:03.784053    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a40aafd9-8rzqh"]
Nov 02 03:16:03.784572 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:16:03.784533    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b51ad1e_c78a_44b8_bad0_870b9e5ec1bf.slice/crio-8ac0b55244475f1704caf35fb9c0f7d6d29d53cba0c43cae9ceb6f2544b7f4ba WatchSource:0}: Error finding container 8ac0b55244475f1704caf35fb9c0f7d6d29d53cba0c43cae9ceb6f2544b7f4ba: Status 404 returned error can't find the container with id 8ac0b55244475f1704caf35fb9c0f7d6d29d53cba0c43cae9ceb6f2544b7f4ba
Nov 02 03:16:04.041146 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:04.041048    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-cbb4b6c58-87h44" event={"ID":"3d90f14a-7e19-49df-8da8-ed96e12d2b96","Type":"ContainerStarted","Data":"5d3affa1a31acb1879d54584321bddc42bbed6140d624a589d4252a299779853"}
Nov 02 03:16:04.041146 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:04.041119    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-cbb4b6c58-87h44" event={"ID":"3d90f14a-7e19-49df-8da8-ed96e12d2b96","Type":"ContainerStarted","Data":"1e5bbc0a21ebe84a4be6e9ee55389135eb34c56a310f8d979fc23ed2c66a4f52"}
Nov 02 03:16:04.041146 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:04.041147    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-cbb4b6c58-87h44"
Nov 02 03:16:04.042905 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:04.042877    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a20dfb7-7s952" event={"ID":"4a16f45f-6782-4818-b31d-a341e55d8017","Type":"ContainerStarted","Data":"b3951b9189d98c23c099a2324723e090841adf39327a9530d56f7e197e18e556"}
Nov 02 03:16:04.043033 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:04.042924    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a20dfb7-7s952" event={"ID":"4a16f45f-6782-4818-b31d-a341e55d8017","Type":"ContainerStarted","Data":"66104dfff6a5b74c4c7699a3b5178d99811d5152480899dd1327058cd5e63502"}
Nov 02 03:16:04.044307 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:04.044279    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a40aafd9-8rzqh" event={"ID":"6b51ad1e-c78a-44b8-bad0-870b9e5ec1bf","Type":"ContainerStarted","Data":"8ac0b55244475f1704caf35fb9c0f7d6d29d53cba0c43cae9ceb6f2544b7f4ba"}
Nov 02 03:16:04.157348 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:04.157303    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-cbb4b6c58-87h44" podStartSLOduration=2.157289161 podStartE2EDuration="2.157289161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:16:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:16:04.156919573 +0000 UTC m=+1704.735526957" watchObservedRunningTime="2026-11-02 03:16:04.157289161 +0000 UTC m=+1704.735896541"
Nov 02 03:16:04.273030 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:04.272966    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a20dfb7-7s952" podStartSLOduration=2.27294777 podStartE2EDuration="2.27294777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:16:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:16:04.270956073 +0000 UTC m=+1704.849563453" watchObservedRunningTime="2026-11-02 03:16:04.27294777 +0000 UTC m=+1704.851555151"
Nov 02 03:16:05.053261 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:05.053225    4496 generic.go:356] "Generic (PLEG): container finished" podID="6b51ad1e-c78a-44b8-bad0-870b9e5ec1bf" containerID="d1430193ebb75e2a8ff159897334a9554c1bc8c5392d7c48c0bbbce5d1ee27a4" exitCode=0
Nov 02 03:16:05.053711 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:05.053320    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a40aafd9-8rzqh" event={"ID":"6b51ad1e-c78a-44b8-bad0-870b9e5ec1bf","Type":"ContainerDied","Data":"d1430193ebb75e2a8ff159897334a9554c1bc8c5392d7c48c0bbbce5d1ee27a4"}
Nov 02 03:16:05.055273 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:05.055249    4496 generic.go:356] "Generic (PLEG): container finished" podID="4a16f45f-6782-4818-b31d-a341e55d8017" containerID="b3951b9189d98c23c099a2324723e090841adf39327a9530d56f7e197e18e556" exitCode=0
Nov 02 03:16:05.055411 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:05.055334    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a20dfb7-7s952" event={"ID":"4a16f45f-6782-4818-b31d-a341e55d8017","Type":"ContainerDied","Data":"b3951b9189d98c23c099a2324723e090841adf39327a9530d56f7e197e18e556"}
Nov 02 03:16:06.223706 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:06.223666    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a40aafd9-8rzqh"
Nov 02 03:16:06.259179 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:06.259148    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a20dfb7-7s952"
Nov 02 03:16:06.300276 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:06.300247    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b51ad1e-c78a-44b8-bad0-870b9e5ec1bf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b51ad1e-c78a-44b8-bad0-870b9e5ec1bf-maas-api-openshift-ca\") pod \"6b51ad1e-c78a-44b8-bad0-870b9e5ec1bf\" (UID: \"6b51ad1e-c78a-44b8-bad0-870b9e5ec1bf\") "
Nov 02 03:16:06.300440 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:06.300427    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b51ad1e-c78a-44b8-bad0-870b9e5ec1bf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b51ad1e-c78a-44b8-bad0-870b9e5ec1bf-maas-api-xks-ca\") pod \"6b51ad1e-c78a-44b8-bad0-870b9e5ec1bf\" (UID: \"6b51ad1e-c78a-44b8-bad0-870b9e5ec1bf\") "
Nov 02 03:16:06.300482 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:06.300467    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b51ad1e-c78a-44b8-bad0-870b9e5ec1bf-kube-api-access-sbnh6\" (UniqueName: \"kubernetes.io/projected/6b51ad1e-c78a-44b8-bad0-870b9e5ec1bf-kube-api-access-sbnh6\") pod \"6b51ad1e-c78a-44b8-bad0-870b9e5ec1bf\" (UID: \"6b51ad1e-c78a-44b8-bad0-870b9e5ec1bf\") "
Nov 02 03:16:06.300578 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:06.300555    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b51ad1e-c78a-44b8-bad0-870b9e5ec1bf-maas-api-openshift-ca" pod "6b51ad1e-c78a-44b8-bad0-870b9e5ec1bf" (UID: "6b51ad1e-c78a-44b8-bad0-870b9e5ec1bf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:16:06.300820 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:06.300797    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b51ad1e-c78a-44b8-bad0-870b9e5ec1bf-maas-api-openshift-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:16:06.302510 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:06.302488    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b51ad1e-c78a-44b8-bad0-870b9e5ec1bf-maas-api-xks-ca" pod "6b51ad1e-c78a-44b8-bad0-870b9e5ec1bf" (UID: "6b51ad1e-c78a-44b8-bad0-870b9e5ec1bf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 03:16:06.302588 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:06.302491    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b51ad1e-c78a-44b8-bad0-870b9e5ec1bf-kube-api-access-sbnh6" pod "6b51ad1e-c78a-44b8-bad0-870b9e5ec1bf" (UID: "6b51ad1e-c78a-44b8-bad0-870b9e5ec1bf"). InnerVolumeSpecName "kube-api-access-sbnh6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:16:06.401977 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:06.401896    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a16f45f-6782-4818-b31d-a341e55d8017-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a16f45f-6782-4818-b31d-a341e55d8017-maas-api-openshift-ca\") pod \"4a16f45f-6782-4818-b31d-a341e55d8017\" (UID: \"4a16f45f-6782-4818-b31d-a341e55d8017\") "
Nov 02 03:16:06.401977 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:06.401945    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a16f45f-6782-4818-b31d-a341e55d8017-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a16f45f-6782-4818-b31d-a341e55d8017-maas-api-xks-ca\") pod \"4a16f45f-6782-4818-b31d-a341e55d8017\" (UID: \"4a16f45f-6782-4818-b31d-a341e55d8017\") "
Nov 02 03:16:06.402230 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:06.402137    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a16f45f-6782-4818-b31d-a341e55d8017-kube-api-access-6nf86\" (UniqueName: \"kubernetes.io/projected/4a16f45f-6782-4818-b31d-a341e55d8017-kube-api-access-6nf86\") pod \"4a16f45f-6782-4818-b31d-a341e55d8017\" (UID: \"4a16f45f-6782-4818-b31d-a341e55d8017\") "
Nov 02 03:16:06.402409 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:06.402378    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a16f45f-6782-4818-b31d-a341e55d8017-maas-api-openshift-ca" pod "4a16f45f-6782-4818-b31d-a341e55d8017" (UID: "4a16f45f-6782-4818-b31d-a341e55d8017"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:16:06.402579 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:06.402559    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbnh6\" (UniqueName: \"kubernetes.io/projected/6b51ad1e-c78a-44b8-bad0-870b9e5ec1bf-kube-api-access-sbnh6\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:16:06.402657 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:06.402581    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a16f45f-6782-4818-b31d-a341e55d8017-maas-api-openshift-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:16:06.402657 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:06.402597    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b51ad1e-c78a-44b8-bad0-870b9e5ec1bf-maas-api-xks-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:16:06.404218 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:06.404199    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a16f45f-6782-4818-b31d-a341e55d8017-kube-api-access-6nf86" pod "4a16f45f-6782-4818-b31d-a341e55d8017" (UID: "4a16f45f-6782-4818-b31d-a341e55d8017"). InnerVolumeSpecName "kube-api-access-6nf86". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:16:06.404218 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:06.404209    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a16f45f-6782-4818-b31d-a341e55d8017-maas-api-xks-ca" pod "4a16f45f-6782-4818-b31d-a341e55d8017" (UID: "4a16f45f-6782-4818-b31d-a341e55d8017"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 03:16:06.503655 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:06.503620    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a16f45f-6782-4818-b31d-a341e55d8017-maas-api-xks-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:16:06.503655 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:06.503649    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nf86\" (UniqueName: \"kubernetes.io/projected/4a16f45f-6782-4818-b31d-a341e55d8017-kube-api-access-6nf86\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:16:07.070138 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:07.070102    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a20dfb7-7s952"
Nov 02 03:16:07.070138 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:07.070126    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a20dfb7-7s952" event={"ID":"4a16f45f-6782-4818-b31d-a341e55d8017","Type":"ContainerDied","Data":"66104dfff6a5b74c4c7699a3b5178d99811d5152480899dd1327058cd5e63502"}
Nov 02 03:16:07.070362 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:07.070182    4496 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="66104dfff6a5b74c4c7699a3b5178d99811d5152480899dd1327058cd5e63502"
Nov 02 03:16:07.071641 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:07.071611    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a40aafd9-8rzqh"
Nov 02 03:16:07.071641 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:07.071629    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a40aafd9-8rzqh" event={"ID":"6b51ad1e-c78a-44b8-bad0-870b9e5ec1bf","Type":"ContainerDied","Data":"8ac0b55244475f1704caf35fb9c0f7d6d29d53cba0c43cae9ceb6f2544b7f4ba"}
Nov 02 03:16:07.071826 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:07.071659    4496 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8ac0b55244475f1704caf35fb9c0f7d6d29d53cba0c43cae9ceb6f2544b7f4ba"
Nov 02 03:16:12.670880 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:16:12.670846    4496 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-06e7ce0d87b6: secret "agc-cred-06e7ce0d87b6" not found
Nov 02 03:16:12.670880 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:16:12.670877    4496 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-08e8b2351e52: secret "agc-cred-08e8b2351e52" not found
Nov 02 03:16:12.671457 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:16:12.670895    4496 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-777d6b586f-brq7x: [secret "agc-cred-06e7ce0d87b6" not found, secret "agc-cred-08e8b2351e52" not found]
Nov 02 03:16:12.671457 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:16:12.670958    4496 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/775f32d1-c694-4cdc-8a83-d82930ffaf37-provider-credentials podName:775f32d1-c694-4cdc-8a83-d82930ffaf37 nodeName:}" failed. No retries permitted until 2026-11-02 03:16:13.170939118 +0000 UTC m=+1713.749546493 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/775f32d1-c694-4cdc-8a83-d82930ffaf37-provider-credentials") pod "payload-processing-external-model-777d6b586f-brq7x" (UID: "775f32d1-c694-4cdc-8a83-d82930ffaf37") : [secret "agc-cred-06e7ce0d87b6" not found, secret "agc-cred-08e8b2351e52" not found]
Nov 02 03:16:13.176736 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:16:13.176703    4496 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-06e7ce0d87b6: secret "agc-cred-06e7ce0d87b6" not found
Nov 02 03:16:13.176736 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:16:13.176733    4496 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-08e8b2351e52: secret "agc-cred-08e8b2351e52" not found
Nov 02 03:16:13.176981 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:16:13.176749    4496 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-777d6b586f-brq7x: [secret "agc-cred-06e7ce0d87b6" not found, secret "agc-cred-08e8b2351e52" not found]
Nov 02 03:16:13.176981 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:16:13.176825    4496 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/775f32d1-c694-4cdc-8a83-d82930ffaf37-provider-credentials podName:775f32d1-c694-4cdc-8a83-d82930ffaf37 nodeName:}" failed. No retries permitted until 2026-11-02 03:16:14.176807429 +0000 UTC m=+1714.755414805 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/775f32d1-c694-4cdc-8a83-d82930ffaf37-provider-credentials") pod "payload-processing-external-model-777d6b586f-brq7x" (UID: "775f32d1-c694-4cdc-8a83-d82930ffaf37") : [secret "agc-cred-06e7ce0d87b6" not found, secret "agc-cred-08e8b2351e52" not found]
Nov 02 03:16:14.188151 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:16:14.188112    4496 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-06e7ce0d87b6: secret "agc-cred-06e7ce0d87b6" not found
Nov 02 03:16:14.189784 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:16:14.189762    4496 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-08e8b2351e52: secret "agc-cred-08e8b2351e52" not found
Nov 02 03:16:14.189967 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:16:14.189953    4496 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-777d6b586f-brq7x: [secret "agc-cred-06e7ce0d87b6" not found, secret "agc-cred-08e8b2351e52" not found]
Nov 02 03:16:14.190164 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:16:14.190148    4496 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/775f32d1-c694-4cdc-8a83-d82930ffaf37-provider-credentials podName:775f32d1-c694-4cdc-8a83-d82930ffaf37 nodeName:}" failed. No retries permitted until 2026-11-02 03:16:16.190125609 +0000 UTC m=+1716.768732987 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/775f32d1-c694-4cdc-8a83-d82930ffaf37-provider-credentials") pod "payload-processing-external-model-777d6b586f-brq7x" (UID: "775f32d1-c694-4cdc-8a83-d82930ffaf37") : [secret "agc-cred-06e7ce0d87b6" not found, secret "agc-cred-08e8b2351e52" not found]
Nov 02 03:16:15.082167 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:15.082126    4496 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-cbb4b6c58-87h44" podUID="3d90f14a-7e19-49df-8da8-ed96e12d2b96" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 03:16:15.597164 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:15.597122    4496 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ca6546/e2e-conflict-a-5eca459f-kserve-5744ffc845-xgm78" podUID="c862a274-1d99-4f90-926e-4a72041456ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.97:8000/ready\": context deadline exceeded"
Nov 02 03:16:15.862447 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:15.611128    4496 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xlbmw" podUID="2baeda96-f36e-49cb-bb8a-3389c9e8d61d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 03:16:15.887399 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:15.868276    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-cbb4b6c58-87h44"]
Nov 02 03:16:15.888804 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:15.887649    4496 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5ac93f/distinct-w1-5ac93f-kserve-64697cb8c9-vj9sk" podUID="75a1c221-38e8-47cf-a8b5-305396cee703" containerName="main" probeResult="failure" output="Get \"https://10.134.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 03:16:15.938295 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:15.911316    4496 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ca6546/simulator-w2-ca6546-kserve-5959bb649b-5x6tf" podUID="4fd6a8f7-0b10-4009-a6db-de0a01aca2b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 03:16:15.938835 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:15.938814    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-777d6b586f-brq7x"]
Nov 02 03:16:15.973184 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:15.973101    4496 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-cbb4b6c58-87h44" podUID="3d90f14a-7e19-49df-8da8-ed96e12d2b96" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 03:16:16.132056 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:16.131930    4496 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-398828/simulator-w4-398828-kserve-6d5bd49c6f-9tw2c" podUID="5b6aecfa-3e3b-4fb3-a80c-35b77de60ce2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 03:16:16.330966 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:16:16.330930    4496 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-06e7ce0d87b6: secret "agc-cred-06e7ce0d87b6" not found
Nov 02 03:16:16.331244 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:16:16.331228    4496 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-08e8b2351e52: secret "agc-cred-08e8b2351e52" not found
Nov 02 03:16:16.331342 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:16:16.331334    4496 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-cb01f95f8193: secret "agc-cred-cb01f95f8193" not found
Nov 02 03:16:16.331422 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:16:16.331412    4496 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-777d6b586f-brq7x: [secret "agc-cred-06e7ce0d87b6" not found, secret "agc-cred-08e8b2351e52" not found, secret "agc-cred-cb01f95f8193" not found]
Nov 02 03:16:16.331576 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:16:16.331564    4496 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/775f32d1-c694-4cdc-8a83-d82930ffaf37-provider-credentials podName:775f32d1-c694-4cdc-8a83-d82930ffaf37 nodeName:}" failed. No retries permitted until 2026-11-02 03:16:20.331525041 +0000 UTC m=+1720.910132417 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/775f32d1-c694-4cdc-8a83-d82930ffaf37-provider-credentials") pod "payload-processing-external-model-777d6b586f-brq7x" (UID: "775f32d1-c694-4cdc-8a83-d82930ffaf37") : [secret "agc-cred-06e7ce0d87b6" not found, secret "agc-cred-08e8b2351e52" not found, secret "agc-cred-cb01f95f8193" not found]
Nov 02 03:16:16.421138 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:16.408469    4496 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-277pb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 03:16:16.421138 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:16.408536    4496 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-277pb" podUID="3051f935-d87a-422e-9073-909b2c2768ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 03:16:16.421138 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:16.408686    4496 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-lpxkv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 03:16:16.421138 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:16.408714    4496 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lpxkv" podUID="f395e8e3-ec87-4a63-94b5-8b29e017c2e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 03:16:18.635908 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:18.635874    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4209a816-mtqkz"]
Nov 02 03:16:18.636654 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:18.636632    4496 cpu_manager.go:414] "containerMap: removing container" podUID="6b51ad1e-c78a-44b8-bad0-870b9e5ec1bf" containerName="revoke-keys"
Nov 02 03:16:18.636757 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:18.636658    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b51ad1e-c78a-44b8-bad0-870b9e5ec1bf" containerName="revoke-keys"
Nov 02 03:16:18.636757 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:18.636710    4496 cpu_manager.go:414] "containerMap: removing container" podUID="4a16f45f-6782-4818-b31d-a341e55d8017" containerName="revoke-keys"
Nov 02 03:16:18.636757 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:18.636719    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a16f45f-6782-4818-b31d-a341e55d8017" containerName="revoke-keys"
Nov 02 03:16:18.636905 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:18.636896    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b51ad1e-c78a-44b8-bad0-870b9e5ec1bf" containerName="revoke-keys"
Nov 02 03:16:18.636956 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:18.636918    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a16f45f-6782-4818-b31d-a341e55d8017" containerName="revoke-keys"
Nov 02 03:16:18.641925 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:18.641902    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4209a816-mtqkz"
Nov 02 03:16:18.681348 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:18.681319    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vlnd\" (UniqueName: \"kubernetes.io/projected/4bb06cf0-f055-46d1-81f2-81aea3dc5cdd-kube-api-access-5vlnd\") pod \"maas-api-revoke-sub-4209a816-mtqkz\" (UID: \"4bb06cf0-f055-46d1-81f2-81aea3dc5cdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4209a816-mtqkz"
Nov 02 03:16:18.681530 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:18.681390    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bb06cf0-f055-46d1-81f2-81aea3dc5cdd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4209a816-mtqkz\" (UID: \"4bb06cf0-f055-46d1-81f2-81aea3dc5cdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4209a816-mtqkz"
Nov 02 03:16:18.681635 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:18.681609    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bb06cf0-f055-46d1-81f2-81aea3dc5cdd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4209a816-mtqkz\" (UID: \"4bb06cf0-f055-46d1-81f2-81aea3dc5cdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4209a816-mtqkz"
Nov 02 03:16:18.695980 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:18.695953    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4209a816-mtqkz"]
Nov 02 03:16:18.741934 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:18.741908    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 03:16:18.741934 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:18.741906    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 03:16:18.741934 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:18.741910    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 03:16:18.745487 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:18.745454    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9z5bk\""
Nov 02 03:16:18.782482 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:18.782455    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vlnd\" (UniqueName: \"kubernetes.io/projected/4bb06cf0-f055-46d1-81f2-81aea3dc5cdd-kube-api-access-5vlnd\") pod \"maas-api-revoke-sub-4209a816-mtqkz\" (UID: \"4bb06cf0-f055-46d1-81f2-81aea3dc5cdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4209a816-mtqkz"
Nov 02 03:16:18.782611 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:18.782521    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bb06cf0-f055-46d1-81f2-81aea3dc5cdd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4209a816-mtqkz\" (UID: \"4bb06cf0-f055-46d1-81f2-81aea3dc5cdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4209a816-mtqkz"
Nov 02 03:16:18.782681 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:18.782611    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bb06cf0-f055-46d1-81f2-81aea3dc5cdd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4209a816-mtqkz\" (UID: \"4bb06cf0-f055-46d1-81f2-81aea3dc5cdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4209a816-mtqkz"
Nov 02 03:16:18.783249 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:18.783224    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bb06cf0-f055-46d1-81f2-81aea3dc5cdd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4209a816-mtqkz\" (UID: \"4bb06cf0-f055-46d1-81f2-81aea3dc5cdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4209a816-mtqkz"
Nov 02 03:16:18.784969 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:18.784948    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bb06cf0-f055-46d1-81f2-81aea3dc5cdd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4209a816-mtqkz\" (UID: \"4bb06cf0-f055-46d1-81f2-81aea3dc5cdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4209a816-mtqkz"
Nov 02 03:16:18.840355 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:18.840325    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vlnd\" (UniqueName: \"kubernetes.io/projected/4bb06cf0-f055-46d1-81f2-81aea3dc5cdd-kube-api-access-5vlnd\") pod \"maas-api-revoke-sub-4209a816-mtqkz\" (UID: \"4bb06cf0-f055-46d1-81f2-81aea3dc5cdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4209a816-mtqkz"
Nov 02 03:16:18.960274 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:18.960188    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4209a816-mtqkz"
Nov 02 03:16:19.011714 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:19.011675    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a22e3d1e-vfjwn"]
Nov 02 03:16:19.019913 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:19.019892    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a22e3d1e-vfjwn"
Nov 02 03:16:19.073947 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:19.073916    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a22e3d1e-vfjwn"]
Nov 02 03:16:19.087056 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:19.086992    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a19741d-1644-42f2-a560-b14b5de2dcf9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a22e3d1e-vfjwn\" (UID: \"2a19741d-1644-42f2-a560-b14b5de2dcf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a22e3d1e-vfjwn"
Nov 02 03:16:19.087182 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:19.087146    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbzw8\" (UniqueName: \"kubernetes.io/projected/2a19741d-1644-42f2-a560-b14b5de2dcf9-kube-api-access-hbzw8\") pod \"maas-api-revoke-sub-a22e3d1e-vfjwn\" (UID: \"2a19741d-1644-42f2-a560-b14b5de2dcf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a22e3d1e-vfjwn"
Nov 02 03:16:19.087351 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:19.087328    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a19741d-1644-42f2-a560-b14b5de2dcf9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a22e3d1e-vfjwn\" (UID: \"2a19741d-1644-42f2-a560-b14b5de2dcf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a22e3d1e-vfjwn"
Nov 02 03:16:19.188624 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:19.188592    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a19741d-1644-42f2-a560-b14b5de2dcf9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a22e3d1e-vfjwn\" (UID: \"2a19741d-1644-42f2-a560-b14b5de2dcf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a22e3d1e-vfjwn"
Nov 02 03:16:19.188769 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:19.188656    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbzw8\" (UniqueName: \"kubernetes.io/projected/2a19741d-1644-42f2-a560-b14b5de2dcf9-kube-api-access-hbzw8\") pod \"maas-api-revoke-sub-a22e3d1e-vfjwn\" (UID: \"2a19741d-1644-42f2-a560-b14b5de2dcf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a22e3d1e-vfjwn"
Nov 02 03:16:19.188769 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:19.188742    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a19741d-1644-42f2-a560-b14b5de2dcf9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a22e3d1e-vfjwn\" (UID: \"2a19741d-1644-42f2-a560-b14b5de2dcf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a22e3d1e-vfjwn"
Nov 02 03:16:19.189251 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:19.189231    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a19741d-1644-42f2-a560-b14b5de2dcf9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a22e3d1e-vfjwn\" (UID: \"2a19741d-1644-42f2-a560-b14b5de2dcf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a22e3d1e-vfjwn"
Nov 02 03:16:19.190960 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:19.190940    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a19741d-1644-42f2-a560-b14b5de2dcf9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a22e3d1e-vfjwn\" (UID: \"2a19741d-1644-42f2-a560-b14b5de2dcf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a22e3d1e-vfjwn"
Nov 02 03:16:19.246215 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:19.246190    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbzw8\" (UniqueName: \"kubernetes.io/projected/2a19741d-1644-42f2-a560-b14b5de2dcf9-kube-api-access-hbzw8\") pod \"maas-api-revoke-sub-a22e3d1e-vfjwn\" (UID: \"2a19741d-1644-42f2-a560-b14b5de2dcf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a22e3d1e-vfjwn"
Nov 02 03:16:19.335351 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:19.335319    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4209a816-mtqkz"]
Nov 02 03:16:19.335481 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:16:19.335407    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bb06cf0_f055_46d1_81f2_81aea3dc5cdd.slice/crio-7e34a1c6c4f1fdfb8c3ba8bfe454d0c20b94f35f5ca357724a3e5a4b60f7fef6 WatchSource:0}: Error finding container 7e34a1c6c4f1fdfb8c3ba8bfe454d0c20b94f35f5ca357724a3e5a4b60f7fef6: Status 404 returned error can't find the container with id 7e34a1c6c4f1fdfb8c3ba8bfe454d0c20b94f35f5ca357724a3e5a4b60f7fef6
Nov 02 03:16:19.339379 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:19.339357    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a22e3d1e-vfjwn"
Nov 02 03:16:19.722790 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:16:19.722759    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a19741d_1644_42f2_a560_b14b5de2dcf9.slice/crio-547c3b0f4a94a538b347a8bebb4ae510c87b149f63fe30d62d168fbb8cf67609 WatchSource:0}: Error finding container 547c3b0f4a94a538b347a8bebb4ae510c87b149f63fe30d62d168fbb8cf67609: Status 404 returned error can't find the container with id 547c3b0f4a94a538b347a8bebb4ae510c87b149f63fe30d62d168fbb8cf67609
Nov 02 03:16:19.722790 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:19.722788    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a22e3d1e-vfjwn"]
Nov 02 03:16:20.285231 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:20.285195    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a22e3d1e-vfjwn" event={"ID":"2a19741d-1644-42f2-a560-b14b5de2dcf9","Type":"ContainerStarted","Data":"eff6ee9c20c069f3e7640ae158d7c32918e21df9be2403347c0a69d525ae905f"}
Nov 02 03:16:20.285433 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:20.285283    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a22e3d1e-vfjwn" event={"ID":"2a19741d-1644-42f2-a560-b14b5de2dcf9","Type":"ContainerStarted","Data":"547c3b0f4a94a538b347a8bebb4ae510c87b149f63fe30d62d168fbb8cf67609"}
Nov 02 03:16:20.287070 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:20.287040    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4209a816-mtqkz" event={"ID":"4bb06cf0-f055-46d1-81f2-81aea3dc5cdd","Type":"ContainerStarted","Data":"eabdf3ad41ba806ce5e37ec69d6535351452c26a36e80f59301011a2ebb6c210"}
Nov 02 03:16:20.287205 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:20.287084    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4209a816-mtqkz" event={"ID":"4bb06cf0-f055-46d1-81f2-81aea3dc5cdd","Type":"ContainerStarted","Data":"7e34a1c6c4f1fdfb8c3ba8bfe454d0c20b94f35f5ca357724a3e5a4b60f7fef6"}
Nov 02 03:16:20.402541 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:20.402476    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a22e3d1e-vfjwn" podStartSLOduration=2.402461331 podStartE2EDuration="2.402461331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:16:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:16:20.399917375 +0000 UTC m=+1720.978524758" watchObservedRunningTime="2026-11-02 03:16:20.402461331 +0000 UTC m=+1720.981068753"
Nov 02 03:16:20.405672 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:16:20.405632    4496 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-06e7ce0d87b6: secret "agc-cred-06e7ce0d87b6" not found
Nov 02 03:16:20.405672 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:16:20.405663    4496 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-08e8b2351e52: secret "agc-cred-08e8b2351e52" not found
Nov 02 03:16:20.405672 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:16:20.405674    4496 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-cb01f95f8193: secret "agc-cred-cb01f95f8193" not found
Nov 02 03:16:20.405882 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:16:20.405686    4496 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-777d6b586f-brq7x: [secret "agc-cred-06e7ce0d87b6" not found, secret "agc-cred-08e8b2351e52" not found, secret "agc-cred-cb01f95f8193" not found]
Nov 02 03:16:20.405882 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:16:20.405749    4496 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/775f32d1-c694-4cdc-8a83-d82930ffaf37-provider-credentials podName:775f32d1-c694-4cdc-8a83-d82930ffaf37 nodeName:}" failed. No retries permitted until 2026-11-02 03:16:28.405730544 +0000 UTC m=+1728.984337905 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/775f32d1-c694-4cdc-8a83-d82930ffaf37-provider-credentials") pod "payload-processing-external-model-777d6b586f-brq7x" (UID: "775f32d1-c694-4cdc-8a83-d82930ffaf37") : [secret "agc-cred-06e7ce0d87b6" not found, secret "agc-cred-08e8b2351e52" not found, secret "agc-cred-cb01f95f8193" not found]
Nov 02 03:16:20.514876 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:20.514831    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4209a816-mtqkz" podStartSLOduration=2.514814788 podStartE2EDuration="2.514814788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:16:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:16:20.514114536 +0000 UTC m=+1721.092721918" watchObservedRunningTime="2026-11-02 03:16:20.514814788 +0000 UTC m=+1721.093422170"
Nov 02 03:16:20.889361 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:20.889323    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-cbb4b6c58-87h44" podUID="3d90f14a-7e19-49df-8da8-ed96e12d2b96" containerName="payload-processing" containerID="cri-o://5d3affa1a31acb1879d54584321bddc42bbed6140d624a589d4252a299779853" gracePeriod=25
Nov 02 03:16:20.939618 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:20.939563    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-777d6b586f-brq7x" podUID="775f32d1-c694-4cdc-8a83-d82930ffaf37" containerName="payload-processing" containerID="cri-o://3cfe823620cb0594ab0260c8a31e8cc8743c7a50b49291679b36ffce70b6416e" gracePeriod=25
Nov 02 03:16:21.224236 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:21.224170    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-cbb4b6c58-87h44"
Nov 02 03:16:21.227417 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:21.227398    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-777d6b586f-brq7x"
Nov 02 03:16:21.295660 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:21.295631    4496 generic.go:356] "Generic (PLEG): container finished" podID="4bb06cf0-f055-46d1-81f2-81aea3dc5cdd" containerID="eabdf3ad41ba806ce5e37ec69d6535351452c26a36e80f59301011a2ebb6c210" exitCode=0
Nov 02 03:16:21.295818 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:21.295705    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4209a816-mtqkz" event={"ID":"4bb06cf0-f055-46d1-81f2-81aea3dc5cdd","Type":"ContainerDied","Data":"eabdf3ad41ba806ce5e37ec69d6535351452c26a36e80f59301011a2ebb6c210"}
Nov 02 03:16:21.297375 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:21.297349    4496 generic.go:356] "Generic (PLEG): container finished" podID="2a19741d-1644-42f2-a560-b14b5de2dcf9" containerID="eff6ee9c20c069f3e7640ae158d7c32918e21df9be2403347c0a69d525ae905f" exitCode=0
Nov 02 03:16:21.297489 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:21.297435    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a22e3d1e-vfjwn" event={"ID":"2a19741d-1644-42f2-a560-b14b5de2dcf9","Type":"ContainerDied","Data":"eff6ee9c20c069f3e7640ae158d7c32918e21df9be2403347c0a69d525ae905f"}
Nov 02 03:16:21.298732 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:21.298709    4496 generic.go:356] "Generic (PLEG): container finished" podID="775f32d1-c694-4cdc-8a83-d82930ffaf37" containerID="3cfe823620cb0594ab0260c8a31e8cc8743c7a50b49291679b36ffce70b6416e" exitCode=0
Nov 02 03:16:21.298845 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:21.298751    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-777d6b586f-brq7x"
Nov 02 03:16:21.298845 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:21.298794    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-777d6b586f-brq7x" event={"ID":"775f32d1-c694-4cdc-8a83-d82930ffaf37","Type":"ContainerDied","Data":"3cfe823620cb0594ab0260c8a31e8cc8743c7a50b49291679b36ffce70b6416e"}
Nov 02 03:16:21.298960 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:21.298847    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-777d6b586f-brq7x" event={"ID":"775f32d1-c694-4cdc-8a83-d82930ffaf37","Type":"ContainerDied","Data":"693d81bd97f75de38bbf8d1d578b99264a16d90c0b3162ef2116537bd42a186d"}
Nov 02 03:16:21.298960 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:21.298879    4496 scope.go:122] "RemoveContainer" containerID="3cfe823620cb0594ab0260c8a31e8cc8743c7a50b49291679b36ffce70b6416e"
Nov 02 03:16:21.300124 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:21.300101    4496 generic.go:356] "Generic (PLEG): container finished" podID="3d90f14a-7e19-49df-8da8-ed96e12d2b96" containerID="5d3affa1a31acb1879d54584321bddc42bbed6140d624a589d4252a299779853" exitCode=0
Nov 02 03:16:21.300226 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:21.300133    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-cbb4b6c58-87h44" event={"ID":"3d90f14a-7e19-49df-8da8-ed96e12d2b96","Type":"ContainerDied","Data":"5d3affa1a31acb1879d54584321bddc42bbed6140d624a589d4252a299779853"}
Nov 02 03:16:21.300226 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:21.300149    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-cbb4b6c58-87h44"
Nov 02 03:16:21.300226 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:21.300170    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-cbb4b6c58-87h44" event={"ID":"3d90f14a-7e19-49df-8da8-ed96e12d2b96","Type":"ContainerDied","Data":"1e5bbc0a21ebe84a4be6e9ee55389135eb34c56a310f8d979fc23ed2c66a4f52"}
Nov 02 03:16:21.309040 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:21.309024    4496 scope.go:122] "RemoveContainer" containerID="3cfe823620cb0594ab0260c8a31e8cc8743c7a50b49291679b36ffce70b6416e"
Nov 02 03:16:21.309321 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:16:21.309304    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cfe823620cb0594ab0260c8a31e8cc8743c7a50b49291679b36ffce70b6416e\": container with ID starting with 3cfe823620cb0594ab0260c8a31e8cc8743c7a50b49291679b36ffce70b6416e not found: ID does not exist" containerID="3cfe823620cb0594ab0260c8a31e8cc8743c7a50b49291679b36ffce70b6416e"
Nov 02 03:16:21.309394 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:21.309332    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cfe823620cb0594ab0260c8a31e8cc8743c7a50b49291679b36ffce70b6416e"} err="failed to get container status \"3cfe823620cb0594ab0260c8a31e8cc8743c7a50b49291679b36ffce70b6416e\": rpc error: code = NotFound desc = could not find container \"3cfe823620cb0594ab0260c8a31e8cc8743c7a50b49291679b36ffce70b6416e\": container with ID starting with 3cfe823620cb0594ab0260c8a31e8cc8743c7a50b49291679b36ffce70b6416e not found: ID does not exist"
Nov 02 03:16:21.309394 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:21.309355    4496 scope.go:122] "RemoveContainer" containerID="5d3affa1a31acb1879d54584321bddc42bbed6140d624a589d4252a299779853"
Nov 02 03:16:21.315536 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:21.315515    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/775f32d1-c694-4cdc-8a83-d82930ffaf37-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/775f32d1-c694-4cdc-8a83-d82930ffaf37-routing-overlay-volume\") pod \"775f32d1-c694-4cdc-8a83-d82930ffaf37\" (UID: \"775f32d1-c694-4cdc-8a83-d82930ffaf37\") "
Nov 02 03:16:21.315626 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:21.315594    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/775f32d1-c694-4cdc-8a83-d82930ffaf37-provider-credentials\" (UniqueName: \"kubernetes.io/projected/775f32d1-c694-4cdc-8a83-d82930ffaf37-provider-credentials\") pod \"775f32d1-c694-4cdc-8a83-d82930ffaf37\" (UID: \"775f32d1-c694-4cdc-8a83-d82930ffaf37\") "
Nov 02 03:16:21.315672 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:21.315630    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d90f14a-7e19-49df-8da8-ed96e12d2b96-kube-api-access-ml52z\" (UniqueName: \"kubernetes.io/projected/3d90f14a-7e19-49df-8da8-ed96e12d2b96-kube-api-access-ml52z\") pod \"3d90f14a-7e19-49df-8da8-ed96e12d2b96\" (UID: \"3d90f14a-7e19-49df-8da8-ed96e12d2b96\") "
Nov 02 03:16:21.315717 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:21.315671    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d90f14a-7e19-49df-8da8-ed96e12d2b96-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3d90f14a-7e19-49df-8da8-ed96e12d2b96-routing-overlay-volume\") pod \"3d90f14a-7e19-49df-8da8-ed96e12d2b96\" (UID: \"3d90f14a-7e19-49df-8da8-ed96e12d2b96\") "
Nov 02 03:16:21.315803 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:21.315784    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d90f14a-7e19-49df-8da8-ed96e12d2b96-provider-credentials\" (UniqueName: \"kubernetes.io/projected/3d90f14a-7e19-49df-8da8-ed96e12d2b96-provider-credentials\") pod \"3d90f14a-7e19-49df-8da8-ed96e12d2b96\" (UID: \"3d90f14a-7e19-49df-8da8-ed96e12d2b96\") "
Nov 02 03:16:21.315888 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:21.315872    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/775f32d1-c694-4cdc-8a83-d82930ffaf37-kube-api-access-mvbkx\" (UniqueName: \"kubernetes.io/projected/775f32d1-c694-4cdc-8a83-d82930ffaf37-kube-api-access-mvbkx\") pod \"775f32d1-c694-4cdc-8a83-d82930ffaf37\" (UID: \"775f32d1-c694-4cdc-8a83-d82930ffaf37\") "
Nov 02 03:16:21.315968 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:21.315924    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/775f32d1-c694-4cdc-8a83-d82930ffaf37-routing-overlay-volume" pod "775f32d1-c694-4cdc-8a83-d82930ffaf37" (UID: "775f32d1-c694-4cdc-8a83-d82930ffaf37"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:16:21.315968 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:21.315958    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d90f14a-7e19-49df-8da8-ed96e12d2b96-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d90f14a-7e19-49df-8da8-ed96e12d2b96-plugins-config-volume\") pod \"3d90f14a-7e19-49df-8da8-ed96e12d2b96\" (UID: \"3d90f14a-7e19-49df-8da8-ed96e12d2b96\") "
Nov 02 03:16:21.316097 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:21.315984    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/775f32d1-c694-4cdc-8a83-d82930ffaf37-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/775f32d1-c694-4cdc-8a83-d82930ffaf37-plugins-config-volume\") pod \"775f32d1-c694-4cdc-8a83-d82930ffaf37\" (UID: \"775f32d1-c694-4cdc-8a83-d82930ffaf37\") "
Nov 02 03:16:21.316097 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:21.316008    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d90f14a-7e19-49df-8da8-ed96e12d2b96-routing-overlay-volume" pod "3d90f14a-7e19-49df-8da8-ed96e12d2b96" (UID: "3d90f14a-7e19-49df-8da8-ed96e12d2b96"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:16:21.316650 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:21.316438    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d90f14a-7e19-49df-8da8-ed96e12d2b96-plugins-config-volume" pod "3d90f14a-7e19-49df-8da8-ed96e12d2b96" (UID: "3d90f14a-7e19-49df-8da8-ed96e12d2b96"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:16:21.316650 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:21.316446    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/775f32d1-c694-4cdc-8a83-d82930ffaf37-plugins-config-volume" pod "775f32d1-c694-4cdc-8a83-d82930ffaf37" (UID: "775f32d1-c694-4cdc-8a83-d82930ffaf37"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:16:21.316650 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:21.316573    4496 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/775f32d1-c694-4cdc-8a83-d82930ffaf37-routing-overlay-volume\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:16:21.316650 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:21.316593    4496 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3d90f14a-7e19-49df-8da8-ed96e12d2b96-routing-overlay-volume\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:16:21.316650 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:21.316609    4496 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d90f14a-7e19-49df-8da8-ed96e12d2b96-plugins-config-volume\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:16:21.316650 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:21.316623    4496 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/775f32d1-c694-4cdc-8a83-d82930ffaf37-plugins-config-volume\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:16:21.317984 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:21.317960    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/775f32d1-c694-4cdc-8a83-d82930ffaf37-kube-api-access-mvbkx" pod "775f32d1-c694-4cdc-8a83-d82930ffaf37" (UID: "775f32d1-c694-4cdc-8a83-d82930ffaf37"). InnerVolumeSpecName "kube-api-access-mvbkx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:16:21.318220 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:21.318006    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d90f14a-7e19-49df-8da8-ed96e12d2b96-provider-credentials" pod "3d90f14a-7e19-49df-8da8-ed96e12d2b96" (UID: "3d90f14a-7e19-49df-8da8-ed96e12d2b96"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:16:21.318220 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:21.318105    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/775f32d1-c694-4cdc-8a83-d82930ffaf37-provider-credentials" pod "775f32d1-c694-4cdc-8a83-d82930ffaf37" (UID: "775f32d1-c694-4cdc-8a83-d82930ffaf37"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:16:21.318373 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:21.318356    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d90f14a-7e19-49df-8da8-ed96e12d2b96-kube-api-access-ml52z" pod "3d90f14a-7e19-49df-8da8-ed96e12d2b96" (UID: "3d90f14a-7e19-49df-8da8-ed96e12d2b96"). InnerVolumeSpecName "kube-api-access-ml52z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:16:21.319299 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:21.319286    4496 scope.go:122] "RemoveContainer" containerID="5d3affa1a31acb1879d54584321bddc42bbed6140d624a589d4252a299779853"
Nov 02 03:16:21.319573 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:16:21.319551    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d3affa1a31acb1879d54584321bddc42bbed6140d624a589d4252a299779853\": container with ID starting with 5d3affa1a31acb1879d54584321bddc42bbed6140d624a589d4252a299779853 not found: ID does not exist" containerID="5d3affa1a31acb1879d54584321bddc42bbed6140d624a589d4252a299779853"
Nov 02 03:16:21.319613 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:21.319580    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d3affa1a31acb1879d54584321bddc42bbed6140d624a589d4252a299779853"} err="failed to get container status \"5d3affa1a31acb1879d54584321bddc42bbed6140d624a589d4252a299779853\": rpc error: code = NotFound desc = could not find container \"5d3affa1a31acb1879d54584321bddc42bbed6140d624a589d4252a299779853\": container with ID starting with 5d3affa1a31acb1879d54584321bddc42bbed6140d624a589d4252a299779853 not found: ID does not exist"
Nov 02 03:16:21.419087 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:21.419052    4496 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/775f32d1-c694-4cdc-8a83-d82930ffaf37-provider-credentials\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:16:21.419087 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:21.419088    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ml52z\" (UniqueName: \"kubernetes.io/projected/3d90f14a-7e19-49df-8da8-ed96e12d2b96-kube-api-access-ml52z\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:16:21.419299 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:21.419103    4496 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3d90f14a-7e19-49df-8da8-ed96e12d2b96-provider-credentials\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:16:21.419299 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:21.419118    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvbkx\" (UniqueName: \"kubernetes.io/projected/775f32d1-c694-4cdc-8a83-d82930ffaf37-kube-api-access-mvbkx\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:16:21.850065 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:21.849967    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-cbb4b6c58-87h44"]
Nov 02 03:16:21.855438 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:21.855412    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-cbb4b6c58-87h44"]
Nov 02 03:16:21.969541 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:21.969509    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-777d6b586f-brq7x"]
Nov 02 03:16:21.975166 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:21.975133    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-777d6b586f-brq7x"]
Nov 02 03:16:22.484193 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:22.484159    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4209a816-mtqkz"
Nov 02 03:16:22.515338 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:22.515312    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a22e3d1e-vfjwn"
Nov 02 03:16:22.531640 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:22.531616    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4bb06cf0-f055-46d1-81f2-81aea3dc5cdd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bb06cf0-f055-46d1-81f2-81aea3dc5cdd-maas-api-xks-ca\") pod \"4bb06cf0-f055-46d1-81f2-81aea3dc5cdd\" (UID: \"4bb06cf0-f055-46d1-81f2-81aea3dc5cdd\") "
Nov 02 03:16:22.531758 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:22.531731    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bb06cf0-f055-46d1-81f2-81aea3dc5cdd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bb06cf0-f055-46d1-81f2-81aea3dc5cdd-maas-api-openshift-ca\") pod \"4bb06cf0-f055-46d1-81f2-81aea3dc5cdd\" (UID: \"4bb06cf0-f055-46d1-81f2-81aea3dc5cdd\") "
Nov 02 03:16:22.531853 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:22.531840    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bb06cf0-f055-46d1-81f2-81aea3dc5cdd-kube-api-access-5vlnd\" (UniqueName: \"kubernetes.io/projected/4bb06cf0-f055-46d1-81f2-81aea3dc5cdd-kube-api-access-5vlnd\") pod \"4bb06cf0-f055-46d1-81f2-81aea3dc5cdd\" (UID: \"4bb06cf0-f055-46d1-81f2-81aea3dc5cdd\") "
Nov 02 03:16:22.532113 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:22.532082    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bb06cf0-f055-46d1-81f2-81aea3dc5cdd-maas-api-openshift-ca" pod "4bb06cf0-f055-46d1-81f2-81aea3dc5cdd" (UID: "4bb06cf0-f055-46d1-81f2-81aea3dc5cdd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:16:22.532206 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:22.532190    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bb06cf0-f055-46d1-81f2-81aea3dc5cdd-maas-api-openshift-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:16:22.533791 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:22.533767    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bb06cf0-f055-46d1-81f2-81aea3dc5cdd-kube-api-access-5vlnd" pod "4bb06cf0-f055-46d1-81f2-81aea3dc5cdd" (UID: "4bb06cf0-f055-46d1-81f2-81aea3dc5cdd"). InnerVolumeSpecName "kube-api-access-5vlnd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:16:22.533863 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:22.533817    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4bb06cf0-f055-46d1-81f2-81aea3dc5cdd-maas-api-xks-ca" pod "4bb06cf0-f055-46d1-81f2-81aea3dc5cdd" (UID: "4bb06cf0-f055-46d1-81f2-81aea3dc5cdd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 03:16:22.633630 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:22.633536    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a19741d-1644-42f2-a560-b14b5de2dcf9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a19741d-1644-42f2-a560-b14b5de2dcf9-maas-api-openshift-ca\") pod \"2a19741d-1644-42f2-a560-b14b5de2dcf9\" (UID: \"2a19741d-1644-42f2-a560-b14b5de2dcf9\") "
Nov 02 03:16:22.633630 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:22.633618    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a19741d-1644-42f2-a560-b14b5de2dcf9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a19741d-1644-42f2-a560-b14b5de2dcf9-maas-api-xks-ca\") pod \"2a19741d-1644-42f2-a560-b14b5de2dcf9\" (UID: \"2a19741d-1644-42f2-a560-b14b5de2dcf9\") "
Nov 02 03:16:22.633842 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:22.633669    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a19741d-1644-42f2-a560-b14b5de2dcf9-kube-api-access-hbzw8\" (UniqueName: \"kubernetes.io/projected/2a19741d-1644-42f2-a560-b14b5de2dcf9-kube-api-access-hbzw8\") pod \"2a19741d-1644-42f2-a560-b14b5de2dcf9\" (UID: \"2a19741d-1644-42f2-a560-b14b5de2dcf9\") "
Nov 02 03:16:22.634009 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:22.633977    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a19741d-1644-42f2-a560-b14b5de2dcf9-maas-api-openshift-ca" pod "2a19741d-1644-42f2-a560-b14b5de2dcf9" (UID: "2a19741d-1644-42f2-a560-b14b5de2dcf9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:16:22.634465 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:22.634440    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a19741d-1644-42f2-a560-b14b5de2dcf9-maas-api-openshift-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:16:22.634560 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:22.634469    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vlnd\" (UniqueName: \"kubernetes.io/projected/4bb06cf0-f055-46d1-81f2-81aea3dc5cdd-kube-api-access-5vlnd\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:16:22.634560 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:22.634484    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bb06cf0-f055-46d1-81f2-81aea3dc5cdd-maas-api-xks-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:16:22.636349 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:22.636320    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a19741d-1644-42f2-a560-b14b5de2dcf9-maas-api-xks-ca" pod "2a19741d-1644-42f2-a560-b14b5de2dcf9" (UID: "2a19741d-1644-42f2-a560-b14b5de2dcf9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 03:16:22.636531 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:22.636507    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a19741d-1644-42f2-a560-b14b5de2dcf9-kube-api-access-hbzw8" pod "2a19741d-1644-42f2-a560-b14b5de2dcf9" (UID: "2a19741d-1644-42f2-a560-b14b5de2dcf9"). InnerVolumeSpecName "kube-api-access-hbzw8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:16:22.735459 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:22.735419    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a19741d-1644-42f2-a560-b14b5de2dcf9-maas-api-xks-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:16:22.735459 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:22.735458    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbzw8\" (UniqueName: \"kubernetes.io/projected/2a19741d-1644-42f2-a560-b14b5de2dcf9-kube-api-access-hbzw8\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:16:23.316562 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:23.316534    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4209a816-mtqkz"
Nov 02 03:16:23.316562 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:23.316548    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4209a816-mtqkz" event={"ID":"4bb06cf0-f055-46d1-81f2-81aea3dc5cdd","Type":"ContainerDied","Data":"7e34a1c6c4f1fdfb8c3ba8bfe454d0c20b94f35f5ca357724a3e5a4b60f7fef6"}
Nov 02 03:16:23.317030 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:23.316603    4496 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7e34a1c6c4f1fdfb8c3ba8bfe454d0c20b94f35f5ca357724a3e5a4b60f7fef6"
Nov 02 03:16:23.318145 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:23.318121    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a22e3d1e-vfjwn" event={"ID":"2a19741d-1644-42f2-a560-b14b5de2dcf9","Type":"ContainerDied","Data":"547c3b0f4a94a538b347a8bebb4ae510c87b149f63fe30d62d168fbb8cf67609"}
Nov 02 03:16:23.318145 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:23.318136    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a22e3d1e-vfjwn"
Nov 02 03:16:23.318317 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:23.318160    4496 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="547c3b0f4a94a538b347a8bebb4ae510c87b149f63fe30d62d168fbb8cf67609"
Nov 02 03:16:23.574389 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:23.574303    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d90f14a-7e19-49df-8da8-ed96e12d2b96" path="/var/lib/kubelet/pods/3d90f14a-7e19-49df-8da8-ed96e12d2b96/volumes"
Nov 02 03:16:23.574814 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:23.574791    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="775f32d1-c694-4cdc-8a83-d82930ffaf37" path="/var/lib/kubelet/pods/775f32d1-c694-4cdc-8a83-d82930ffaf37/volumes"
Nov 02 03:16:29.413349 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:29.413317    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d71d8e5-rscj2"]
Nov 02 03:16:29.413920 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:29.413905    4496 cpu_manager.go:414] "containerMap: removing container" podUID="2a19741d-1644-42f2-a560-b14b5de2dcf9" containerName="revoke-keys"
Nov 02 03:16:29.413969 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:29.413921    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a19741d-1644-42f2-a560-b14b5de2dcf9" containerName="revoke-keys"
Nov 02 03:16:29.413969 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:29.413943    4496 cpu_manager.go:414] "containerMap: removing container" podUID="775f32d1-c694-4cdc-8a83-d82930ffaf37" containerName="payload-processing"
Nov 02 03:16:29.413969 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:29.413948    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="775f32d1-c694-4cdc-8a83-d82930ffaf37" containerName="payload-processing"
Nov 02 03:16:29.413969 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:29.413964    4496 cpu_manager.go:414] "containerMap: removing container" podUID="3d90f14a-7e19-49df-8da8-ed96e12d2b96" containerName="payload-processing"
Nov 02 03:16:29.413969 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:29.413969    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d90f14a-7e19-49df-8da8-ed96e12d2b96" containerName="payload-processing"
Nov 02 03:16:29.414152 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:29.413992    4496 cpu_manager.go:414] "containerMap: removing container" podUID="4bb06cf0-f055-46d1-81f2-81aea3dc5cdd" containerName="revoke-keys"
Nov 02 03:16:29.414152 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:29.413997    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bb06cf0-f055-46d1-81f2-81aea3dc5cdd" containerName="revoke-keys"
Nov 02 03:16:29.414152 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:29.414119    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bb06cf0-f055-46d1-81f2-81aea3dc5cdd" containerName="revoke-keys"
Nov 02 03:16:29.414152 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:29.414132    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a19741d-1644-42f2-a560-b14b5de2dcf9" containerName="revoke-keys"
Nov 02 03:16:29.414152 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:29.414140    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="775f32d1-c694-4cdc-8a83-d82930ffaf37" containerName="payload-processing"
Nov 02 03:16:29.414152 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:29.414148    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d90f14a-7e19-49df-8da8-ed96e12d2b96" containerName="payload-processing"
Nov 02 03:16:29.418756 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:29.418739    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d71d8e5-rscj2"
Nov 02 03:16:29.437155 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:29.437118    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfgs8\" (UniqueName: \"kubernetes.io/projected/71f8d674-60a8-4b1f-85c7-80f97b47623e-kube-api-access-cfgs8\") pod \"maas-api-revoke-sub-2d71d8e5-rscj2\" (UID: \"71f8d674-60a8-4b1f-85c7-80f97b47623e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d71d8e5-rscj2"
Nov 02 03:16:29.437267 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:29.437160    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71f8d674-60a8-4b1f-85c7-80f97b47623e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d71d8e5-rscj2\" (UID: \"71f8d674-60a8-4b1f-85c7-80f97b47623e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d71d8e5-rscj2"
Nov 02 03:16:29.437387 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:29.437369    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71f8d674-60a8-4b1f-85c7-80f97b47623e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d71d8e5-rscj2\" (UID: \"71f8d674-60a8-4b1f-85c7-80f97b47623e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d71d8e5-rscj2"
Nov 02 03:16:29.474503 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:29.474473    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d71d8e5-rscj2"]
Nov 02 03:16:29.519900 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:29.519874    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 03:16:29.519900 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:29.519890    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 03:16:29.520113 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:29.519902    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 03:16:29.522404 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:29.522388    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9z5bk\""
Nov 02 03:16:29.538985 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:29.538955    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71f8d674-60a8-4b1f-85c7-80f97b47623e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d71d8e5-rscj2\" (UID: \"71f8d674-60a8-4b1f-85c7-80f97b47623e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d71d8e5-rscj2"
Nov 02 03:16:29.539135 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:29.539067    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfgs8\" (UniqueName: \"kubernetes.io/projected/71f8d674-60a8-4b1f-85c7-80f97b47623e-kube-api-access-cfgs8\") pod \"maas-api-revoke-sub-2d71d8e5-rscj2\" (UID: \"71f8d674-60a8-4b1f-85c7-80f97b47623e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d71d8e5-rscj2"
Nov 02 03:16:29.539135 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:29.539095    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71f8d674-60a8-4b1f-85c7-80f97b47623e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d71d8e5-rscj2\" (UID: \"71f8d674-60a8-4b1f-85c7-80f97b47623e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d71d8e5-rscj2"
Nov 02 03:16:29.539645 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:29.539624    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71f8d674-60a8-4b1f-85c7-80f97b47623e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d71d8e5-rscj2\" (UID: \"71f8d674-60a8-4b1f-85c7-80f97b47623e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d71d8e5-rscj2"
Nov 02 03:16:29.541699 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:29.541674    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71f8d674-60a8-4b1f-85c7-80f97b47623e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d71d8e5-rscj2\" (UID: \"71f8d674-60a8-4b1f-85c7-80f97b47623e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d71d8e5-rscj2"
Nov 02 03:16:29.596781 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:29.596747    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfgs8\" (UniqueName: \"kubernetes.io/projected/71f8d674-60a8-4b1f-85c7-80f97b47623e-kube-api-access-cfgs8\") pod \"maas-api-revoke-sub-2d71d8e5-rscj2\" (UID: \"71f8d674-60a8-4b1f-85c7-80f97b47623e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d71d8e5-rscj2"
Nov 02 03:16:29.731793 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:29.731766    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d71d8e5-rscj2"
Nov 02 03:16:30.116609 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:16:30.116572    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71f8d674_60a8_4b1f_85c7_80f97b47623e.slice/crio-90f900e656b4766ebe60cb2103bd6edd38583fbf39948604aa8a52e3e1ce1063 WatchSource:0}: Error finding container 90f900e656b4766ebe60cb2103bd6edd38583fbf39948604aa8a52e3e1ce1063: Status 404 returned error can't find the container with id 90f900e656b4766ebe60cb2103bd6edd38583fbf39948604aa8a52e3e1ce1063
Nov 02 03:16:30.119273 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:30.119248    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d71d8e5-rscj2"]
Nov 02 03:16:30.405005 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:30.404964    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d71d8e5-rscj2" event={"ID":"71f8d674-60a8-4b1f-85c7-80f97b47623e","Type":"ContainerStarted","Data":"90f900e656b4766ebe60cb2103bd6edd38583fbf39948604aa8a52e3e1ce1063"}
Nov 02 03:16:31.415185 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:31.415101    4496 generic.go:356] "Generic (PLEG): container finished" podID="71f8d674-60a8-4b1f-85c7-80f97b47623e" containerID="f7128300b8d451451498b3dfe1fca4de99070c3e2371d412ba2bbee2c21530cf" exitCode=0
Nov 02 03:16:31.415557 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:31.415179    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d71d8e5-rscj2" event={"ID":"71f8d674-60a8-4b1f-85c7-80f97b47623e","Type":"ContainerDied","Data":"f7128300b8d451451498b3dfe1fca4de99070c3e2371d412ba2bbee2c21530cf"}
Nov 02 03:16:32.580548 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:32.580519    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d71d8e5-rscj2"
Nov 02 03:16:32.683880 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:32.683808    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71f8d674-60a8-4b1f-85c7-80f97b47623e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71f8d674-60a8-4b1f-85c7-80f97b47623e-maas-api-xks-ca\") pod \"71f8d674-60a8-4b1f-85c7-80f97b47623e\" (UID: \"71f8d674-60a8-4b1f-85c7-80f97b47623e\") "
Nov 02 03:16:32.684051 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:32.683904    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71f8d674-60a8-4b1f-85c7-80f97b47623e-kube-api-access-cfgs8\" (UniqueName: \"kubernetes.io/projected/71f8d674-60a8-4b1f-85c7-80f97b47623e-kube-api-access-cfgs8\") pod \"71f8d674-60a8-4b1f-85c7-80f97b47623e\" (UID: \"71f8d674-60a8-4b1f-85c7-80f97b47623e\") "
Nov 02 03:16:32.684051 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:32.683928    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71f8d674-60a8-4b1f-85c7-80f97b47623e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71f8d674-60a8-4b1f-85c7-80f97b47623e-maas-api-openshift-ca\") pod \"71f8d674-60a8-4b1f-85c7-80f97b47623e\" (UID: \"71f8d674-60a8-4b1f-85c7-80f97b47623e\") "
Nov 02 03:16:32.684356 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:32.684315    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71f8d674-60a8-4b1f-85c7-80f97b47623e-maas-api-openshift-ca" pod "71f8d674-60a8-4b1f-85c7-80f97b47623e" (UID: "71f8d674-60a8-4b1f-85c7-80f97b47623e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:16:32.684476 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:32.684458    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71f8d674-60a8-4b1f-85c7-80f97b47623e-maas-api-openshift-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:16:32.685869 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:32.685839    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71f8d674-60a8-4b1f-85c7-80f97b47623e-maas-api-xks-ca" pod "71f8d674-60a8-4b1f-85c7-80f97b47623e" (UID: "71f8d674-60a8-4b1f-85c7-80f97b47623e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 03:16:32.685983 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:32.685956    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71f8d674-60a8-4b1f-85c7-80f97b47623e-kube-api-access-cfgs8" pod "71f8d674-60a8-4b1f-85c7-80f97b47623e" (UID: "71f8d674-60a8-4b1f-85c7-80f97b47623e"). InnerVolumeSpecName "kube-api-access-cfgs8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:16:32.785864 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:32.785836    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71f8d674-60a8-4b1f-85c7-80f97b47623e-maas-api-xks-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:16:32.785864 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:32.785862    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfgs8\" (UniqueName: \"kubernetes.io/projected/71f8d674-60a8-4b1f-85c7-80f97b47623e-kube-api-access-cfgs8\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:16:33.430556 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:33.430529    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d71d8e5-rscj2"
Nov 02 03:16:33.430756 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:33.430528    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d71d8e5-rscj2" event={"ID":"71f8d674-60a8-4b1f-85c7-80f97b47623e","Type":"ContainerDied","Data":"90f900e656b4766ebe60cb2103bd6edd38583fbf39948604aa8a52e3e1ce1063"}
Nov 02 03:16:33.430756 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:33.430652    4496 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="90f900e656b4766ebe60cb2103bd6edd38583fbf39948604aa8a52e3e1ce1063"
Nov 02 03:16:33.526584 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:16:33.526548    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 03:16:34.973126 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:34.973090    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94a98249-75mhd"]
Nov 02 03:16:34.973810 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:34.973791    4496 cpu_manager.go:414] "containerMap: removing container" podUID="71f8d674-60a8-4b1f-85c7-80f97b47623e" containerName="revoke-keys"
Nov 02 03:16:34.973810 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:34.973809    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71f8d674-60a8-4b1f-85c7-80f97b47623e" containerName="revoke-keys"
Nov 02 03:16:34.973960 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:34.973945    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="71f8d674-60a8-4b1f-85c7-80f97b47623e" containerName="revoke-keys"
Nov 02 03:16:34.979146 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:34.979119    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94a98249-75mhd"
Nov 02 03:16:35.034604 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:35.034567    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94a98249-75mhd"]
Nov 02 03:16:35.080513 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:35.080479    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 03:16:35.080513 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:35.080478    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 03:16:35.080718 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:35.080523    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 03:16:35.080718 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:35.080619    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9z5bk\""
Nov 02 03:16:35.114329 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:35.114302    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3261337-656c-4a60-8aa9-46d16545bb2e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-94a98249-75mhd\" (UID: \"f3261337-656c-4a60-8aa9-46d16545bb2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94a98249-75mhd"
Nov 02 03:16:35.114490 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:35.114374    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sj8f\" (UniqueName: \"kubernetes.io/projected/f3261337-656c-4a60-8aa9-46d16545bb2e-kube-api-access-6sj8f\") pod \"maas-api-revoke-sub-94a98249-75mhd\" (UID: \"f3261337-656c-4a60-8aa9-46d16545bb2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94a98249-75mhd"
Nov 02 03:16:35.114490 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:35.114463    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3261337-656c-4a60-8aa9-46d16545bb2e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-94a98249-75mhd\" (UID: \"f3261337-656c-4a60-8aa9-46d16545bb2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94a98249-75mhd"
Nov 02 03:16:35.215769 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:35.215737    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3261337-656c-4a60-8aa9-46d16545bb2e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-94a98249-75mhd\" (UID: \"f3261337-656c-4a60-8aa9-46d16545bb2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94a98249-75mhd"
Nov 02 03:16:35.215961 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:35.215806    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sj8f\" (UniqueName: \"kubernetes.io/projected/f3261337-656c-4a60-8aa9-46d16545bb2e-kube-api-access-6sj8f\") pod \"maas-api-revoke-sub-94a98249-75mhd\" (UID: \"f3261337-656c-4a60-8aa9-46d16545bb2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94a98249-75mhd"
Nov 02 03:16:35.215961 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:35.215833    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3261337-656c-4a60-8aa9-46d16545bb2e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-94a98249-75mhd\" (UID: \"f3261337-656c-4a60-8aa9-46d16545bb2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94a98249-75mhd"
Nov 02 03:16:35.216497 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:35.216469    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3261337-656c-4a60-8aa9-46d16545bb2e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-94a98249-75mhd\" (UID: \"f3261337-656c-4a60-8aa9-46d16545bb2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94a98249-75mhd"
Nov 02 03:16:35.218189 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:35.218169    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3261337-656c-4a60-8aa9-46d16545bb2e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-94a98249-75mhd\" (UID: \"f3261337-656c-4a60-8aa9-46d16545bb2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94a98249-75mhd"
Nov 02 03:16:35.272980 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:35.272943    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sj8f\" (UniqueName: \"kubernetes.io/projected/f3261337-656c-4a60-8aa9-46d16545bb2e-kube-api-access-6sj8f\") pod \"maas-api-revoke-sub-94a98249-75mhd\" (UID: \"f3261337-656c-4a60-8aa9-46d16545bb2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94a98249-75mhd"
Nov 02 03:16:35.293935 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:35.293905    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94a98249-75mhd"
Nov 02 03:16:35.677414 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:16:35.677370    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3261337_656c_4a60_8aa9_46d16545bb2e.slice/crio-51a5824f71f597e9aa2e967360bbe9023afc65be52b96788b97173adfe223072 WatchSource:0}: Error finding container 51a5824f71f597e9aa2e967360bbe9023afc65be52b96788b97173adfe223072: Status 404 returned error can't find the container with id 51a5824f71f597e9aa2e967360bbe9023afc65be52b96788b97173adfe223072
Nov 02 03:16:35.677662 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:35.677643    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94a98249-75mhd"]
Nov 02 03:16:36.456084 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:36.456041    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94a98249-75mhd" event={"ID":"f3261337-656c-4a60-8aa9-46d16545bb2e","Type":"ContainerStarted","Data":"dbb18c9e28037021d0458cdd0e34d2cfda0714f564ccadea9cd81f2172435be3"}
Nov 02 03:16:36.456518 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:36.456096    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94a98249-75mhd" event={"ID":"f3261337-656c-4a60-8aa9-46d16545bb2e","Type":"ContainerStarted","Data":"51a5824f71f597e9aa2e967360bbe9023afc65be52b96788b97173adfe223072"}
Nov 02 03:16:36.573377 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:36.573328    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94a98249-75mhd" podStartSLOduration=2.573314983 podStartE2EDuration="2.573314983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:16:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:16:36.57064164 +0000 UTC m=+1737.149249031" watchObservedRunningTime="2026-11-02 03:16:36.573314983 +0000 UTC m=+1737.151922362"
Nov 02 03:16:37.072672 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:37.072640    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xstsp"]
Nov 02 03:16:37.077529 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:37.077504    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xstsp"
Nov 02 03:16:37.142353 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:37.138610    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0031b60a-f888-4c85-8999-07bddc61a513-credential-socket\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xstsp\" (UID: \"0031b60a-f888-4c85-8999-07bddc61a513\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xstsp"
Nov 02 03:16:37.142353 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:37.138661    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0031b60a-f888-4c85-8999-07bddc61a513-istio-token\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xstsp\" (UID: \"0031b60a-f888-4c85-8999-07bddc61a513\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xstsp"
Nov 02 03:16:37.142353 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:37.138706    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0031b60a-f888-4c85-8999-07bddc61a513-istiod-ca-cert\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xstsp\" (UID: \"0031b60a-f888-4c85-8999-07bddc61a513\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xstsp"
Nov 02 03:16:37.142353 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:37.138756    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0031b60a-f888-4c85-8999-07bddc61a513-workload-socket\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xstsp\" (UID: \"0031b60a-f888-4c85-8999-07bddc61a513\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xstsp"
Nov 02 03:16:37.142353 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:37.138835    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0031b60a-f888-4c85-8999-07bddc61a513-istio-data\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xstsp\" (UID: \"0031b60a-f888-4c85-8999-07bddc61a513\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xstsp"
Nov 02 03:16:37.142353 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:37.138868    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0031b60a-f888-4c85-8999-07bddc61a513-workload-certs\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xstsp\" (UID: \"0031b60a-f888-4c85-8999-07bddc61a513\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xstsp"
Nov 02 03:16:37.142353 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:37.138964    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0031b60a-f888-4c85-8999-07bddc61a513-istio-podinfo\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xstsp\" (UID: \"0031b60a-f888-4c85-8999-07bddc61a513\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xstsp"
Nov 02 03:16:37.142353 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:37.138996    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0031b60a-f888-4c85-8999-07bddc61a513-istio-envoy\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xstsp\" (UID: \"0031b60a-f888-4c85-8999-07bddc61a513\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xstsp"
Nov 02 03:16:37.142353 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:37.139081    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ltmb\" (UniqueName: \"kubernetes.io/projected/0031b60a-f888-4c85-8999-07bddc61a513-kube-api-access-2ltmb\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xstsp\" (UID: \"0031b60a-f888-4c85-8999-07bddc61a513\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xstsp"
Nov 02 03:16:37.170701 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:37.170662    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-ca6546/e2e-conflict-a-5eca459f-kserve-5744ffc845-xgm78"]
Nov 02 03:16:37.179775 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:37.179737    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xstsp"]
Nov 02 03:16:37.240261 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:37.240228    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0031b60a-f888-4c85-8999-07bddc61a513-istio-data\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xstsp\" (UID: \"0031b60a-f888-4c85-8999-07bddc61a513\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xstsp"
Nov 02 03:16:37.240261 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:37.240267    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0031b60a-f888-4c85-8999-07bddc61a513-workload-certs\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xstsp\" (UID: \"0031b60a-f888-4c85-8999-07bddc61a513\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xstsp"
Nov 02 03:16:37.240510 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:37.240335    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0031b60a-f888-4c85-8999-07bddc61a513-istio-podinfo\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xstsp\" (UID: \"0031b60a-f888-4c85-8999-07bddc61a513\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xstsp"
Nov 02 03:16:37.240510 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:37.240359    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0031b60a-f888-4c85-8999-07bddc61a513-istio-envoy\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xstsp\" (UID: \"0031b60a-f888-4c85-8999-07bddc61a513\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xstsp"
Nov 02 03:16:37.240510 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:37.240425    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ltmb\" (UniqueName: \"kubernetes.io/projected/0031b60a-f888-4c85-8999-07bddc61a513-kube-api-access-2ltmb\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xstsp\" (UID: \"0031b60a-f888-4c85-8999-07bddc61a513\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xstsp"
Nov 02 03:16:37.240510 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:37.240499    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0031b60a-f888-4c85-8999-07bddc61a513-credential-socket\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xstsp\" (UID: \"0031b60a-f888-4c85-8999-07bddc61a513\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xstsp"
Nov 02 03:16:37.240718 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:37.240534    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0031b60a-f888-4c85-8999-07bddc61a513-istio-token\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xstsp\" (UID: \"0031b60a-f888-4c85-8999-07bddc61a513\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xstsp"
Nov 02 03:16:37.240718 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:37.240590    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0031b60a-f888-4c85-8999-07bddc61a513-istiod-ca-cert\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xstsp\" (UID: \"0031b60a-f888-4c85-8999-07bddc61a513\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xstsp"
Nov 02 03:16:37.240834 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:37.240712    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0031b60a-f888-4c85-8999-07bddc61a513-istio-data\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xstsp\" (UID: \"0031b60a-f888-4c85-8999-07bddc61a513\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xstsp"
Nov 02 03:16:37.240834 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:37.240719    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0031b60a-f888-4c85-8999-07bddc61a513-workload-socket\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xstsp\" (UID: \"0031b60a-f888-4c85-8999-07bddc61a513\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xstsp"
Nov 02 03:16:37.241139 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:37.241122    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0031b60a-f888-4c85-8999-07bddc61a513-workload-socket\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xstsp\" (UID: \"0031b60a-f888-4c85-8999-07bddc61a513\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xstsp"
Nov 02 03:16:37.241341 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:37.241316    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0031b60a-f888-4c85-8999-07bddc61a513-credential-socket\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xstsp\" (UID: \"0031b60a-f888-4c85-8999-07bddc61a513\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xstsp"
Nov 02 03:16:37.241568 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:37.241544    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0031b60a-f888-4c85-8999-07bddc61a513-workload-certs\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xstsp\" (UID: \"0031b60a-f888-4c85-8999-07bddc61a513\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xstsp"
Nov 02 03:16:37.242393 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:37.242366    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0031b60a-f888-4c85-8999-07bddc61a513-istiod-ca-cert\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xstsp\" (UID: \"0031b60a-f888-4c85-8999-07bddc61a513\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xstsp"
Nov 02 03:16:37.243779 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:37.243753    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0031b60a-f888-4c85-8999-07bddc61a513-istio-envoy\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xstsp\" (UID: \"0031b60a-f888-4c85-8999-07bddc61a513\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xstsp"
Nov 02 03:16:37.245691 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:37.245674    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0031b60a-f888-4c85-8999-07bddc61a513-istio-podinfo\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xstsp\" (UID: \"0031b60a-f888-4c85-8999-07bddc61a513\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xstsp"
Nov 02 03:16:37.298967 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:37.298935    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0031b60a-f888-4c85-8999-07bddc61a513-istio-token\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xstsp\" (UID: \"0031b60a-f888-4c85-8999-07bddc61a513\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xstsp"
Nov 02 03:16:37.299174 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:37.298985    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ltmb\" (UniqueName: \"kubernetes.io/projected/0031b60a-f888-4c85-8999-07bddc61a513-kube-api-access-2ltmb\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xstsp\" (UID: \"0031b60a-f888-4c85-8999-07bddc61a513\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xstsp"
Nov 02 03:16:37.397530 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:37.397446    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xstsp"
Nov 02 03:16:37.468999 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:37.468970    4496 generic.go:356] "Generic (PLEG): container finished" podID="f3261337-656c-4a60-8aa9-46d16545bb2e" containerID="dbb18c9e28037021d0458cdd0e34d2cfda0714f564ccadea9cd81f2172435be3" exitCode=0
Nov 02 03:16:37.469405 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:37.469122    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94a98249-75mhd" event={"ID":"f3261337-656c-4a60-8aa9-46d16545bb2e","Type":"ContainerDied","Data":"dbb18c9e28037021d0458cdd0e34d2cfda0714f564ccadea9cd81f2172435be3"}
Nov 02 03:16:37.830951 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:16:37.830923    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0031b60a_f888_4c85_8999_07bddc61a513.slice/crio-e3c56b1411f6453bf70fecc9dfd7bd8414a73414691a4308a3ca62970c04cd74 WatchSource:0}: Error finding container e3c56b1411f6453bf70fecc9dfd7bd8414a73414691a4308a3ca62970c04cd74: Status 404 returned error can't find the container with id e3c56b1411f6453bf70fecc9dfd7bd8414a73414691a4308a3ca62970c04cd74
Nov 02 03:16:37.831337 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:37.831312    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xstsp"]
Nov 02 03:16:37.833731 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:37.833703    4496 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 03:16:37.833795 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:37.833764    4496 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 03:16:37.833830 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:37.833795    4496 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 03:16:38.478647 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:38.478606    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xstsp" event={"ID":"0031b60a-f888-4c85-8999-07bddc61a513","Type":"ContainerStarted","Data":"a0fbd673106ccac658e925ad94554076f70660e4f737063374097d2828d37ad8"}
Nov 02 03:16:38.479110 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:38.478673    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xstsp" event={"ID":"0031b60a-f888-4c85-8999-07bddc61a513","Type":"ContainerStarted","Data":"e3c56b1411f6453bf70fecc9dfd7bd8414a73414691a4308a3ca62970c04cd74"}
Nov 02 03:16:38.596612 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:38.596559    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xstsp" podStartSLOduration=1.596544621 podStartE2EDuration="1.596544621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:16:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:16:38.59547498 +0000 UTC m=+1739.174082360" watchObservedRunningTime="2026-11-02 03:16:38.596544621 +0000 UTC m=+1739.175152000"
Nov 02 03:16:38.709036 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:38.703320    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94a98249-75mhd"
Nov 02 03:16:38.868123 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:38.866477    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3261337-656c-4a60-8aa9-46d16545bb2e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3261337-656c-4a60-8aa9-46d16545bb2e-maas-api-openshift-ca\") pod \"f3261337-656c-4a60-8aa9-46d16545bb2e\" (UID: \"f3261337-656c-4a60-8aa9-46d16545bb2e\") "
Nov 02 03:16:38.868123 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:38.866586    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3261337-656c-4a60-8aa9-46d16545bb2e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3261337-656c-4a60-8aa9-46d16545bb2e-maas-api-xks-ca\") pod \"f3261337-656c-4a60-8aa9-46d16545bb2e\" (UID: \"f3261337-656c-4a60-8aa9-46d16545bb2e\") "
Nov 02 03:16:38.868123 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:38.866670    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3261337-656c-4a60-8aa9-46d16545bb2e-kube-api-access-6sj8f\" (UniqueName: \"kubernetes.io/projected/f3261337-656c-4a60-8aa9-46d16545bb2e-kube-api-access-6sj8f\") pod \"f3261337-656c-4a60-8aa9-46d16545bb2e\" (UID: \"f3261337-656c-4a60-8aa9-46d16545bb2e\") "
Nov 02 03:16:38.870486 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:38.870451    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3261337-656c-4a60-8aa9-46d16545bb2e-kube-api-access-6sj8f" pod "f3261337-656c-4a60-8aa9-46d16545bb2e" (UID: "f3261337-656c-4a60-8aa9-46d16545bb2e"). InnerVolumeSpecName "kube-api-access-6sj8f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:16:38.870890 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:38.870866    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3261337-656c-4a60-8aa9-46d16545bb2e-maas-api-openshift-ca" pod "f3261337-656c-4a60-8aa9-46d16545bb2e" (UID: "f3261337-656c-4a60-8aa9-46d16545bb2e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:16:38.873178 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:38.873132    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3261337-656c-4a60-8aa9-46d16545bb2e-maas-api-xks-ca" pod "f3261337-656c-4a60-8aa9-46d16545bb2e" (UID: "f3261337-656c-4a60-8aa9-46d16545bb2e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 03:16:38.968570 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:38.968530    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3261337-656c-4a60-8aa9-46d16545bb2e-maas-api-xks-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:16:38.968570 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:38.968570    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6sj8f\" (UniqueName: \"kubernetes.io/projected/f3261337-656c-4a60-8aa9-46d16545bb2e-kube-api-access-6sj8f\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:16:38.968782 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:38.968588    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3261337-656c-4a60-8aa9-46d16545bb2e-maas-api-openshift-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:16:39.397961 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:39.397922    4496 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xstsp"
Nov 02 03:16:39.399437 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:39.399412    4496 patch_prober.go:29] interesting pod/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xstsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body=
Nov 02 03:16:39.399572 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:39.399461    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xstsp" podUID="0031b60a-f888-4c85-8999-07bddc61a513" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused"
Nov 02 03:16:39.488490 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:39.488459    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94a98249-75mhd"
Nov 02 03:16:39.488943 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:39.488508    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94a98249-75mhd" event={"ID":"f3261337-656c-4a60-8aa9-46d16545bb2e","Type":"ContainerDied","Data":"51a5824f71f597e9aa2e967360bbe9023afc65be52b96788b97173adfe223072"}
Nov 02 03:16:39.488943 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:39.488577    4496 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="51a5824f71f597e9aa2e967360bbe9023afc65be52b96788b97173adfe223072"
Nov 02 03:16:40.398158 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:40.398074    4496 patch_prober.go:29] interesting pod/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xstsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body=
Nov 02 03:16:40.398158 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:40.398134    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xstsp" podUID="0031b60a-f888-4c85-8999-07bddc61a513" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused"
Nov 02 03:16:41.398304 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:41.398265    4496 patch_prober.go:29] interesting pod/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xstsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body=
Nov 02 03:16:41.398736 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:41.398342    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xstsp" podUID="0031b60a-f888-4c85-8999-07bddc61a513" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused"
Nov 02 03:16:42.398074 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:42.398034    4496 patch_prober.go:29] interesting pod/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xstsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body=
Nov 02 03:16:42.398247 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:42.398103    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xstsp" podUID="0031b60a-f888-4c85-8999-07bddc61a513" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused"
Nov 02 03:16:43.398851 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:43.398809    4496 patch_prober.go:29] interesting pod/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xstsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body=
Nov 02 03:16:43.399324 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:43.398869    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xstsp" podUID="0031b60a-f888-4c85-8999-07bddc61a513" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused"
Nov 02 03:16:44.398623 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:44.398588    4496 patch_prober.go:29] interesting pod/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xstsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body=
Nov 02 03:16:44.398804 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:44.398642    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xstsp" podUID="0031b60a-f888-4c85-8999-07bddc61a513" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused"
Nov 02 03:16:45.401660 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:45.401628    4496 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xstsp"
Nov 02 03:16:45.402163 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:45.402122    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xstsp"
Nov 02 03:16:45.402837 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:45.402814    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xstsp"
Nov 02 03:16:45.683054 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:45.682948    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c249c442-t79d4"]
Nov 02 03:16:45.683558 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:45.683543    4496 cpu_manager.go:414] "containerMap: removing container" podUID="f3261337-656c-4a60-8aa9-46d16545bb2e" containerName="revoke-keys"
Nov 02 03:16:45.683604 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:45.683561    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3261337-656c-4a60-8aa9-46d16545bb2e" containerName="revoke-keys"
Nov 02 03:16:45.683711 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:45.683701    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3261337-656c-4a60-8aa9-46d16545bb2e" containerName="revoke-keys"
Nov 02 03:16:45.688953 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:45.688930    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c249c442-t79d4"
Nov 02 03:16:45.745914 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:45.745870    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9z5bk\""
Nov 02 03:16:45.745914 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:45.745932    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 03:16:45.746195 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:45.745958    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 03:16:45.746195 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:45.746079    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 03:16:45.751244 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:45.751219    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c249c442-t79d4"]
Nov 02 03:16:45.849331 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:45.849295    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95533a1b-befb-4b8c-89cb-ee3d53329c12-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c249c442-t79d4\" (UID: \"95533a1b-befb-4b8c-89cb-ee3d53329c12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c249c442-t79d4"
Nov 02 03:16:45.849523 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:45.849360    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95533a1b-befb-4b8c-89cb-ee3d53329c12-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c249c442-t79d4\" (UID: \"95533a1b-befb-4b8c-89cb-ee3d53329c12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c249c442-t79d4"
Nov 02 03:16:45.849523 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:45.849497    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zxj8\" (UniqueName: \"kubernetes.io/projected/95533a1b-befb-4b8c-89cb-ee3d53329c12-kube-api-access-2zxj8\") pod \"maas-api-revoke-sub-c249c442-t79d4\" (UID: \"95533a1b-befb-4b8c-89cb-ee3d53329c12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c249c442-t79d4"
Nov 02 03:16:45.951138 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:45.951060    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95533a1b-befb-4b8c-89cb-ee3d53329c12-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c249c442-t79d4\" (UID: \"95533a1b-befb-4b8c-89cb-ee3d53329c12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c249c442-t79d4"
Nov 02 03:16:45.951282 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:45.951152    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95533a1b-befb-4b8c-89cb-ee3d53329c12-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c249c442-t79d4\" (UID: \"95533a1b-befb-4b8c-89cb-ee3d53329c12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c249c442-t79d4"
Nov 02 03:16:45.951282 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:45.951227    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zxj8\" (UniqueName: \"kubernetes.io/projected/95533a1b-befb-4b8c-89cb-ee3d53329c12-kube-api-access-2zxj8\") pod \"maas-api-revoke-sub-c249c442-t79d4\" (UID: \"95533a1b-befb-4b8c-89cb-ee3d53329c12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c249c442-t79d4"
Nov 02 03:16:45.951754 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:45.951733    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95533a1b-befb-4b8c-89cb-ee3d53329c12-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c249c442-t79d4\" (UID: \"95533a1b-befb-4b8c-89cb-ee3d53329c12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c249c442-t79d4"
Nov 02 03:16:45.953536 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:45.953514    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95533a1b-befb-4b8c-89cb-ee3d53329c12-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c249c442-t79d4\" (UID: \"95533a1b-befb-4b8c-89cb-ee3d53329c12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c249c442-t79d4"
Nov 02 03:16:46.018068 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:46.018035    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zxj8\" (UniqueName: \"kubernetes.io/projected/95533a1b-befb-4b8c-89cb-ee3d53329c12-kube-api-access-2zxj8\") pod \"maas-api-revoke-sub-c249c442-t79d4\" (UID: \"95533a1b-befb-4b8c-89cb-ee3d53329c12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c249c442-t79d4"
Nov 02 03:16:46.305839 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:46.305804    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c249c442-t79d4"
Nov 02 03:16:46.693095 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:46.693064    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c249c442-t79d4"]
Nov 02 03:16:46.694817 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:16:46.694782    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95533a1b_befb_4b8c_89cb_ee3d53329c12.slice/crio-404bf197a00d822717749cd34a4974ef3854358604cf0f3f24154468baeb8c24 WatchSource:0}: Error finding container 404bf197a00d822717749cd34a4974ef3854358604cf0f3f24154468baeb8c24: Status 404 returned error can't find the container with id 404bf197a00d822717749cd34a4974ef3854358604cf0f3f24154468baeb8c24
Nov 02 03:16:47.634092 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:47.627369    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c249c442-t79d4" event={"ID":"95533a1b-befb-4b8c-89cb-ee3d53329c12","Type":"ContainerStarted","Data":"9eb93870a5f91da823f9159681d7487d6058e537f2de8d4818e4a86ce2ce80b1"}
Nov 02 03:16:47.634092 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:47.627434    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c249c442-t79d4" event={"ID":"95533a1b-befb-4b8c-89cb-ee3d53329c12","Type":"ContainerStarted","Data":"404bf197a00d822717749cd34a4974ef3854358604cf0f3f24154468baeb8c24"}
Nov 02 03:16:47.754041 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:47.745486    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c249c442-t79d4" podStartSLOduration=2.745458264 podStartE2EDuration="2.745458264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:16:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:16:47.74341185 +0000 UTC m=+1748.322019233" watchObservedRunningTime="2026-11-02 03:16:47.745458264 +0000 UTC m=+1748.324065646"
Nov 02 03:16:48.636830 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:48.636790    4496 generic.go:356] "Generic (PLEG): container finished" podID="95533a1b-befb-4b8c-89cb-ee3d53329c12" containerID="9eb93870a5f91da823f9159681d7487d6058e537f2de8d4818e4a86ce2ce80b1" exitCode=0
Nov 02 03:16:48.637046 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:48.636873    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c249c442-t79d4" event={"ID":"95533a1b-befb-4b8c-89cb-ee3d53329c12","Type":"ContainerDied","Data":"9eb93870a5f91da823f9159681d7487d6058e537f2de8d4818e4a86ce2ce80b1"}
Nov 02 03:16:49.793833 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:49.793792    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c249c442-t79d4"
Nov 02 03:16:49.847226 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:49.847201    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95533a1b-befb-4b8c-89cb-ee3d53329c12-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95533a1b-befb-4b8c-89cb-ee3d53329c12-maas-api-xks-ca\") pod \"95533a1b-befb-4b8c-89cb-ee3d53329c12\" (UID: \"95533a1b-befb-4b8c-89cb-ee3d53329c12\") "
Nov 02 03:16:49.847370 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:49.847271    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95533a1b-befb-4b8c-89cb-ee3d53329c12-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95533a1b-befb-4b8c-89cb-ee3d53329c12-maas-api-openshift-ca\") pod \"95533a1b-befb-4b8c-89cb-ee3d53329c12\" (UID: \"95533a1b-befb-4b8c-89cb-ee3d53329c12\") "
Nov 02 03:16:49.847370 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:49.847310    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95533a1b-befb-4b8c-89cb-ee3d53329c12-kube-api-access-2zxj8\" (UniqueName: \"kubernetes.io/projected/95533a1b-befb-4b8c-89cb-ee3d53329c12-kube-api-access-2zxj8\") pod \"95533a1b-befb-4b8c-89cb-ee3d53329c12\" (UID: \"95533a1b-befb-4b8c-89cb-ee3d53329c12\") "
Nov 02 03:16:49.847657 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:49.847632    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95533a1b-befb-4b8c-89cb-ee3d53329c12-maas-api-openshift-ca" pod "95533a1b-befb-4b8c-89cb-ee3d53329c12" (UID: "95533a1b-befb-4b8c-89cb-ee3d53329c12"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:16:49.847826 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:49.847809    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95533a1b-befb-4b8c-89cb-ee3d53329c12-maas-api-openshift-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:16:49.849429 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:49.849395    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95533a1b-befb-4b8c-89cb-ee3d53329c12-maas-api-xks-ca" pod "95533a1b-befb-4b8c-89cb-ee3d53329c12" (UID: "95533a1b-befb-4b8c-89cb-ee3d53329c12"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 03:16:49.849843 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:49.849822    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95533a1b-befb-4b8c-89cb-ee3d53329c12-kube-api-access-2zxj8" pod "95533a1b-befb-4b8c-89cb-ee3d53329c12" (UID: "95533a1b-befb-4b8c-89cb-ee3d53329c12"). InnerVolumeSpecName "kube-api-access-2zxj8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:16:49.949385 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:49.949295    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zxj8\" (UniqueName: \"kubernetes.io/projected/95533a1b-befb-4b8c-89cb-ee3d53329c12-kube-api-access-2zxj8\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:16:49.949385 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:49.949334    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95533a1b-befb-4b8c-89cb-ee3d53329c12-maas-api-xks-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:16:50.501579 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:50.501544    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-zf5k4"]
Nov 02 03:16:50.502269 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:50.502252    4496 cpu_manager.go:414] "containerMap: removing container" podUID="95533a1b-befb-4b8c-89cb-ee3d53329c12" containerName="revoke-keys"
Nov 02 03:16:50.502323 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:50.502270    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95533a1b-befb-4b8c-89cb-ee3d53329c12" containerName="revoke-keys"
Nov 02 03:16:50.502396 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:50.502386    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="95533a1b-befb-4b8c-89cb-ee3d53329c12" containerName="revoke-keys"
Nov 02 03:16:50.506055 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:50.506038    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-zf5k4"
Nov 02 03:16:50.556301 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:50.556268    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d695fa41-7976-4a64-b9e2-4642ab0ed15f-istio-data\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-zf5k4\" (UID: \"d695fa41-7976-4a64-b9e2-4642ab0ed15f\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-zf5k4"
Nov 02 03:16:50.556416 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:50.556316    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d695fa41-7976-4a64-b9e2-4642ab0ed15f-istiod-ca-cert\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-zf5k4\" (UID: \"d695fa41-7976-4a64-b9e2-4642ab0ed15f\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-zf5k4"
Nov 02 03:16:50.556461 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:50.556440    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d695fa41-7976-4a64-b9e2-4642ab0ed15f-credential-socket\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-zf5k4\" (UID: \"d695fa41-7976-4a64-b9e2-4642ab0ed15f\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-zf5k4"
Nov 02 03:16:50.556532 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:50.556513    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d695fa41-7976-4a64-b9e2-4642ab0ed15f-istio-podinfo\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-zf5k4\" (UID: \"d695fa41-7976-4a64-b9e2-4642ab0ed15f\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-zf5k4"
Nov 02 03:16:50.556566 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:50.556550    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d695fa41-7976-4a64-b9e2-4642ab0ed15f-istio-token\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-zf5k4\" (UID: \"d695fa41-7976-4a64-b9e2-4642ab0ed15f\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-zf5k4"
Nov 02 03:16:50.556620 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:50.556601    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d695fa41-7976-4a64-b9e2-4642ab0ed15f-workload-certs\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-zf5k4\" (UID: \"d695fa41-7976-4a64-b9e2-4642ab0ed15f\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-zf5k4"
Nov 02 03:16:50.556672 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:50.556646    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d695fa41-7976-4a64-b9e2-4642ab0ed15f-workload-socket\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-zf5k4\" (UID: \"d695fa41-7976-4a64-b9e2-4642ab0ed15f\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-zf5k4"
Nov 02 03:16:50.556742 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:50.556719    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d695fa41-7976-4a64-b9e2-4642ab0ed15f-istio-envoy\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-zf5k4\" (UID: \"d695fa41-7976-4a64-b9e2-4642ab0ed15f\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-zf5k4"
Nov 02 03:16:50.556791 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:50.556780    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2knn7\" (UniqueName: \"kubernetes.io/projected/d695fa41-7976-4a64-b9e2-4642ab0ed15f-kube-api-access-2knn7\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-zf5k4\" (UID: \"d695fa41-7976-4a64-b9e2-4642ab0ed15f\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-zf5k4"
Nov 02 03:16:50.654983 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:50.654952    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c249c442-t79d4"
Nov 02 03:16:50.655163 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:50.654993    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c249c442-t79d4" event={"ID":"95533a1b-befb-4b8c-89cb-ee3d53329c12","Type":"ContainerDied","Data":"404bf197a00d822717749cd34a4974ef3854358604cf0f3f24154468baeb8c24"}
Nov 02 03:16:50.655163 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:50.655075    4496 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="404bf197a00d822717749cd34a4974ef3854358604cf0f3f24154468baeb8c24"
Nov 02 03:16:50.657718 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:50.657687    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d695fa41-7976-4a64-b9e2-4642ab0ed15f-istio-podinfo\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-zf5k4\" (UID: \"d695fa41-7976-4a64-b9e2-4642ab0ed15f\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-zf5k4"
Nov 02 03:16:50.658048 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:50.658002    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d695fa41-7976-4a64-b9e2-4642ab0ed15f-istio-token\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-zf5k4\" (UID: \"d695fa41-7976-4a64-b9e2-4642ab0ed15f\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-zf5k4"
Nov 02 03:16:50.658140 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:50.658067    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d695fa41-7976-4a64-b9e2-4642ab0ed15f-workload-certs\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-zf5k4\" (UID: \"d695fa41-7976-4a64-b9e2-4642ab0ed15f\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-zf5k4"
Nov 02 03:16:50.658140 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:50.658113    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d695fa41-7976-4a64-b9e2-4642ab0ed15f-workload-socket\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-zf5k4\" (UID: \"d695fa41-7976-4a64-b9e2-4642ab0ed15f\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-zf5k4"
Nov 02 03:16:50.658262 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:50.658163    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d695fa41-7976-4a64-b9e2-4642ab0ed15f-istio-envoy\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-zf5k4\" (UID: \"d695fa41-7976-4a64-b9e2-4642ab0ed15f\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-zf5k4"
Nov 02 03:16:50.658262 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:50.658229    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2knn7\" (UniqueName: \"kubernetes.io/projected/d695fa41-7976-4a64-b9e2-4642ab0ed15f-kube-api-access-2knn7\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-zf5k4\" (UID: \"d695fa41-7976-4a64-b9e2-4642ab0ed15f\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-zf5k4"
Nov 02 03:16:50.658400 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:50.658381    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d695fa41-7976-4a64-b9e2-4642ab0ed15f-istio-data\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-zf5k4\" (UID: \"d695fa41-7976-4a64-b9e2-4642ab0ed15f\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-zf5k4"
Nov 02 03:16:50.658467 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:50.658446    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d695fa41-7976-4a64-b9e2-4642ab0ed15f-istiod-ca-cert\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-zf5k4\" (UID: \"d695fa41-7976-4a64-b9e2-4642ab0ed15f\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-zf5k4"
Nov 02 03:16:50.658520 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:50.658505    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d695fa41-7976-4a64-b9e2-4642ab0ed15f-credential-socket\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-zf5k4\" (UID: \"d695fa41-7976-4a64-b9e2-4642ab0ed15f\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-zf5k4"
Nov 02 03:16:50.658648 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:50.658617    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d695fa41-7976-4a64-b9e2-4642ab0ed15f-workload-certs\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-zf5k4\" (UID: \"d695fa41-7976-4a64-b9e2-4642ab0ed15f\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-zf5k4"
Nov 02 03:16:50.658828 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:50.658807    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d695fa41-7976-4a64-b9e2-4642ab0ed15f-credential-socket\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-zf5k4\" (UID: \"d695fa41-7976-4a64-b9e2-4642ab0ed15f\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-zf5k4"
Nov 02 03:16:50.659112 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:50.659089    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d695fa41-7976-4a64-b9e2-4642ab0ed15f-workload-socket\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-zf5k4\" (UID: \"d695fa41-7976-4a64-b9e2-4642ab0ed15f\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-zf5k4"
Nov 02 03:16:50.659220 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:50.659159    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d695fa41-7976-4a64-b9e2-4642ab0ed15f-istio-data\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-zf5k4\" (UID: \"d695fa41-7976-4a64-b9e2-4642ab0ed15f\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-zf5k4"
Nov 02 03:16:50.660004 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:50.659981    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d695fa41-7976-4a64-b9e2-4642ab0ed15f-istiod-ca-cert\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-zf5k4\" (UID: \"d695fa41-7976-4a64-b9e2-4642ab0ed15f\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-zf5k4"
Nov 02 03:16:50.663506 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:50.663484    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d695fa41-7976-4a64-b9e2-4642ab0ed15f-istio-envoy\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-zf5k4\" (UID: \"d695fa41-7976-4a64-b9e2-4642ab0ed15f\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-zf5k4"
Nov 02 03:16:50.664689 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:50.664664    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d695fa41-7976-4a64-b9e2-4642ab0ed15f-istio-podinfo\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-zf5k4\" (UID: \"d695fa41-7976-4a64-b9e2-4642ab0ed15f\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-zf5k4"
Nov 02 03:16:50.710542 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:50.710507    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-zf5k4"]
Nov 02 03:16:50.716453 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:50.716429    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d695fa41-7976-4a64-b9e2-4642ab0ed15f-istio-token\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-zf5k4\" (UID: \"d695fa41-7976-4a64-b9e2-4642ab0ed15f\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-zf5k4"
Nov 02 03:16:50.731124 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:50.731099    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2knn7\" (UniqueName: \"kubernetes.io/projected/d695fa41-7976-4a64-b9e2-4642ab0ed15f-kube-api-access-2knn7\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-zf5k4\" (UID: \"d695fa41-7976-4a64-b9e2-4642ab0ed15f\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-zf5k4"
Nov 02 03:16:50.820615 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:50.820538    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-zf5k4"
Nov 02 03:16:51.187654 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:51.187577    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84a7e2e3-bpr4q"]
Nov 02 03:16:51.193746 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:51.193725    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84a7e2e3-bpr4q"
Nov 02 03:16:51.248615 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:51.248048    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9z5bk\""
Nov 02 03:16:51.248615 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:51.248110    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 03:16:51.248615 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:51.248165    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 03:16:51.248615 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:51.248177    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 03:16:51.252329 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:51.249842    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84a7e2e3-bpr4q"]
Nov 02 03:16:51.265967 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:51.265937    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9a2f2b6-c3a0-414f-89fa-50ce0279fb82-maas-api-xks-ca\") pod \"maas-api-revoke-sub-84a7e2e3-bpr4q\" (UID: \"a9a2f2b6-c3a0-414f-89fa-50ce0279fb82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84a7e2e3-bpr4q"
Nov 02 03:16:51.266117 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:51.266080    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9a2f2b6-c3a0-414f-89fa-50ce0279fb82-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-84a7e2e3-bpr4q\" (UID: \"a9a2f2b6-c3a0-414f-89fa-50ce0279fb82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84a7e2e3-bpr4q"
Nov 02 03:16:51.266186 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:51.266170    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klntk\" (UniqueName: \"kubernetes.io/projected/a9a2f2b6-c3a0-414f-89fa-50ce0279fb82-kube-api-access-klntk\") pod \"maas-api-revoke-sub-84a7e2e3-bpr4q\" (UID: \"a9a2f2b6-c3a0-414f-89fa-50ce0279fb82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84a7e2e3-bpr4q"
Nov 02 03:16:51.307397 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:16:51.307137    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd695fa41_7976_4a64_b9e2_4642ab0ed15f.slice/crio-010412e342a85f05f004fd724ce4da9b2101998c52abdcbb4c54ab189f77fd35 WatchSource:0}: Error finding container 010412e342a85f05f004fd724ce4da9b2101998c52abdcbb4c54ab189f77fd35: Status 404 returned error can't find the container with id 010412e342a85f05f004fd724ce4da9b2101998c52abdcbb4c54ab189f77fd35
Nov 02 03:16:51.308247 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:51.308230    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-zf5k4"]
Nov 02 03:16:51.310024 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:51.309980    4496 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 03:16:51.310093 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:51.310075    4496 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 03:16:51.310126 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:51.310114    4496 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 03:16:51.367569 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:51.367538    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9a2f2b6-c3a0-414f-89fa-50ce0279fb82-maas-api-xks-ca\") pod \"maas-api-revoke-sub-84a7e2e3-bpr4q\" (UID: \"a9a2f2b6-c3a0-414f-89fa-50ce0279fb82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84a7e2e3-bpr4q"
Nov 02 03:16:51.367669 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:51.367611    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9a2f2b6-c3a0-414f-89fa-50ce0279fb82-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-84a7e2e3-bpr4q\" (UID: \"a9a2f2b6-c3a0-414f-89fa-50ce0279fb82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84a7e2e3-bpr4q"
Nov 02 03:16:51.367747 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:51.367720    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klntk\" (UniqueName: \"kubernetes.io/projected/a9a2f2b6-c3a0-414f-89fa-50ce0279fb82-kube-api-access-klntk\") pod \"maas-api-revoke-sub-84a7e2e3-bpr4q\" (UID: \"a9a2f2b6-c3a0-414f-89fa-50ce0279fb82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84a7e2e3-bpr4q"
Nov 02 03:16:51.368476 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:51.368445    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9a2f2b6-c3a0-414f-89fa-50ce0279fb82-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-84a7e2e3-bpr4q\" (UID: \"a9a2f2b6-c3a0-414f-89fa-50ce0279fb82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84a7e2e3-bpr4q"
Nov 02 03:16:51.370400 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:51.370373    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9a2f2b6-c3a0-414f-89fa-50ce0279fb82-maas-api-xks-ca\") pod \"maas-api-revoke-sub-84a7e2e3-bpr4q\" (UID: \"a9a2f2b6-c3a0-414f-89fa-50ce0279fb82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84a7e2e3-bpr4q"
Nov 02 03:16:51.425681 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:51.425649    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klntk\" (UniqueName: \"kubernetes.io/projected/a9a2f2b6-c3a0-414f-89fa-50ce0279fb82-kube-api-access-klntk\") pod \"maas-api-revoke-sub-84a7e2e3-bpr4q\" (UID: \"a9a2f2b6-c3a0-414f-89fa-50ce0279fb82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84a7e2e3-bpr4q"
Nov 02 03:16:51.512758 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:51.512730    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84a7e2e3-bpr4q"
Nov 02 03:16:51.666486 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:51.666434    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-zf5k4" event={"ID":"d695fa41-7976-4a64-b9e2-4642ab0ed15f","Type":"ContainerStarted","Data":"efe2ac77c34e85a4a29e78deb7391c2336e8ed6476be5ba780462d7ee01b78cb"}
Nov 02 03:16:51.666742 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:51.666718    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-zf5k4" event={"ID":"d695fa41-7976-4a64-b9e2-4642ab0ed15f","Type":"ContainerStarted","Data":"010412e342a85f05f004fd724ce4da9b2101998c52abdcbb4c54ab189f77fd35"}
Nov 02 03:16:51.787705 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:51.787610    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-zf5k4" podStartSLOduration=1.7875922659999999 podStartE2EDuration="1.787592266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:16:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:16:51.785631787 +0000 UTC m=+1752.364239169" watchObservedRunningTime="2026-11-02 03:16:51.787592266 +0000 UTC m=+1752.366199647"
Nov 02 03:16:51.823138 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:51.823101    4496 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-zf5k4"
Nov 02 03:16:51.901914 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:16:51.901854    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9a2f2b6_c3a0_414f_89fa_50ce0279fb82.slice/crio-4303dd98d55c18de180e9dc2fd9d63e25c40772ab574324de8a648fcd7e653f9 WatchSource:0}: Error finding container 4303dd98d55c18de180e9dc2fd9d63e25c40772ab574324de8a648fcd7e653f9: Status 404 returned error can't find the container with id 4303dd98d55c18de180e9dc2fd9d63e25c40772ab574324de8a648fcd7e653f9
Nov 02 03:16:51.902180 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:51.902155    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84a7e2e3-bpr4q"]
Nov 02 03:16:52.172828 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:52.172798    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hdjcq"]
Nov 02 03:16:52.178214 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:52.178191    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hdjcq"
Nov 02 03:16:52.199154 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:52.199124    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-ca6546/e2e-conflict-a-5eca459f-kserve-5744ffc845-xgm78" podUID="c862a274-1d99-4f90-926e-4a72041456ab" containerName="main" containerID="cri-o://5c584c396c8865dbcf4ed5cc3f60d9a19378c86111d39791bd2d38849e58ffaf" gracePeriod=45
Nov 02 03:16:52.260040 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:52.256033    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hdjcq"]
Nov 02 03:16:52.286698 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:52.277936    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f-istio-data\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hdjcq\" (UID: \"37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hdjcq"
Nov 02 03:16:52.286698 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:52.277984    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f-workload-certs\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hdjcq\" (UID: \"37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hdjcq"
Nov 02 03:16:52.286698 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:52.278163    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f-istiod-ca-cert\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hdjcq\" (UID: \"37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hdjcq"
Nov 02 03:16:52.286698 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:52.278278    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f-istio-envoy\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hdjcq\" (UID: \"37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hdjcq"
Nov 02 03:16:52.286698 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:52.278383    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f-istio-token\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hdjcq\" (UID: \"37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hdjcq"
Nov 02 03:16:52.286698 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:52.278496    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f-credential-socket\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hdjcq\" (UID: \"37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hdjcq"
Nov 02 03:16:52.286698 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:52.278608    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f-istio-podinfo\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hdjcq\" (UID: \"37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hdjcq"
Nov 02 03:16:52.286698 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:52.278684    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9flpz\" (UniqueName: \"kubernetes.io/projected/37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f-kube-api-access-9flpz\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hdjcq\" (UID: \"37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hdjcq"
Nov 02 03:16:52.286698 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:52.278775    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f-workload-socket\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hdjcq\" (UID: \"37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hdjcq"
Nov 02 03:16:52.380256 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:52.380227    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f-credential-socket\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hdjcq\" (UID: \"37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hdjcq"
Nov 02 03:16:52.380381 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:52.380296    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f-istio-podinfo\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hdjcq\" (UID: \"37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hdjcq"
Nov 02 03:16:52.380381 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:52.380333    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9flpz\" (UniqueName: \"kubernetes.io/projected/37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f-kube-api-access-9flpz\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hdjcq\" (UID: \"37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hdjcq"
Nov 02 03:16:52.380498 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:52.380378    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f-workload-socket\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hdjcq\" (UID: \"37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hdjcq"
Nov 02 03:16:52.380498 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:52.380425    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f-istio-data\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hdjcq\" (UID: \"37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hdjcq"
Nov 02 03:16:52.380498 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:52.380462    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f-workload-certs\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hdjcq\" (UID: \"37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hdjcq"
Nov 02 03:16:52.380655 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:52.380539    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f-istiod-ca-cert\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hdjcq\" (UID: \"37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hdjcq"
Nov 02 03:16:52.380655 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:52.380602    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f-istio-envoy\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hdjcq\" (UID: \"37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hdjcq"
Nov 02 03:16:52.380760 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:52.380660    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f-istio-token\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hdjcq\" (UID: \"37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hdjcq"
Nov 02 03:16:52.381177 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:52.381141    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f-workload-socket\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hdjcq\" (UID: \"37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hdjcq"
Nov 02 03:16:52.381297 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:52.381224    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f-workload-certs\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hdjcq\" (UID: \"37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hdjcq"
Nov 02 03:16:52.381408 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:52.381390    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f-istio-data\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hdjcq\" (UID: \"37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hdjcq"
Nov 02 03:16:52.381484 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:52.381456    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f-credential-socket\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hdjcq\" (UID: \"37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hdjcq"
Nov 02 03:16:52.381760 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:52.381740    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f-istiod-ca-cert\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hdjcq\" (UID: \"37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hdjcq"
Nov 02 03:16:52.383438 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:52.383392    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f-istio-envoy\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hdjcq\" (UID: \"37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hdjcq"
Nov 02 03:16:52.383553 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:52.383535    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f-istio-podinfo\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hdjcq\" (UID: \"37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hdjcq"
Nov 02 03:16:52.467143 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:52.466768    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9flpz\" (UniqueName: \"kubernetes.io/projected/37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f-kube-api-access-9flpz\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hdjcq\" (UID: \"37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hdjcq"
Nov 02 03:16:52.469356 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:52.469330    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f-istio-token\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hdjcq\" (UID: \"37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hdjcq"
Nov 02 03:16:52.492271 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:52.492246    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hdjcq"
Nov 02 03:16:52.493464 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:52.493446    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-ca6546/e2e-conflict-a-5eca459f-kserve-5744ffc845-xgm78"
Nov 02 03:16:52.586856 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:52.586823    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c862a274-1d99-4f90-926e-4a72041456ab-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c862a274-1d99-4f90-926e-4a72041456ab-tmp-dir\") pod \"c862a274-1d99-4f90-926e-4a72041456ab\" (UID: \"c862a274-1d99-4f90-926e-4a72041456ab\") "
Nov 02 03:16:52.586941 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:52.586919    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c862a274-1d99-4f90-926e-4a72041456ab-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c862a274-1d99-4f90-926e-4a72041456ab-dshm\") pod \"c862a274-1d99-4f90-926e-4a72041456ab\" (UID: \"c862a274-1d99-4f90-926e-4a72041456ab\") "
Nov 02 03:16:52.587784 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:52.586998    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c862a274-1d99-4f90-926e-4a72041456ab-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c862a274-1d99-4f90-926e-4a72041456ab-model-cache\") pod \"c862a274-1d99-4f90-926e-4a72041456ab\" (UID: \"c862a274-1d99-4f90-926e-4a72041456ab\") "
Nov 02 03:16:52.587784 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:52.587054    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c862a274-1d99-4f90-926e-4a72041456ab-tls-certs\" (UniqueName: \"kubernetes.io/secret/c862a274-1d99-4f90-926e-4a72041456ab-tls-certs\") pod \"c862a274-1d99-4f90-926e-4a72041456ab\" (UID: \"c862a274-1d99-4f90-926e-4a72041456ab\") "
Nov 02 03:16:52.587784 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:52.587105    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c862a274-1d99-4f90-926e-4a72041456ab-home\" (UniqueName: \"kubernetes.io/empty-dir/c862a274-1d99-4f90-926e-4a72041456ab-home\") pod \"c862a274-1d99-4f90-926e-4a72041456ab\" (UID: \"c862a274-1d99-4f90-926e-4a72041456ab\") "
Nov 02 03:16:52.587784 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:52.587209    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c862a274-1d99-4f90-926e-4a72041456ab-kube-api-access-28mvw\" (UniqueName: \"kubernetes.io/projected/c862a274-1d99-4f90-926e-4a72041456ab-kube-api-access-28mvw\") pod \"c862a274-1d99-4f90-926e-4a72041456ab\" (UID: \"c862a274-1d99-4f90-926e-4a72041456ab\") "
Nov 02 03:16:52.587784 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:52.587263    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c862a274-1d99-4f90-926e-4a72041456ab-tmp-dir" pod "c862a274-1d99-4f90-926e-4a72041456ab" (UID: "c862a274-1d99-4f90-926e-4a72041456ab"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:16:52.587784 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:52.587451    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c862a274-1d99-4f90-926e-4a72041456ab-model-cache" pod "c862a274-1d99-4f90-926e-4a72041456ab" (UID: "c862a274-1d99-4f90-926e-4a72041456ab"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:16:52.588219 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:52.587854    4496 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c862a274-1d99-4f90-926e-4a72041456ab-tmp-dir\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:16:52.588219 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:52.587875    4496 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c862a274-1d99-4f90-926e-4a72041456ab-model-cache\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:16:52.588620 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:52.588595    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c862a274-1d99-4f90-926e-4a72041456ab-home" pod "c862a274-1d99-4f90-926e-4a72041456ab" (UID: "c862a274-1d99-4f90-926e-4a72041456ab"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:16:52.590389 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:52.590351    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c862a274-1d99-4f90-926e-4a72041456ab-kube-api-access-28mvw" pod "c862a274-1d99-4f90-926e-4a72041456ab" (UID: "c862a274-1d99-4f90-926e-4a72041456ab"). InnerVolumeSpecName "kube-api-access-28mvw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:16:52.591326 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:52.591292    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c862a274-1d99-4f90-926e-4a72041456ab-tls-certs" pod "c862a274-1d99-4f90-926e-4a72041456ab" (UID: "c862a274-1d99-4f90-926e-4a72041456ab"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 03:16:52.594861 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:52.594839    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c862a274-1d99-4f90-926e-4a72041456ab-dshm" pod "c862a274-1d99-4f90-926e-4a72041456ab" (UID: "c862a274-1d99-4f90-926e-4a72041456ab"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:16:52.692277 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:52.689622    4496 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c862a274-1d99-4f90-926e-4a72041456ab-tls-certs\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:16:52.692277 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:52.689656    4496 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c862a274-1d99-4f90-926e-4a72041456ab-home\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:16:52.692277 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:52.689673    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28mvw\" (UniqueName: \"kubernetes.io/projected/c862a274-1d99-4f90-926e-4a72041456ab-kube-api-access-28mvw\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:16:52.692277 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:52.689693    4496 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c862a274-1d99-4f90-926e-4a72041456ab-dshm\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:16:52.693701 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:52.693665    4496 generic.go:356] "Generic (PLEG): container finished" podID="c862a274-1d99-4f90-926e-4a72041456ab" containerID="5c584c396c8865dbcf4ed5cc3f60d9a19378c86111d39791bd2d38849e58ffaf" exitCode=0
Nov 02 03:16:52.694144 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:52.693878    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-ca6546/e2e-conflict-a-5eca459f-kserve-5744ffc845-xgm78"
Nov 02 03:16:52.694614 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:52.694409    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-ca6546/e2e-conflict-a-5eca459f-kserve-5744ffc845-xgm78" event={"ID":"c862a274-1d99-4f90-926e-4a72041456ab","Type":"ContainerDied","Data":"5c584c396c8865dbcf4ed5cc3f60d9a19378c86111d39791bd2d38849e58ffaf"}
Nov 02 03:16:52.694614 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:52.694460    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-ca6546/e2e-conflict-a-5eca459f-kserve-5744ffc845-xgm78" event={"ID":"c862a274-1d99-4f90-926e-4a72041456ab","Type":"ContainerDied","Data":"d1de3356e1478bf1a600eff01d8073d980aeb7f60a896dab0dea4595d9e0aa60"}
Nov 02 03:16:52.694614 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:52.694497    4496 scope.go:122] "RemoveContainer" containerID="5c584c396c8865dbcf4ed5cc3f60d9a19378c86111d39791bd2d38849e58ffaf"
Nov 02 03:16:52.699199 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:52.699171    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84a7e2e3-bpr4q" event={"ID":"a9a2f2b6-c3a0-414f-89fa-50ce0279fb82","Type":"ContainerStarted","Data":"bb960bcc3372d1a20c682cac680245835e5d6c5bedabcecd4c36d5d7695ee8f2"}
Nov 02 03:16:52.699301 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:52.699268    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84a7e2e3-bpr4q" event={"ID":"a9a2f2b6-c3a0-414f-89fa-50ce0279fb82","Type":"ContainerStarted","Data":"4303dd98d55c18de180e9dc2fd9d63e25c40772ab574324de8a648fcd7e653f9"}
Nov 02 03:16:52.708741 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:52.708721    4496 scope.go:122] "RemoveContainer" containerID="5c584c396c8865dbcf4ed5cc3f60d9a19378c86111d39791bd2d38849e58ffaf"
Nov 02 03:16:52.709073 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:16:52.709007    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c584c396c8865dbcf4ed5cc3f60d9a19378c86111d39791bd2d38849e58ffaf\": container with ID starting with 5c584c396c8865dbcf4ed5cc3f60d9a19378c86111d39791bd2d38849e58ffaf not found: ID does not exist" containerID="5c584c396c8865dbcf4ed5cc3f60d9a19378c86111d39791bd2d38849e58ffaf"
Nov 02 03:16:52.709161 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:52.709078    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c584c396c8865dbcf4ed5cc3f60d9a19378c86111d39791bd2d38849e58ffaf"} err="failed to get container status \"5c584c396c8865dbcf4ed5cc3f60d9a19378c86111d39791bd2d38849e58ffaf\": rpc error: code = NotFound desc = could not find container \"5c584c396c8865dbcf4ed5cc3f60d9a19378c86111d39791bd2d38849e58ffaf\": container with ID starting with 5c584c396c8865dbcf4ed5cc3f60d9a19378c86111d39791bd2d38849e58ffaf not found: ID does not exist"
Nov 02 03:16:52.823235 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:52.823200    4496 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-zf5k4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body=
Nov 02 03:16:52.823561 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:52.823260    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-zf5k4" podUID="d695fa41-7976-4a64-b9e2-4642ab0ed15f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused"
Nov 02 03:16:52.871009 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:52.870976    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-ca6546/e2e-conflict-a-5eca459f-kserve-5744ffc845-xgm78"]
Nov 02 03:16:52.874933 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:52.874902    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-ca6546/e2e-conflict-a-5eca459f-kserve-5744ffc845-xgm78"]
Nov 02 03:16:52.963902 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:52.963875    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hdjcq"]
Nov 02 03:16:52.964595 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:16:52.964569    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37c23d26_aa7d_41dd_b3d9_7a3408ec2f1f.slice/crio-b898b0cc29ae06258212b8ad9b2a98cc991ed84aa7d88b6e144823f40cf386e6 WatchSource:0}: Error finding container b898b0cc29ae06258212b8ad9b2a98cc991ed84aa7d88b6e144823f40cf386e6: Status 404 returned error can't find the container with id b898b0cc29ae06258212b8ad9b2a98cc991ed84aa7d88b6e144823f40cf386e6
Nov 02 03:16:52.967079 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:52.967049    4496 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 03:16:52.967179 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:52.967112    4496 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 03:16:52.967179 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:52.967145    4496 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 03:16:52.985940 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:52.985894    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84a7e2e3-bpr4q" podStartSLOduration=1.985881073 podStartE2EDuration="1.985881073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:16:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:16:52.983887534 +0000 UTC m=+1753.562494915" watchObservedRunningTime="2026-11-02 03:16:52.985881073 +0000 UTC m=+1753.564488497"
Nov 02 03:16:53.584039 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:53.580183    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c862a274-1d99-4f90-926e-4a72041456ab" path="/var/lib/kubelet/pods/c862a274-1d99-4f90-926e-4a72041456ab/volumes"
Nov 02 03:16:53.707438 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:53.707403    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hdjcq" event={"ID":"37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f","Type":"ContainerStarted","Data":"3c99703636204b5e27d2c604df31d3018e4377683b9d37883fa2f3b33a659b2d"}
Nov 02 03:16:53.707608 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:53.707457    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hdjcq" event={"ID":"37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f","Type":"ContainerStarted","Data":"b898b0cc29ae06258212b8ad9b2a98cc991ed84aa7d88b6e144823f40cf386e6"}
Nov 02 03:16:53.710111 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:53.710091    4496 generic.go:356] "Generic (PLEG): container finished" podID="a9a2f2b6-c3a0-414f-89fa-50ce0279fb82" containerID="bb960bcc3372d1a20c682cac680245835e5d6c5bedabcecd4c36d5d7695ee8f2" exitCode=0
Nov 02 03:16:53.710298 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:53.710166    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84a7e2e3-bpr4q" event={"ID":"a9a2f2b6-c3a0-414f-89fa-50ce0279fb82","Type":"ContainerDied","Data":"bb960bcc3372d1a20c682cac680245835e5d6c5bedabcecd4c36d5d7695ee8f2"}
Nov 02 03:16:53.821000 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:53.820966    4496 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-zf5k4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body=
Nov 02 03:16:53.821182 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:53.821061    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-zf5k4" podUID="d695fa41-7976-4a64-b9e2-4642ab0ed15f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused"
Nov 02 03:16:53.830738 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:53.830687    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hdjcq" podStartSLOduration=1.830672737 podStartE2EDuration="1.830672737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:16:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:16:53.826315657 +0000 UTC m=+1754.404923038" watchObservedRunningTime="2026-11-02 03:16:53.830672737 +0000 UTC m=+1754.409280117"
Nov 02 03:16:54.494209 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:54.494160    4496 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hdjcq"
Nov 02 03:16:54.495856 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:54.495825    4496 patch_prober.go:29] interesting pod/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hdjcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body=
Nov 02 03:16:54.495924 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:54.495897    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hdjcq" podUID="37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused"
Nov 02 03:16:54.791077 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:54.785392    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-398828/distinct-2-w4-398828-kserve-545449d49b-6mg4b"]
Nov 02 03:16:54.791077 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:54.786398    4496 cpu_manager.go:414] "containerMap: removing container" podUID="c862a274-1d99-4f90-926e-4a72041456ab" containerName="main"
Nov 02 03:16:54.791077 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:54.786419    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c862a274-1d99-4f90-926e-4a72041456ab" containerName="main"
Nov 02 03:16:54.791077 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:54.786577    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="c862a274-1d99-4f90-926e-4a72041456ab" containerName="main"
Nov 02 03:16:54.805979 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:54.805545    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-398828/distinct-2-w4-398828-kserve-545449d49b-6mg4b"
Nov 02 03:16:54.809283 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:54.809256    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-398828/distinct-2-w4-398828-kserve-545449d49b-6mg4b"]
Nov 02 03:16:54.820445 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:54.818139    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36c1912c-2ab1-40c8-a09d-2425ab08ec8d-dshm\") pod \"distinct-2-w4-398828-kserve-545449d49b-6mg4b\" (UID: \"36c1912c-2ab1-40c8-a09d-2425ab08ec8d\") " pod="e2e-models-e2e-worker-w4-398828/distinct-2-w4-398828-kserve-545449d49b-6mg4b"
Nov 02 03:16:54.820445 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:54.818191    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36c1912c-2ab1-40c8-a09d-2425ab08ec8d-tls-certs\") pod \"distinct-2-w4-398828-kserve-545449d49b-6mg4b\" (UID: \"36c1912c-2ab1-40c8-a09d-2425ab08ec8d\") " pod="e2e-models-e2e-worker-w4-398828/distinct-2-w4-398828-kserve-545449d49b-6mg4b"
Nov 02 03:16:54.820445 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:54.818223    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36c1912c-2ab1-40c8-a09d-2425ab08ec8d-model-cache\") pod \"distinct-2-w4-398828-kserve-545449d49b-6mg4b\" (UID: \"36c1912c-2ab1-40c8-a09d-2425ab08ec8d\") " pod="e2e-models-e2e-worker-w4-398828/distinct-2-w4-398828-kserve-545449d49b-6mg4b"
Nov 02 03:16:54.820445 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:54.818335    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36c1912c-2ab1-40c8-a09d-2425ab08ec8d-home\") pod \"distinct-2-w4-398828-kserve-545449d49b-6mg4b\" (UID: \"36c1912c-2ab1-40c8-a09d-2425ab08ec8d\") " pod="e2e-models-e2e-worker-w4-398828/distinct-2-w4-398828-kserve-545449d49b-6mg4b"
Nov 02 03:16:54.820445 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:54.818473    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fn527\" (UniqueName: \"kubernetes.io/projected/36c1912c-2ab1-40c8-a09d-2425ab08ec8d-kube-api-access-fn527\") pod \"distinct-2-w4-398828-kserve-545449d49b-6mg4b\" (UID: \"36c1912c-2ab1-40c8-a09d-2425ab08ec8d\") " pod="e2e-models-e2e-worker-w4-398828/distinct-2-w4-398828-kserve-545449d49b-6mg4b"
Nov 02 03:16:54.820445 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:54.818535    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36c1912c-2ab1-40c8-a09d-2425ab08ec8d-tmp-dir\") pod \"distinct-2-w4-398828-kserve-545449d49b-6mg4b\" (UID: \"36c1912c-2ab1-40c8-a09d-2425ab08ec8d\") " pod="e2e-models-e2e-worker-w4-398828/distinct-2-w4-398828-kserve-545449d49b-6mg4b"
Nov 02 03:16:54.820985 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:54.820967    4496 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-zf5k4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body=
Nov 02 03:16:54.821063 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:54.821006    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-zf5k4" podUID="d695fa41-7976-4a64-b9e2-4642ab0ed15f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused"
Nov 02 03:16:54.923829 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:54.923308    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36c1912c-2ab1-40c8-a09d-2425ab08ec8d-home\") pod \"distinct-2-w4-398828-kserve-545449d49b-6mg4b\" (UID: \"36c1912c-2ab1-40c8-a09d-2425ab08ec8d\") " pod="e2e-models-e2e-worker-w4-398828/distinct-2-w4-398828-kserve-545449d49b-6mg4b"
Nov 02 03:16:54.924281 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:54.923787    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36c1912c-2ab1-40c8-a09d-2425ab08ec8d-home\") pod \"distinct-2-w4-398828-kserve-545449d49b-6mg4b\" (UID: \"36c1912c-2ab1-40c8-a09d-2425ab08ec8d\") " pod="e2e-models-e2e-worker-w4-398828/distinct-2-w4-398828-kserve-545449d49b-6mg4b"
Nov 02 03:16:54.924281 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:54.924044    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fn527\" (UniqueName: \"kubernetes.io/projected/36c1912c-2ab1-40c8-a09d-2425ab08ec8d-kube-api-access-fn527\") pod \"distinct-2-w4-398828-kserve-545449d49b-6mg4b\" (UID: \"36c1912c-2ab1-40c8-a09d-2425ab08ec8d\") " pod="e2e-models-e2e-worker-w4-398828/distinct-2-w4-398828-kserve-545449d49b-6mg4b"
Nov 02 03:16:54.926352 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:54.924429    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36c1912c-2ab1-40c8-a09d-2425ab08ec8d-tmp-dir\") pod \"distinct-2-w4-398828-kserve-545449d49b-6mg4b\" (UID: \"36c1912c-2ab1-40c8-a09d-2425ab08ec8d\") " pod="e2e-models-e2e-worker-w4-398828/distinct-2-w4-398828-kserve-545449d49b-6mg4b"
Nov 02 03:16:54.926352 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:54.924728    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36c1912c-2ab1-40c8-a09d-2425ab08ec8d-tmp-dir\") pod \"distinct-2-w4-398828-kserve-545449d49b-6mg4b\" (UID: \"36c1912c-2ab1-40c8-a09d-2425ab08ec8d\") " pod="e2e-models-e2e-worker-w4-398828/distinct-2-w4-398828-kserve-545449d49b-6mg4b"
Nov 02 03:16:54.926352 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:54.925004    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36c1912c-2ab1-40c8-a09d-2425ab08ec8d-dshm\") pod \"distinct-2-w4-398828-kserve-545449d49b-6mg4b\" (UID: \"36c1912c-2ab1-40c8-a09d-2425ab08ec8d\") " pod="e2e-models-e2e-worker-w4-398828/distinct-2-w4-398828-kserve-545449d49b-6mg4b"
Nov 02 03:16:54.926352 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:54.925403    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36c1912c-2ab1-40c8-a09d-2425ab08ec8d-tls-certs\") pod \"distinct-2-w4-398828-kserve-545449d49b-6mg4b\" (UID: \"36c1912c-2ab1-40c8-a09d-2425ab08ec8d\") " pod="e2e-models-e2e-worker-w4-398828/distinct-2-w4-398828-kserve-545449d49b-6mg4b"
Nov 02 03:16:54.926352 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:54.925761    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36c1912c-2ab1-40c8-a09d-2425ab08ec8d-model-cache\") pod \"distinct-2-w4-398828-kserve-545449d49b-6mg4b\" (UID: \"36c1912c-2ab1-40c8-a09d-2425ab08ec8d\") " pod="e2e-models-e2e-worker-w4-398828/distinct-2-w4-398828-kserve-545449d49b-6mg4b"
Nov 02 03:16:54.926352 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:54.925432    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36c1912c-2ab1-40c8-a09d-2425ab08ec8d-model-cache\") pod \"distinct-2-w4-398828-kserve-545449d49b-6mg4b\" (UID: \"36c1912c-2ab1-40c8-a09d-2425ab08ec8d\") " pod="e2e-models-e2e-worker-w4-398828/distinct-2-w4-398828-kserve-545449d49b-6mg4b"
Nov 02 03:16:54.957039 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:54.956325    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-398828\"/\"distinct-2-w4-398828-kserve-self-signed-certs\""
Nov 02 03:16:54.972039 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:54.969034    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36c1912c-2ab1-40c8-a09d-2425ab08ec8d-dshm\") pod \"distinct-2-w4-398828-kserve-545449d49b-6mg4b\" (UID: \"36c1912c-2ab1-40c8-a09d-2425ab08ec8d\") " pod="e2e-models-e2e-worker-w4-398828/distinct-2-w4-398828-kserve-545449d49b-6mg4b"
Nov 02 03:16:54.996803 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:54.996772    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fn527\" (UniqueName: \"kubernetes.io/projected/36c1912c-2ab1-40c8-a09d-2425ab08ec8d-kube-api-access-fn527\") pod \"distinct-2-w4-398828-kserve-545449d49b-6mg4b\" (UID: \"36c1912c-2ab1-40c8-a09d-2425ab08ec8d\") " pod="e2e-models-e2e-worker-w4-398828/distinct-2-w4-398828-kserve-545449d49b-6mg4b"
Nov 02 03:16:55.000839 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:55.000808    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36c1912c-2ab1-40c8-a09d-2425ab08ec8d-tls-certs\") pod \"distinct-2-w4-398828-kserve-545449d49b-6mg4b\" (UID: \"36c1912c-2ab1-40c8-a09d-2425ab08ec8d\") " pod="e2e-models-e2e-worker-w4-398828/distinct-2-w4-398828-kserve-545449d49b-6mg4b"
Nov 02 03:16:55.094923 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:55.094893    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84a7e2e3-bpr4q"
Nov 02 03:16:55.122881 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:55.122851    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-398828/distinct-2-w4-398828-kserve-545449d49b-6mg4b"
Nov 02 03:16:55.127632 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:55.127477    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9a2f2b6-c3a0-414f-89fa-50ce0279fb82-kube-api-access-klntk\" (UniqueName: \"kubernetes.io/projected/a9a2f2b6-c3a0-414f-89fa-50ce0279fb82-kube-api-access-klntk\") pod \"a9a2f2b6-c3a0-414f-89fa-50ce0279fb82\" (UID: \"a9a2f2b6-c3a0-414f-89fa-50ce0279fb82\") "
Nov 02 03:16:55.127632 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:55.127608    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9a2f2b6-c3a0-414f-89fa-50ce0279fb82-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9a2f2b6-c3a0-414f-89fa-50ce0279fb82-maas-api-xks-ca\") pod \"a9a2f2b6-c3a0-414f-89fa-50ce0279fb82\" (UID: \"a9a2f2b6-c3a0-414f-89fa-50ce0279fb82\") "
Nov 02 03:16:55.127981 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:55.127796    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9a2f2b6-c3a0-414f-89fa-50ce0279fb82-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9a2f2b6-c3a0-414f-89fa-50ce0279fb82-maas-api-openshift-ca\") pod \"a9a2f2b6-c3a0-414f-89fa-50ce0279fb82\" (UID: \"a9a2f2b6-c3a0-414f-89fa-50ce0279fb82\") "
Nov 02 03:16:55.128442 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:55.128422    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9a2f2b6-c3a0-414f-89fa-50ce0279fb82-maas-api-openshift-ca" pod "a9a2f2b6-c3a0-414f-89fa-50ce0279fb82" (UID: "a9a2f2b6-c3a0-414f-89fa-50ce0279fb82"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:16:55.129842 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:55.129821    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9a2f2b6-c3a0-414f-89fa-50ce0279fb82-kube-api-access-klntk" pod "a9a2f2b6-c3a0-414f-89fa-50ce0279fb82" (UID: "a9a2f2b6-c3a0-414f-89fa-50ce0279fb82"). InnerVolumeSpecName "kube-api-access-klntk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:16:55.130109 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:55.130090    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9a2f2b6-c3a0-414f-89fa-50ce0279fb82-maas-api-xks-ca" pod "a9a2f2b6-c3a0-414f-89fa-50ce0279fb82" (UID: "a9a2f2b6-c3a0-414f-89fa-50ce0279fb82"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 03:16:55.242082 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:55.230155    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klntk\" (UniqueName: \"kubernetes.io/projected/a9a2f2b6-c3a0-414f-89fa-50ce0279fb82-kube-api-access-klntk\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:16:55.242082 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:55.230191    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9a2f2b6-c3a0-414f-89fa-50ce0279fb82-maas-api-xks-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:16:55.242082 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:55.230208    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9a2f2b6-c3a0-414f-89fa-50ce0279fb82-maas-api-openshift-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:16:55.492780 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:55.492601    4496 patch_prober.go:29] interesting pod/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hdjcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body=
Nov 02 03:16:55.492780 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:55.492666    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hdjcq" podUID="37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused"
Nov 02 03:16:55.669040 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:55.665526    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-398828/distinct-2-w4-398828-kserve-545449d49b-6mg4b"]
Nov 02 03:16:55.699651 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:16:55.699619    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod36c1912c_2ab1_40c8_a09d_2425ab08ec8d.slice/crio-934ef081fedfdc5a0fa622ff6f3a69696e3dc0de9b850088633abaa8578471f1 WatchSource:0}: Error finding container 934ef081fedfdc5a0fa622ff6f3a69696e3dc0de9b850088633abaa8578471f1: Status 404 returned error can't find the container with id 934ef081fedfdc5a0fa622ff6f3a69696e3dc0de9b850088633abaa8578471f1
Nov 02 03:16:55.727796 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:55.727772    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84a7e2e3-bpr4q" event={"ID":"a9a2f2b6-c3a0-414f-89fa-50ce0279fb82","Type":"ContainerDied","Data":"4303dd98d55c18de180e9dc2fd9d63e25c40772ab574324de8a648fcd7e653f9"}
Nov 02 03:16:55.727955 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:55.727938    4496 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4303dd98d55c18de180e9dc2fd9d63e25c40772ab574324de8a648fcd7e653f9"
Nov 02 03:16:55.727955 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:55.727779    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84a7e2e3-bpr4q"
Nov 02 03:16:55.729648 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:55.729623    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-398828/distinct-2-w4-398828-kserve-545449d49b-6mg4b" event={"ID":"36c1912c-2ab1-40c8-a09d-2425ab08ec8d","Type":"ContainerStarted","Data":"934ef081fedfdc5a0fa622ff6f3a69696e3dc0de9b850088633abaa8578471f1"}
Nov 02 03:16:55.821077 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:55.821046    4496 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-zf5k4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body=
Nov 02 03:16:55.821209 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:55.821090    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-zf5k4" podUID="d695fa41-7976-4a64-b9e2-4642ab0ed15f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused"
Nov 02 03:16:56.493702 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:56.493351    4496 patch_prober.go:29] interesting pod/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hdjcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body=
Nov 02 03:16:56.493702 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:56.493409    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hdjcq" podUID="37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused"
Nov 02 03:16:56.769099 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:56.768126    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-398828/distinct-2-w4-398828-kserve-545449d49b-6mg4b" event={"ID":"36c1912c-2ab1-40c8-a09d-2425ab08ec8d","Type":"ContainerStarted","Data":"c415850ad33f904fe4b236a1dff920b58a59368ea86dbd194172fde21cc3214f"}
Nov 02 03:16:56.821448 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:56.821414    4496 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-zf5k4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body=
Nov 02 03:16:56.821599 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:56.821466    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-zf5k4" podUID="d695fa41-7976-4a64-b9e2-4642ab0ed15f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused"
Nov 02 03:16:56.895061 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:56.892982    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-398828/distinct-2-w4-398828-kserve-545449d49b-6mg4b" podStartSLOduration=2.892963559 podStartE2EDuration="2.892963559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:16:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:16:56.889721925 +0000 UTC m=+1757.468329308" watchObservedRunningTime="2026-11-02 03:16:56.892963559 +0000 UTC m=+1757.471570940"
Nov 02 03:16:57.124065 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:57.123964    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-398828/distinct-2-w4-398828-kserve-545449d49b-6mg4b"
Nov 02 03:16:57.493412 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:57.493381    4496 patch_prober.go:29] interesting pod/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hdjcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body=
Nov 02 03:16:57.493600 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:57.493437    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hdjcq" podUID="37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused"
Nov 02 03:16:57.821424 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:57.821334    4496 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-zf5k4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body=
Nov 02 03:16:57.821424 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:57.821382    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-zf5k4" podUID="d695fa41-7976-4a64-b9e2-4642ab0ed15f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused"
Nov 02 03:16:58.493637 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:58.493600    4496 patch_prober.go:29] interesting pod/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hdjcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body=
Nov 02 03:16:58.493790 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:58.493665    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hdjcq" podUID="37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused"
Nov 02 03:16:58.627774 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:58.627736    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-858236/distinct-w3-858236-kserve-df6dbd64b-qbk9g"]
Nov 02 03:16:58.628733 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:58.628710    4496 cpu_manager.go:414] "containerMap: removing container" podUID="a9a2f2b6-c3a0-414f-89fa-50ce0279fb82" containerName="revoke-keys"
Nov 02 03:16:58.628823 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:58.628737    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9a2f2b6-c3a0-414f-89fa-50ce0279fb82" containerName="revoke-keys"
Nov 02 03:16:58.628909 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:58.628895    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9a2f2b6-c3a0-414f-89fa-50ce0279fb82" containerName="revoke-keys"
Nov 02 03:16:58.657034 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:58.650294    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-858236/distinct-w3-858236-kserve-df6dbd64b-qbk9g"
Nov 02 03:16:58.717778 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:58.716686    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-858236/distinct-w3-858236-kserve-df6dbd64b-qbk9g"]
Nov 02 03:16:58.717778 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:58.717229    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/719bb9e5-ea72-4782-bb17-034b909f874d-tls-certs\") pod \"distinct-w3-858236-kserve-df6dbd64b-qbk9g\" (UID: \"719bb9e5-ea72-4782-bb17-034b909f874d\") " pod="e2e-models-e2e-worker-w3-858236/distinct-w3-858236-kserve-df6dbd64b-qbk9g"
Nov 02 03:16:58.717778 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:58.717287    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/719bb9e5-ea72-4782-bb17-034b909f874d-tmp-dir\") pod \"distinct-w3-858236-kserve-df6dbd64b-qbk9g\" (UID: \"719bb9e5-ea72-4782-bb17-034b909f874d\") " pod="e2e-models-e2e-worker-w3-858236/distinct-w3-858236-kserve-df6dbd64b-qbk9g"
Nov 02 03:16:58.717778 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:58.717440    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/719bb9e5-ea72-4782-bb17-034b909f874d-home\") pod \"distinct-w3-858236-kserve-df6dbd64b-qbk9g\" (UID: \"719bb9e5-ea72-4782-bb17-034b909f874d\") " pod="e2e-models-e2e-worker-w3-858236/distinct-w3-858236-kserve-df6dbd64b-qbk9g"
Nov 02 03:16:58.717778 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:58.717493    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/719bb9e5-ea72-4782-bb17-034b909f874d-dshm\") pod \"distinct-w3-858236-kserve-df6dbd64b-qbk9g\" (UID: \"719bb9e5-ea72-4782-bb17-034b909f874d\") " pod="e2e-models-e2e-worker-w3-858236/distinct-w3-858236-kserve-df6dbd64b-qbk9g"
Nov 02 03:16:58.717778 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:58.717517    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66dpg\" (UniqueName: \"kubernetes.io/projected/719bb9e5-ea72-4782-bb17-034b909f874d-kube-api-access-66dpg\") pod \"distinct-w3-858236-kserve-df6dbd64b-qbk9g\" (UID: \"719bb9e5-ea72-4782-bb17-034b909f874d\") " pod="e2e-models-e2e-worker-w3-858236/distinct-w3-858236-kserve-df6dbd64b-qbk9g"
Nov 02 03:16:58.717778 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:58.717609    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/719bb9e5-ea72-4782-bb17-034b909f874d-model-cache\") pod \"distinct-w3-858236-kserve-df6dbd64b-qbk9g\" (UID: \"719bb9e5-ea72-4782-bb17-034b909f874d\") " pod="e2e-models-e2e-worker-w3-858236/distinct-w3-858236-kserve-df6dbd64b-qbk9g"
Nov 02 03:16:58.760085 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:58.739849    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-858236\"/\"distinct-w3-858236-kserve-self-signed-certs\""
Nov 02 03:16:58.760085 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:58.739951    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-858236\"/\"default-dockercfg-qtpc8\""
Nov 02 03:16:58.760085 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:58.758309    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-858236\"/\"openshift-service-ca.crt\""
Nov 02 03:16:58.760085 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:58.758383    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-858236\"/\"kube-root-ca.crt\""
Nov 02 03:16:58.818860 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:58.818821    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/719bb9e5-ea72-4782-bb17-034b909f874d-dshm\") pod \"distinct-w3-858236-kserve-df6dbd64b-qbk9g\" (UID: \"719bb9e5-ea72-4782-bb17-034b909f874d\") " pod="e2e-models-e2e-worker-w3-858236/distinct-w3-858236-kserve-df6dbd64b-qbk9g"
Nov 02 03:16:58.819038 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:58.818875    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66dpg\" (UniqueName: \"kubernetes.io/projected/719bb9e5-ea72-4782-bb17-034b909f874d-kube-api-access-66dpg\") pod \"distinct-w3-858236-kserve-df6dbd64b-qbk9g\" (UID: \"719bb9e5-ea72-4782-bb17-034b909f874d\") " pod="e2e-models-e2e-worker-w3-858236/distinct-w3-858236-kserve-df6dbd64b-qbk9g"
Nov 02 03:16:58.819038 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:58.818973    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/719bb9e5-ea72-4782-bb17-034b909f874d-model-cache\") pod \"distinct-w3-858236-kserve-df6dbd64b-qbk9g\" (UID: \"719bb9e5-ea72-4782-bb17-034b909f874d\") " pod="e2e-models-e2e-worker-w3-858236/distinct-w3-858236-kserve-df6dbd64b-qbk9g"
Nov 02 03:16:58.819119 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:58.819066    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/719bb9e5-ea72-4782-bb17-034b909f874d-tls-certs\") pod \"distinct-w3-858236-kserve-df6dbd64b-qbk9g\" (UID: \"719bb9e5-ea72-4782-bb17-034b909f874d\") " pod="e2e-models-e2e-worker-w3-858236/distinct-w3-858236-kserve-df6dbd64b-qbk9g"
Nov 02 03:16:58.819151 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:58.819116    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/719bb9e5-ea72-4782-bb17-034b909f874d-tmp-dir\") pod \"distinct-w3-858236-kserve-df6dbd64b-qbk9g\" (UID: \"719bb9e5-ea72-4782-bb17-034b909f874d\") " pod="e2e-models-e2e-worker-w3-858236/distinct-w3-858236-kserve-df6dbd64b-qbk9g"
Nov 02 03:16:58.819300 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:58.819278    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/719bb9e5-ea72-4782-bb17-034b909f874d-home\") pod \"distinct-w3-858236-kserve-df6dbd64b-qbk9g\" (UID: \"719bb9e5-ea72-4782-bb17-034b909f874d\") " pod="e2e-models-e2e-worker-w3-858236/distinct-w3-858236-kserve-df6dbd64b-qbk9g"
Nov 02 03:16:58.819955 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:58.819933    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/719bb9e5-ea72-4782-bb17-034b909f874d-tmp-dir\") pod \"distinct-w3-858236-kserve-df6dbd64b-qbk9g\" (UID: \"719bb9e5-ea72-4782-bb17-034b909f874d\") " pod="e2e-models-e2e-worker-w3-858236/distinct-w3-858236-kserve-df6dbd64b-qbk9g"
Nov 02 03:16:58.820062 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:58.819978    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/719bb9e5-ea72-4782-bb17-034b909f874d-model-cache\") pod \"distinct-w3-858236-kserve-df6dbd64b-qbk9g\" (UID: \"719bb9e5-ea72-4782-bb17-034b909f874d\") " pod="e2e-models-e2e-worker-w3-858236/distinct-w3-858236-kserve-df6dbd64b-qbk9g"
Nov 02 03:16:58.820248 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:58.820224    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/719bb9e5-ea72-4782-bb17-034b909f874d-home\") pod \"distinct-w3-858236-kserve-df6dbd64b-qbk9g\" (UID: \"719bb9e5-ea72-4782-bb17-034b909f874d\") " pod="e2e-models-e2e-worker-w3-858236/distinct-w3-858236-kserve-df6dbd64b-qbk9g"
Nov 02 03:16:58.822190 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:58.822163    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/719bb9e5-ea72-4782-bb17-034b909f874d-dshm\") pod \"distinct-w3-858236-kserve-df6dbd64b-qbk9g\" (UID: \"719bb9e5-ea72-4782-bb17-034b909f874d\") " pod="e2e-models-e2e-worker-w3-858236/distinct-w3-858236-kserve-df6dbd64b-qbk9g"
Nov 02 03:16:58.822598 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:58.822525    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/719bb9e5-ea72-4782-bb17-034b909f874d-tls-certs\") pod \"distinct-w3-858236-kserve-df6dbd64b-qbk9g\" (UID: \"719bb9e5-ea72-4782-bb17-034b909f874d\") " pod="e2e-models-e2e-worker-w3-858236/distinct-w3-858236-kserve-df6dbd64b-qbk9g"
Nov 02 03:16:58.887662 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:58.887626    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66dpg\" (UniqueName: \"kubernetes.io/projected/719bb9e5-ea72-4782-bb17-034b909f874d-kube-api-access-66dpg\") pod \"distinct-w3-858236-kserve-df6dbd64b-qbk9g\" (UID: \"719bb9e5-ea72-4782-bb17-034b909f874d\") " pod="e2e-models-e2e-worker-w3-858236/distinct-w3-858236-kserve-df6dbd64b-qbk9g"
Nov 02 03:16:59.014088 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:59.013975    4496 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-zf5k4"
Nov 02 03:16:59.014827 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:59.014809    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-zf5k4"
Nov 02 03:16:59.015712 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:59.015691    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-zf5k4"
Nov 02 03:16:59.022053 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:59.022034    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-858236/distinct-w3-858236-kserve-df6dbd64b-qbk9g"
Nov 02 03:16:59.466784 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:16:59.466751    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod719bb9e5_ea72_4782_bb17_034b909f874d.slice/crio-e4390ffb363b5104dfae4ca0666e21c1e89fd5ef5d82caa3fff7bf9d990a84e5 WatchSource:0}: Error finding container e4390ffb363b5104dfae4ca0666e21c1e89fd5ef5d82caa3fff7bf9d990a84e5: Status 404 returned error can't find the container with id e4390ffb363b5104dfae4ca0666e21c1e89fd5ef5d82caa3fff7bf9d990a84e5
Nov 02 03:16:59.468204 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:59.468176    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-858236/distinct-w3-858236-kserve-df6dbd64b-qbk9g"]
Nov 02 03:16:59.493908 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:59.493885    4496 patch_prober.go:29] interesting pod/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hdjcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body=
Nov 02 03:16:59.494045 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:59.493932    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hdjcq" podUID="37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused"
Nov 02 03:16:59.809184 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:59.809136    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-858236/distinct-w3-858236-kserve-df6dbd64b-qbk9g" event={"ID":"719bb9e5-ea72-4782-bb17-034b909f874d","Type":"ContainerStarted","Data":"d8a5a9e207fa197916278177f145dd27a6a162de125fc82e25b9d35d237a634d"}
Nov 02 03:16:59.809184 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:59.809187    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-858236/distinct-w3-858236-kserve-df6dbd64b-qbk9g" event={"ID":"719bb9e5-ea72-4782-bb17-034b909f874d","Type":"ContainerStarted","Data":"e4390ffb363b5104dfae4ca0666e21c1e89fd5ef5d82caa3fff7bf9d990a84e5"}
Nov 02 03:16:59.929818 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:16:59.929752    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-858236/distinct-w3-858236-kserve-df6dbd64b-qbk9g" podStartSLOduration=1.9297336409999999 podStartE2EDuration="1.929733641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:16:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:16:59.925393696 +0000 UTC m=+1760.504001077" watchObservedRunningTime="2026-11-02 03:16:59.929733641 +0000 UTC m=+1760.508341022"
Nov 02 03:17:00.022828 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:00.022794    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-858236/distinct-w3-858236-kserve-df6dbd64b-qbk9g"
Nov 02 03:17:00.501885 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:00.501854    4496 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hdjcq"
Nov 02 03:17:00.502334 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:00.502306    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hdjcq"
Nov 02 03:17:00.503644 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:00.503623    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hdjcq"
Nov 02 03:17:01.934441 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:01.934406    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cea9a103-k9xxp"]
Nov 02 03:17:01.938730 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:01.938712    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cea9a103-k9xxp"
Nov 02 03:17:01.994438 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:01.994406    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cea9a103-k9xxp"]
Nov 02 03:17:02.039925 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:02.039874    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 03:17:02.039925 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:02.039918    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 03:17:02.040166 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:02.039985    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9z5bk\""
Nov 02 03:17:02.042384 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:02.042369    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 03:17:02.059993 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:02.059953    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e29ac61-8a7e-41b7-ac1f-2067cd40dec9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cea9a103-k9xxp\" (UID: \"7e29ac61-8a7e-41b7-ac1f-2067cd40dec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cea9a103-k9xxp"
Nov 02 03:17:02.060095 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:02.060048    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e29ac61-8a7e-41b7-ac1f-2067cd40dec9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cea9a103-k9xxp\" (UID: \"7e29ac61-8a7e-41b7-ac1f-2067cd40dec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cea9a103-k9xxp"
Nov 02 03:17:02.060195 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:02.060181    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cp72w\" (UniqueName: \"kubernetes.io/projected/7e29ac61-8a7e-41b7-ac1f-2067cd40dec9-kube-api-access-cp72w\") pod \"maas-api-revoke-sub-cea9a103-k9xxp\" (UID: \"7e29ac61-8a7e-41b7-ac1f-2067cd40dec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cea9a103-k9xxp"
Nov 02 03:17:02.161678 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:02.161641    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cp72w\" (UniqueName: \"kubernetes.io/projected/7e29ac61-8a7e-41b7-ac1f-2067cd40dec9-kube-api-access-cp72w\") pod \"maas-api-revoke-sub-cea9a103-k9xxp\" (UID: \"7e29ac61-8a7e-41b7-ac1f-2067cd40dec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cea9a103-k9xxp"
Nov 02 03:17:02.161868 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:02.161754    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e29ac61-8a7e-41b7-ac1f-2067cd40dec9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cea9a103-k9xxp\" (UID: \"7e29ac61-8a7e-41b7-ac1f-2067cd40dec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cea9a103-k9xxp"
Nov 02 03:17:02.161868 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:02.161819    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e29ac61-8a7e-41b7-ac1f-2067cd40dec9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cea9a103-k9xxp\" (UID: \"7e29ac61-8a7e-41b7-ac1f-2067cd40dec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cea9a103-k9xxp"
Nov 02 03:17:02.162407 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:02.162385    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e29ac61-8a7e-41b7-ac1f-2067cd40dec9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cea9a103-k9xxp\" (UID: \"7e29ac61-8a7e-41b7-ac1f-2067cd40dec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cea9a103-k9xxp"
Nov 02 03:17:02.164117 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:02.164098    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e29ac61-8a7e-41b7-ac1f-2067cd40dec9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cea9a103-k9xxp\" (UID: \"7e29ac61-8a7e-41b7-ac1f-2067cd40dec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cea9a103-k9xxp"
Nov 02 03:17:02.220299 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:02.220232    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cp72w\" (UniqueName: \"kubernetes.io/projected/7e29ac61-8a7e-41b7-ac1f-2067cd40dec9-kube-api-access-cp72w\") pod \"maas-api-revoke-sub-cea9a103-k9xxp\" (UID: \"7e29ac61-8a7e-41b7-ac1f-2067cd40dec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cea9a103-k9xxp"
Nov 02 03:17:02.253140 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:02.253110    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cea9a103-k9xxp"
Nov 02 03:17:02.634413 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:17:02.634379    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e29ac61_8a7e_41b7_ac1f_2067cd40dec9.slice/crio-1b06b2142b4747d7fe3811237c8808b0872c60f020789c577fe6f477179513d2 WatchSource:0}: Error finding container 1b06b2142b4747d7fe3811237c8808b0872c60f020789c577fe6f477179513d2: Status 404 returned error can't find the container with id 1b06b2142b4747d7fe3811237c8808b0872c60f020789c577fe6f477179513d2
Nov 02 03:17:02.634840 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:02.634818    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cea9a103-k9xxp"]
Nov 02 03:17:02.831763 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:02.831727    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cea9a103-k9xxp" event={"ID":"7e29ac61-8a7e-41b7-ac1f-2067cd40dec9","Type":"ContainerStarted","Data":"1b06b2142b4747d7fe3811237c8808b0872c60f020789c577fe6f477179513d2"}
Nov 02 03:17:03.342070 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:03.342031    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4b46d8f-m97rk"]
Nov 02 03:17:03.346200 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:03.346183    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4b46d8f-m97rk"
Nov 02 03:17:03.403733 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:03.403699    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4b46d8f-m97rk"]
Nov 02 03:17:03.477958 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:03.477925    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgk9v\" (UniqueName: \"kubernetes.io/projected/d847479f-5794-498f-81b8-4834ee8d6b6b-kube-api-access-hgk9v\") pod \"maas-api-revoke-sub-b4b46d8f-m97rk\" (UID: \"d847479f-5794-498f-81b8-4834ee8d6b6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4b46d8f-m97rk"
Nov 02 03:17:03.478171 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:03.477983    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d847479f-5794-498f-81b8-4834ee8d6b6b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b4b46d8f-m97rk\" (UID: \"d847479f-5794-498f-81b8-4834ee8d6b6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4b46d8f-m97rk"
Nov 02 03:17:03.478171 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:03.478106    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d847479f-5794-498f-81b8-4834ee8d6b6b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b4b46d8f-m97rk\" (UID: \"d847479f-5794-498f-81b8-4834ee8d6b6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4b46d8f-m97rk"
Nov 02 03:17:03.527059 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:17:03.527001    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 03:17:03.579101 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:03.579068    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d847479f-5794-498f-81b8-4834ee8d6b6b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b4b46d8f-m97rk\" (UID: \"d847479f-5794-498f-81b8-4834ee8d6b6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4b46d8f-m97rk"
Nov 02 03:17:03.579277 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:03.579148    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d847479f-5794-498f-81b8-4834ee8d6b6b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b4b46d8f-m97rk\" (UID: \"d847479f-5794-498f-81b8-4834ee8d6b6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4b46d8f-m97rk"
Nov 02 03:17:03.579355 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:03.579331    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgk9v\" (UniqueName: \"kubernetes.io/projected/d847479f-5794-498f-81b8-4834ee8d6b6b-kube-api-access-hgk9v\") pod \"maas-api-revoke-sub-b4b46d8f-m97rk\" (UID: \"d847479f-5794-498f-81b8-4834ee8d6b6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4b46d8f-m97rk"
Nov 02 03:17:03.579923 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:03.579904    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d847479f-5794-498f-81b8-4834ee8d6b6b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b4b46d8f-m97rk\" (UID: \"d847479f-5794-498f-81b8-4834ee8d6b6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4b46d8f-m97rk"
Nov 02 03:17:03.581554 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:03.581533    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d847479f-5794-498f-81b8-4834ee8d6b6b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b4b46d8f-m97rk\" (UID: \"d847479f-5794-498f-81b8-4834ee8d6b6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4b46d8f-m97rk"
Nov 02 03:17:03.637591 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:03.637504    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgk9v\" (UniqueName: \"kubernetes.io/projected/d847479f-5794-498f-81b8-4834ee8d6b6b-kube-api-access-hgk9v\") pod \"maas-api-revoke-sub-b4b46d8f-m97rk\" (UID: \"d847479f-5794-498f-81b8-4834ee8d6b6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4b46d8f-m97rk"
Nov 02 03:17:03.660321 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:03.660285    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4b46d8f-m97rk"
Nov 02 03:17:03.840737 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:03.840703    4496 generic.go:356] "Generic (PLEG): container finished" podID="7e29ac61-8a7e-41b7-ac1f-2067cd40dec9" containerID="c3da87de80b80e9357eda1f208e5baf3b419fce4ea1622766d27caee29ad4710" exitCode=0
Nov 02 03:17:03.840888 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:03.840790    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cea9a103-k9xxp" event={"ID":"7e29ac61-8a7e-41b7-ac1f-2067cd40dec9","Type":"ContainerDied","Data":"c3da87de80b80e9357eda1f208e5baf3b419fce4ea1622766d27caee29ad4710"}
Nov 02 03:17:04.068167 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:04.068137    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4b46d8f-m97rk"]
Nov 02 03:17:04.068786 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:17:04.068760    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd847479f_5794_498f_81b8_4834ee8d6b6b.slice/crio-8c3f3cc13a99967443794aa618d5e109ff121ce7c92d1154939a28491123c88e WatchSource:0}: Error finding container 8c3f3cc13a99967443794aa618d5e109ff121ce7c92d1154939a28491123c88e: Status 404 returned error can't find the container with id 8c3f3cc13a99967443794aa618d5e109ff121ce7c92d1154939a28491123c88e
Nov 02 03:17:04.852901 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:04.852869    4496 generic.go:356] "Generic (PLEG): container finished" podID="d847479f-5794-498f-81b8-4834ee8d6b6b" containerID="7a3938a92e4156c9ff8203a871f406e24ee6e45d44fd1e8afee5d5610c40376a" exitCode=0
Nov 02 03:17:04.853297 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:04.853005    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4b46d8f-m97rk" event={"ID":"d847479f-5794-498f-81b8-4834ee8d6b6b","Type":"ContainerDied","Data":"7a3938a92e4156c9ff8203a871f406e24ee6e45d44fd1e8afee5d5610c40376a"}
Nov 02 03:17:04.853297 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:04.853069    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4b46d8f-m97rk" event={"ID":"d847479f-5794-498f-81b8-4834ee8d6b6b","Type":"ContainerStarted","Data":"8c3f3cc13a99967443794aa618d5e109ff121ce7c92d1154939a28491123c88e"}
Nov 02 03:17:05.023868 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:05.023840    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cea9a103-k9xxp"
Nov 02 03:17:05.051406 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:05.051376    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29677b55-wmfd7"]
Nov 02 03:17:05.055855 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:05.055831    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29677b55-wmfd7"]
Nov 02 03:17:05.099213 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:05.099135    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e29ac61-8a7e-41b7-ac1f-2067cd40dec9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e29ac61-8a7e-41b7-ac1f-2067cd40dec9-maas-api-xks-ca\") pod \"7e29ac61-8a7e-41b7-ac1f-2067cd40dec9\" (UID: \"7e29ac61-8a7e-41b7-ac1f-2067cd40dec9\") "
Nov 02 03:17:05.099349 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:05.099222    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e29ac61-8a7e-41b7-ac1f-2067cd40dec9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e29ac61-8a7e-41b7-ac1f-2067cd40dec9-maas-api-openshift-ca\") pod \"7e29ac61-8a7e-41b7-ac1f-2067cd40dec9\" (UID: \"7e29ac61-8a7e-41b7-ac1f-2067cd40dec9\") "
Nov 02 03:17:05.099349 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:05.099328    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e29ac61-8a7e-41b7-ac1f-2067cd40dec9-kube-api-access-cp72w\" (UniqueName: \"kubernetes.io/projected/7e29ac61-8a7e-41b7-ac1f-2067cd40dec9-kube-api-access-cp72w\") pod \"7e29ac61-8a7e-41b7-ac1f-2067cd40dec9\" (UID: \"7e29ac61-8a7e-41b7-ac1f-2067cd40dec9\") "
Nov 02 03:17:05.099570 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:05.099548    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e29ac61-8a7e-41b7-ac1f-2067cd40dec9-maas-api-openshift-ca" pod "7e29ac61-8a7e-41b7-ac1f-2067cd40dec9" (UID: "7e29ac61-8a7e-41b7-ac1f-2067cd40dec9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:17:05.099805 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:05.099786    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e29ac61-8a7e-41b7-ac1f-2067cd40dec9-maas-api-openshift-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:17:05.101425 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:05.101398    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e29ac61-8a7e-41b7-ac1f-2067cd40dec9-maas-api-xks-ca" pod "7e29ac61-8a7e-41b7-ac1f-2067cd40dec9" (UID: "7e29ac61-8a7e-41b7-ac1f-2067cd40dec9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 03:17:05.101786 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:05.101757    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e29ac61-8a7e-41b7-ac1f-2067cd40dec9-kube-api-access-cp72w" pod "7e29ac61-8a7e-41b7-ac1f-2067cd40dec9" (UID: "7e29ac61-8a7e-41b7-ac1f-2067cd40dec9"). InnerVolumeSpecName "kube-api-access-cp72w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:17:05.123250 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:05.123230    4496 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-398828/distinct-2-w4-398828-kserve-545449d49b-6mg4b"
Nov 02 03:17:05.135741 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:05.135723    4496 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-398828/distinct-2-w4-398828-kserve-545449d49b-6mg4b"
Nov 02 03:17:05.200708 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:05.200673    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e29ac61-8a7e-41b7-ac1f-2067cd40dec9-maas-api-xks-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:17:05.200708 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:05.200701    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cp72w\" (UniqueName: \"kubernetes.io/projected/7e29ac61-8a7e-41b7-ac1f-2067cd40dec9-kube-api-access-cp72w\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:17:05.575594 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:05.575556    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04fb8775-f59b-44c4-a384-92f3d3a09497" path="/var/lib/kubelet/pods/04fb8775-f59b-44c4-a384-92f3d3a09497/volumes"
Nov 02 03:17:05.864527 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:05.864442    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cea9a103-k9xxp"
Nov 02 03:17:05.864527 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:05.864468    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cea9a103-k9xxp" event={"ID":"7e29ac61-8a7e-41b7-ac1f-2067cd40dec9","Type":"ContainerDied","Data":"1b06b2142b4747d7fe3811237c8808b0872c60f020789c577fe6f477179513d2"}
Nov 02 03:17:05.864968 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:05.864529    4496 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1b06b2142b4747d7fe3811237c8808b0872c60f020789c577fe6f477179513d2"
Nov 02 03:17:05.881841 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:05.881814    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-398828/distinct-2-w4-398828-kserve-545449d49b-6mg4b"
Nov 02 03:17:06.029288 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:06.029267    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4b46d8f-m97rk"
Nov 02 03:17:06.111321 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:06.111289    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d847479f-5794-498f-81b8-4834ee8d6b6b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d847479f-5794-498f-81b8-4834ee8d6b6b-maas-api-openshift-ca\") pod \"d847479f-5794-498f-81b8-4834ee8d6b6b\" (UID: \"d847479f-5794-498f-81b8-4834ee8d6b6b\") "
Nov 02 03:17:06.111526 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:06.111479    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d847479f-5794-498f-81b8-4834ee8d6b6b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d847479f-5794-498f-81b8-4834ee8d6b6b-maas-api-xks-ca\") pod \"d847479f-5794-498f-81b8-4834ee8d6b6b\" (UID: \"d847479f-5794-498f-81b8-4834ee8d6b6b\") "
Nov 02 03:17:06.111605 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:06.111527    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d847479f-5794-498f-81b8-4834ee8d6b6b-kube-api-access-hgk9v\" (UniqueName: \"kubernetes.io/projected/d847479f-5794-498f-81b8-4834ee8d6b6b-kube-api-access-hgk9v\") pod \"d847479f-5794-498f-81b8-4834ee8d6b6b\" (UID: \"d847479f-5794-498f-81b8-4834ee8d6b6b\") "
Nov 02 03:17:06.111665 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:06.111633    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d847479f-5794-498f-81b8-4834ee8d6b6b-maas-api-openshift-ca" pod "d847479f-5794-498f-81b8-4834ee8d6b6b" (UID: "d847479f-5794-498f-81b8-4834ee8d6b6b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:17:06.112000 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:06.111975    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d847479f-5794-498f-81b8-4834ee8d6b6b-maas-api-openshift-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:17:06.113517 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:06.113495    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d847479f-5794-498f-81b8-4834ee8d6b6b-maas-api-xks-ca" pod "d847479f-5794-498f-81b8-4834ee8d6b6b" (UID: "d847479f-5794-498f-81b8-4834ee8d6b6b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 03:17:06.114122 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:06.114096    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d847479f-5794-498f-81b8-4834ee8d6b6b-kube-api-access-hgk9v" pod "d847479f-5794-498f-81b8-4834ee8d6b6b" (UID: "d847479f-5794-498f-81b8-4834ee8d6b6b"). InnerVolumeSpecName "kube-api-access-hgk9v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:17:06.213529 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:06.213499    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d847479f-5794-498f-81b8-4834ee8d6b6b-maas-api-xks-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:17:06.213529 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:06.213523    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgk9v\" (UniqueName: \"kubernetes.io/projected/d847479f-5794-498f-81b8-4834ee8d6b6b-kube-api-access-hgk9v\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:17:06.886553 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:06.886514    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4b46d8f-m97rk"
Nov 02 03:17:06.886553 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:06.886514    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4b46d8f-m97rk" event={"ID":"d847479f-5794-498f-81b8-4834ee8d6b6b","Type":"ContainerDied","Data":"8c3f3cc13a99967443794aa618d5e109ff121ce7c92d1154939a28491123c88e"}
Nov 02 03:17:06.887138 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:06.886627    4496 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c3f3cc13a99967443794aa618d5e109ff121ce7c92d1154939a28491123c88e"
Nov 02 03:17:09.033954 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:09.033082    4496 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-858236/distinct-w3-858236-kserve-df6dbd64b-qbk9g"
Nov 02 03:17:09.130160 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:09.130125    4496 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-858236/distinct-w3-858236-kserve-df6dbd64b-qbk9g"
Nov 02 03:17:09.993307 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:09.993276    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-858236/distinct-w3-858236-kserve-df6dbd64b-qbk9g"
Nov 02 03:17:14.088316 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:14.088279    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b77760e-5fwc5"]
Nov 02 03:17:14.088920 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:14.088904    4496 cpu_manager.go:414] "containerMap: removing container" podUID="7e29ac61-8a7e-41b7-ac1f-2067cd40dec9" containerName="revoke-keys"
Nov 02 03:17:14.088963 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:14.088921    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e29ac61-8a7e-41b7-ac1f-2067cd40dec9" containerName="revoke-keys"
Nov 02 03:17:14.088963 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:14.088954    4496 cpu_manager.go:414] "containerMap: removing container" podUID="d847479f-5794-498f-81b8-4834ee8d6b6b" containerName="revoke-keys"
Nov 02 03:17:14.088963 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:14.088959    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d847479f-5794-498f-81b8-4834ee8d6b6b" containerName="revoke-keys"
Nov 02 03:17:14.089103 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:14.089092    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="d847479f-5794-498f-81b8-4834ee8d6b6b" containerName="revoke-keys"
Nov 02 03:17:14.089138 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:14.089111    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e29ac61-8a7e-41b7-ac1f-2067cd40dec9" containerName="revoke-keys"
Nov 02 03:17:14.093934 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:14.093911    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b77760e-5fwc5"
Nov 02 03:17:14.147640 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:14.147608    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b77760e-5fwc5"]
Nov 02 03:17:14.179900 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:14.179865    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86fbde4b-8928-459a-bcc4-764f91c15ac4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b77760e-5fwc5\" (UID: \"86fbde4b-8928-459a-bcc4-764f91c15ac4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b77760e-5fwc5"
Nov 02 03:17:14.180122 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:14.180071    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9n5h4\" (UniqueName: \"kubernetes.io/projected/86fbde4b-8928-459a-bcc4-764f91c15ac4-kube-api-access-9n5h4\") pod \"maas-api-revoke-sub-9b77760e-5fwc5\" (UID: \"86fbde4b-8928-459a-bcc4-764f91c15ac4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b77760e-5fwc5"
Nov 02 03:17:14.180308 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:14.180286    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86fbde4b-8928-459a-bcc4-764f91c15ac4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b77760e-5fwc5\" (UID: \"86fbde4b-8928-459a-bcc4-764f91c15ac4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b77760e-5fwc5"
Nov 02 03:17:14.193488 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:14.193453    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 03:17:14.193629 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:14.193504    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 03:17:14.193629 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:14.193569    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 03:17:14.193713 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:14.193637    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9z5bk\""
Nov 02 03:17:14.281186 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:14.281149    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86fbde4b-8928-459a-bcc4-764f91c15ac4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b77760e-5fwc5\" (UID: \"86fbde4b-8928-459a-bcc4-764f91c15ac4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b77760e-5fwc5"
Nov 02 03:17:14.281345 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:14.281225    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86fbde4b-8928-459a-bcc4-764f91c15ac4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b77760e-5fwc5\" (UID: \"86fbde4b-8928-459a-bcc4-764f91c15ac4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b77760e-5fwc5"
Nov 02 03:17:14.281345 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:14.281298    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9n5h4\" (UniqueName: \"kubernetes.io/projected/86fbde4b-8928-459a-bcc4-764f91c15ac4-kube-api-access-9n5h4\") pod \"maas-api-revoke-sub-9b77760e-5fwc5\" (UID: \"86fbde4b-8928-459a-bcc4-764f91c15ac4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b77760e-5fwc5"
Nov 02 03:17:14.281875 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:14.281854    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86fbde4b-8928-459a-bcc4-764f91c15ac4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b77760e-5fwc5\" (UID: \"86fbde4b-8928-459a-bcc4-764f91c15ac4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b77760e-5fwc5"
Nov 02 03:17:14.283513 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:14.283495    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86fbde4b-8928-459a-bcc4-764f91c15ac4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b77760e-5fwc5\" (UID: \"86fbde4b-8928-459a-bcc4-764f91c15ac4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b77760e-5fwc5"
Nov 02 03:17:14.339152 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:14.339086    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9n5h4\" (UniqueName: \"kubernetes.io/projected/86fbde4b-8928-459a-bcc4-764f91c15ac4-kube-api-access-9n5h4\") pod \"maas-api-revoke-sub-9b77760e-5fwc5\" (UID: \"86fbde4b-8928-459a-bcc4-764f91c15ac4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b77760e-5fwc5"
Nov 02 03:17:14.408587 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:14.408557    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b77760e-5fwc5"
Nov 02 03:17:14.785144 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:17:14.785113    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86fbde4b_8928_459a_bcc4_764f91c15ac4.slice/crio-b3fc03e67ae7eb24a972a63bfb77767f89ff6433cebb0b80b13d2cb3ece0cfad WatchSource:0}: Error finding container b3fc03e67ae7eb24a972a63bfb77767f89ff6433cebb0b80b13d2cb3ece0cfad: Status 404 returned error can't find the container with id b3fc03e67ae7eb24a972a63bfb77767f89ff6433cebb0b80b13d2cb3ece0cfad
Nov 02 03:17:14.785513 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:14.785490    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b77760e-5fwc5"]
Nov 02 03:17:15.018448 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:15.018412    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b77760e-5fwc5" event={"ID":"86fbde4b-8928-459a-bcc4-764f91c15ac4","Type":"ContainerStarted","Data":"b3fc03e67ae7eb24a972a63bfb77767f89ff6433cebb0b80b13d2cb3ece0cfad"}
Nov 02 03:17:16.027225 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:16.027195    4496 generic.go:356] "Generic (PLEG): container finished" podID="86fbde4b-8928-459a-bcc4-764f91c15ac4" containerID="40e9b46633730fe9758c269f0d869e2fb8335fbd7b402e3731cc369ab268b69b" exitCode=0
Nov 02 03:17:16.027620 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:16.027279    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b77760e-5fwc5" event={"ID":"86fbde4b-8928-459a-bcc4-764f91c15ac4","Type":"ContainerDied","Data":"40e9b46633730fe9758c269f0d869e2fb8335fbd7b402e3731cc369ab268b69b"}
Nov 02 03:17:17.204914 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:17.204890    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b77760e-5fwc5"
Nov 02 03:17:17.317434 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:17.317357    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86fbde4b-8928-459a-bcc4-764f91c15ac4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86fbde4b-8928-459a-bcc4-764f91c15ac4-maas-api-xks-ca\") pod \"86fbde4b-8928-459a-bcc4-764f91c15ac4\" (UID: \"86fbde4b-8928-459a-bcc4-764f91c15ac4\") "
Nov 02 03:17:17.317580 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:17.317449    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86fbde4b-8928-459a-bcc4-764f91c15ac4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86fbde4b-8928-459a-bcc4-764f91c15ac4-maas-api-openshift-ca\") pod \"86fbde4b-8928-459a-bcc4-764f91c15ac4\" (UID: \"86fbde4b-8928-459a-bcc4-764f91c15ac4\") "
Nov 02 03:17:17.317580 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:17.317517    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86fbde4b-8928-459a-bcc4-764f91c15ac4-kube-api-access-9n5h4\" (UniqueName: \"kubernetes.io/projected/86fbde4b-8928-459a-bcc4-764f91c15ac4-kube-api-access-9n5h4\") pod \"86fbde4b-8928-459a-bcc4-764f91c15ac4\" (UID: \"86fbde4b-8928-459a-bcc4-764f91c15ac4\") "
Nov 02 03:17:17.317828 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:17.317795    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86fbde4b-8928-459a-bcc4-764f91c15ac4-maas-api-openshift-ca" pod "86fbde4b-8928-459a-bcc4-764f91c15ac4" (UID: "86fbde4b-8928-459a-bcc4-764f91c15ac4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:17:17.319530 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:17.319506    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86fbde4b-8928-459a-bcc4-764f91c15ac4-kube-api-access-9n5h4" pod "86fbde4b-8928-459a-bcc4-764f91c15ac4" (UID: "86fbde4b-8928-459a-bcc4-764f91c15ac4"). InnerVolumeSpecName "kube-api-access-9n5h4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:17:17.319530 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:17.319521    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86fbde4b-8928-459a-bcc4-764f91c15ac4-maas-api-xks-ca" pod "86fbde4b-8928-459a-bcc4-764f91c15ac4" (UID: "86fbde4b-8928-459a-bcc4-764f91c15ac4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 03:17:17.418478 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:17.418445    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86fbde4b-8928-459a-bcc4-764f91c15ac4-maas-api-openshift-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:17:17.418478 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:17.418475    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9n5h4\" (UniqueName: \"kubernetes.io/projected/86fbde4b-8928-459a-bcc4-764f91c15ac4-kube-api-access-9n5h4\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:17:17.418669 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:17.418490    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86fbde4b-8928-459a-bcc4-764f91c15ac4-maas-api-xks-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:17:18.044078 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:18.044043    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b77760e-5fwc5"
Nov 02 03:17:18.044257 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:18.044077    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b77760e-5fwc5" event={"ID":"86fbde4b-8928-459a-bcc4-764f91c15ac4","Type":"ContainerDied","Data":"b3fc03e67ae7eb24a972a63bfb77767f89ff6433cebb0b80b13d2cb3ece0cfad"}
Nov 02 03:17:18.044257 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:18.044134    4496 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b3fc03e67ae7eb24a972a63bfb77767f89ff6433cebb0b80b13d2cb3ece0cfad"
Nov 02 03:17:19.012756 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:19.012725    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69fd8800-z6r62"]
Nov 02 03:17:19.013348 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:19.013333    4496 cpu_manager.go:414] "containerMap: removing container" podUID="86fbde4b-8928-459a-bcc4-764f91c15ac4" containerName="revoke-keys"
Nov 02 03:17:19.013396 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:19.013348    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86fbde4b-8928-459a-bcc4-764f91c15ac4" containerName="revoke-keys"
Nov 02 03:17:19.013478 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:19.013462    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="86fbde4b-8928-459a-bcc4-764f91c15ac4" containerName="revoke-keys"
Nov 02 03:17:19.017041 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:19.016984    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69fd8800-z6r62"
Nov 02 03:17:19.072864 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:19.072833    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69fd8800-z6r62"]
Nov 02 03:17:19.118211 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:19.118178    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 03:17:19.118376 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:19.118260    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 03:17:19.118376 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:19.118281    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 03:17:19.120595 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:19.120576    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9z5bk\""
Nov 02 03:17:19.139282 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:19.139257    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b852e4a-9db3-499e-8cc5-acd426d9247d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69fd8800-z6r62\" (UID: \"8b852e4a-9db3-499e-8cc5-acd426d9247d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69fd8800-z6r62"
Nov 02 03:17:19.139397 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:19.139317    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhmb2\" (UniqueName: \"kubernetes.io/projected/8b852e4a-9db3-499e-8cc5-acd426d9247d-kube-api-access-mhmb2\") pod \"maas-api-revoke-sub-69fd8800-z6r62\" (UID: \"8b852e4a-9db3-499e-8cc5-acd426d9247d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69fd8800-z6r62"
Nov 02 03:17:19.139439 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:19.139413    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b852e4a-9db3-499e-8cc5-acd426d9247d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69fd8800-z6r62\" (UID: \"8b852e4a-9db3-499e-8cc5-acd426d9247d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69fd8800-z6r62"
Nov 02 03:17:19.240232 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:19.240204    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b852e4a-9db3-499e-8cc5-acd426d9247d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69fd8800-z6r62\" (UID: \"8b852e4a-9db3-499e-8cc5-acd426d9247d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69fd8800-z6r62"
Nov 02 03:17:19.240363 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:19.240265    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhmb2\" (UniqueName: \"kubernetes.io/projected/8b852e4a-9db3-499e-8cc5-acd426d9247d-kube-api-access-mhmb2\") pod \"maas-api-revoke-sub-69fd8800-z6r62\" (UID: \"8b852e4a-9db3-499e-8cc5-acd426d9247d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69fd8800-z6r62"
Nov 02 03:17:19.240363 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:19.240308    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b852e4a-9db3-499e-8cc5-acd426d9247d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69fd8800-z6r62\" (UID: \"8b852e4a-9db3-499e-8cc5-acd426d9247d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69fd8800-z6r62"
Nov 02 03:17:19.240848 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:19.240822    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b852e4a-9db3-499e-8cc5-acd426d9247d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69fd8800-z6r62\" (UID: \"8b852e4a-9db3-499e-8cc5-acd426d9247d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69fd8800-z6r62"
Nov 02 03:17:19.242758 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:19.242734    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b852e4a-9db3-499e-8cc5-acd426d9247d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69fd8800-z6r62\" (UID: \"8b852e4a-9db3-499e-8cc5-acd426d9247d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69fd8800-z6r62"
Nov 02 03:17:19.297726 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:19.297657    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhmb2\" (UniqueName: \"kubernetes.io/projected/8b852e4a-9db3-499e-8cc5-acd426d9247d-kube-api-access-mhmb2\") pod \"maas-api-revoke-sub-69fd8800-z6r62\" (UID: \"8b852e4a-9db3-499e-8cc5-acd426d9247d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69fd8800-z6r62"
Nov 02 03:17:19.331684 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:19.331647    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69fd8800-z6r62"
Nov 02 03:17:19.707770 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:17:19.707741    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b852e4a_9db3_499e_8cc5_acd426d9247d.slice/crio-b4245a33bc8f0954c00042abaa9dc59fafac8b367ef3a3f626e5cd3db49a3d39 WatchSource:0}: Error finding container b4245a33bc8f0954c00042abaa9dc59fafac8b367ef3a3f626e5cd3db49a3d39: Status 404 returned error can't find the container with id b4245a33bc8f0954c00042abaa9dc59fafac8b367ef3a3f626e5cd3db49a3d39
Nov 02 03:17:19.708747 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:19.708726    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69fd8800-z6r62"]
Nov 02 03:17:20.060546 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:20.060513    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69fd8800-z6r62" event={"ID":"8b852e4a-9db3-499e-8cc5-acd426d9247d","Type":"ContainerStarted","Data":"6bb217f3bcc2a4b6f4811595c68285f342fe405464a1d17312a994f99f4dcb90"}
Nov 02 03:17:20.060979 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:20.060595    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69fd8800-z6r62" event={"ID":"8b852e4a-9db3-499e-8cc5-acd426d9247d","Type":"ContainerStarted","Data":"b4245a33bc8f0954c00042abaa9dc59fafac8b367ef3a3f626e5cd3db49a3d39"}
Nov 02 03:17:20.180142 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:20.180079    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69fd8800-z6r62" podStartSLOduration=2.180060882 podStartE2EDuration="2.180060882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:17:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:17:20.173771189 +0000 UTC m=+1780.752378572" watchObservedRunningTime="2026-11-02 03:17:20.180060882 +0000 UTC m=+1780.758668262"
Nov 02 03:17:21.068911 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:21.068881    4496 generic.go:356] "Generic (PLEG): container finished" podID="8b852e4a-9db3-499e-8cc5-acd426d9247d" containerID="6bb217f3bcc2a4b6f4811595c68285f342fe405464a1d17312a994f99f4dcb90" exitCode=0
Nov 02 03:17:21.069309 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:21.068930    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69fd8800-z6r62" event={"ID":"8b852e4a-9db3-499e-8cc5-acd426d9247d","Type":"ContainerDied","Data":"6bb217f3bcc2a4b6f4811595c68285f342fe405464a1d17312a994f99f4dcb90"}
Nov 02 03:17:22.234557 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:22.234521    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69fd8800-z6r62"
Nov 02 03:17:22.376445 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:22.376368    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b852e4a-9db3-499e-8cc5-acd426d9247d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b852e4a-9db3-499e-8cc5-acd426d9247d-maas-api-openshift-ca\") pod \"8b852e4a-9db3-499e-8cc5-acd426d9247d\" (UID: \"8b852e4a-9db3-499e-8cc5-acd426d9247d\") "
Nov 02 03:17:22.376581 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:22.376531    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b852e4a-9db3-499e-8cc5-acd426d9247d-kube-api-access-mhmb2\" (UniqueName: \"kubernetes.io/projected/8b852e4a-9db3-499e-8cc5-acd426d9247d-kube-api-access-mhmb2\") pod \"8b852e4a-9db3-499e-8cc5-acd426d9247d\" (UID: \"8b852e4a-9db3-499e-8cc5-acd426d9247d\") "
Nov 02 03:17:22.376704 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:22.376689    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b852e4a-9db3-499e-8cc5-acd426d9247d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b852e4a-9db3-499e-8cc5-acd426d9247d-maas-api-xks-ca\") pod \"8b852e4a-9db3-499e-8cc5-acd426d9247d\" (UID: \"8b852e4a-9db3-499e-8cc5-acd426d9247d\") "
Nov 02 03:17:22.376738 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:22.376713    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b852e4a-9db3-499e-8cc5-acd426d9247d-maas-api-openshift-ca" pod "8b852e4a-9db3-499e-8cc5-acd426d9247d" (UID: "8b852e4a-9db3-499e-8cc5-acd426d9247d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:17:22.377182 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:22.377158    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b852e4a-9db3-499e-8cc5-acd426d9247d-maas-api-openshift-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:17:22.378612 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:22.378587    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b852e4a-9db3-499e-8cc5-acd426d9247d-kube-api-access-mhmb2" pod "8b852e4a-9db3-499e-8cc5-acd426d9247d" (UID: "8b852e4a-9db3-499e-8cc5-acd426d9247d"). InnerVolumeSpecName "kube-api-access-mhmb2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:17:22.379176 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:22.379156    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b852e4a-9db3-499e-8cc5-acd426d9247d-maas-api-xks-ca" pod "8b852e4a-9db3-499e-8cc5-acd426d9247d" (UID: "8b852e4a-9db3-499e-8cc5-acd426d9247d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 03:17:22.477912 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:22.477878    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhmb2\" (UniqueName: \"kubernetes.io/projected/8b852e4a-9db3-499e-8cc5-acd426d9247d-kube-api-access-mhmb2\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:17:22.477912 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:22.477912    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b852e4a-9db3-499e-8cc5-acd426d9247d-maas-api-xks-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:17:23.086193 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:23.086158    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69fd8800-z6r62" event={"ID":"8b852e4a-9db3-499e-8cc5-acd426d9247d","Type":"ContainerDied","Data":"b4245a33bc8f0954c00042abaa9dc59fafac8b367ef3a3f626e5cd3db49a3d39"}
Nov 02 03:17:23.086193 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:23.086182    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69fd8800-z6r62"
Nov 02 03:17:23.086402 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:23.086208    4496 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b4245a33bc8f0954c00042abaa9dc59fafac8b367ef3a3f626e5cd3db49a3d39"
Nov 02 03:17:24.049684 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:24.049642    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4ec0495-hdpkf"]
Nov 02 03:17:24.105403 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:24.105371    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4ec0495-hdpkf"]
Nov 02 03:17:25.575201 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:25.575168    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48727286-46ff-4aae-bfc5-69236842e26e" path="/var/lib/kubelet/pods/48727286-46ff-4aae-bfc5-69236842e26e/volumes"
Nov 02 03:17:25.701976 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:25.701938    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-lm5rm"]
Nov 02 03:17:25.702573 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:25.702558    4496 cpu_manager.go:414] "containerMap: removing container" podUID="8b852e4a-9db3-499e-8cc5-acd426d9247d" containerName="revoke-keys"
Nov 02 03:17:25.702624 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:25.702574    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b852e4a-9db3-499e-8cc5-acd426d9247d" containerName="revoke-keys"
Nov 02 03:17:25.702675 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:25.702666    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b852e4a-9db3-499e-8cc5-acd426d9247d" containerName="revoke-keys"
Nov 02 03:17:25.707409 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:25.707392    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-lm5rm"
Nov 02 03:17:25.760391 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:25.760364    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-398828-openshift-default-dockercfg-dqq8t\""
Nov 02 03:17:25.815809 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:25.815775    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-lm5rm"]
Nov 02 03:17:25.822798 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:25.822766    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05c9d201-5455-4a37-8382-8188a407e044-istio-podinfo\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-lm5rm\" (UID: \"05c9d201-5455-4a37-8382-8188a407e044\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-lm5rm"
Nov 02 03:17:25.822937 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:25.822837    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05c9d201-5455-4a37-8382-8188a407e044-istio-envoy\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-lm5rm\" (UID: \"05c9d201-5455-4a37-8382-8188a407e044\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-lm5rm"
Nov 02 03:17:25.823002 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:25.822974    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05c9d201-5455-4a37-8382-8188a407e044-workload-socket\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-lm5rm\" (UID: \"05c9d201-5455-4a37-8382-8188a407e044\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-lm5rm"
Nov 02 03:17:25.823131 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:25.823048    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05c9d201-5455-4a37-8382-8188a407e044-credential-socket\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-lm5rm\" (UID: \"05c9d201-5455-4a37-8382-8188a407e044\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-lm5rm"
Nov 02 03:17:25.823131 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:25.823078    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05c9d201-5455-4a37-8382-8188a407e044-istiod-ca-cert\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-lm5rm\" (UID: \"05c9d201-5455-4a37-8382-8188a407e044\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-lm5rm"
Nov 02 03:17:25.823248 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:25.823179    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05c9d201-5455-4a37-8382-8188a407e044-istio-data\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-lm5rm\" (UID: \"05c9d201-5455-4a37-8382-8188a407e044\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-lm5rm"
Nov 02 03:17:25.823248 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:25.823222    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05c9d201-5455-4a37-8382-8188a407e044-workload-certs\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-lm5rm\" (UID: \"05c9d201-5455-4a37-8382-8188a407e044\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-lm5rm"
Nov 02 03:17:25.823248 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:25.823245    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5zjq\" (UniqueName: \"kubernetes.io/projected/05c9d201-5455-4a37-8382-8188a407e044-kube-api-access-c5zjq\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-lm5rm\" (UID: \"05c9d201-5455-4a37-8382-8188a407e044\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-lm5rm"
Nov 02 03:17:25.823404 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:25.823270    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05c9d201-5455-4a37-8382-8188a407e044-istio-token\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-lm5rm\" (UID: \"05c9d201-5455-4a37-8382-8188a407e044\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-lm5rm"
Nov 02 03:17:25.924599 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:25.924508    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05c9d201-5455-4a37-8382-8188a407e044-workload-socket\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-lm5rm\" (UID: \"05c9d201-5455-4a37-8382-8188a407e044\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-lm5rm"
Nov 02 03:17:25.924599 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:25.924543    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05c9d201-5455-4a37-8382-8188a407e044-credential-socket\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-lm5rm\" (UID: \"05c9d201-5455-4a37-8382-8188a407e044\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-lm5rm"
Nov 02 03:17:25.924599 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:25.924559    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05c9d201-5455-4a37-8382-8188a407e044-istiod-ca-cert\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-lm5rm\" (UID: \"05c9d201-5455-4a37-8382-8188a407e044\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-lm5rm"
Nov 02 03:17:25.924881 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:25.924607    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05c9d201-5455-4a37-8382-8188a407e044-istio-data\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-lm5rm\" (UID: \"05c9d201-5455-4a37-8382-8188a407e044\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-lm5rm"
Nov 02 03:17:25.924881 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:25.924635    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05c9d201-5455-4a37-8382-8188a407e044-workload-certs\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-lm5rm\" (UID: \"05c9d201-5455-4a37-8382-8188a407e044\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-lm5rm"
Nov 02 03:17:25.924881 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:25.924660    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5zjq\" (UniqueName: \"kubernetes.io/projected/05c9d201-5455-4a37-8382-8188a407e044-kube-api-access-c5zjq\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-lm5rm\" (UID: \"05c9d201-5455-4a37-8382-8188a407e044\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-lm5rm"
Nov 02 03:17:25.924881 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:25.924689    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05c9d201-5455-4a37-8382-8188a407e044-istio-token\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-lm5rm\" (UID: \"05c9d201-5455-4a37-8382-8188a407e044\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-lm5rm"
Nov 02 03:17:25.924881 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:25.924783    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05c9d201-5455-4a37-8382-8188a407e044-istio-podinfo\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-lm5rm\" (UID: \"05c9d201-5455-4a37-8382-8188a407e044\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-lm5rm"
Nov 02 03:17:25.924881 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:25.924841    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05c9d201-5455-4a37-8382-8188a407e044-istio-envoy\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-lm5rm\" (UID: \"05c9d201-5455-4a37-8382-8188a407e044\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-lm5rm"
Nov 02 03:17:25.925255 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:25.924984    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05c9d201-5455-4a37-8382-8188a407e044-credential-socket\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-lm5rm\" (UID: \"05c9d201-5455-4a37-8382-8188a407e044\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-lm5rm"
Nov 02 03:17:25.925255 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:25.925059    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05c9d201-5455-4a37-8382-8188a407e044-istio-data\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-lm5rm\" (UID: \"05c9d201-5455-4a37-8382-8188a407e044\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-lm5rm"
Nov 02 03:17:25.925255 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:25.925155    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05c9d201-5455-4a37-8382-8188a407e044-workload-certs\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-lm5rm\" (UID: \"05c9d201-5455-4a37-8382-8188a407e044\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-lm5rm"
Nov 02 03:17:25.925418 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:25.925318    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05c9d201-5455-4a37-8382-8188a407e044-workload-socket\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-lm5rm\" (UID: \"05c9d201-5455-4a37-8382-8188a407e044\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-lm5rm"
Nov 02 03:17:25.925534 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:25.925512    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05c9d201-5455-4a37-8382-8188a407e044-istiod-ca-cert\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-lm5rm\" (UID: \"05c9d201-5455-4a37-8382-8188a407e044\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-lm5rm"
Nov 02 03:17:25.927380 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:25.927356    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05c9d201-5455-4a37-8382-8188a407e044-istio-envoy\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-lm5rm\" (UID: \"05c9d201-5455-4a37-8382-8188a407e044\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-lm5rm"
Nov 02 03:17:25.927485 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:25.927443    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05c9d201-5455-4a37-8382-8188a407e044-istio-podinfo\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-lm5rm\" (UID: \"05c9d201-5455-4a37-8382-8188a407e044\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-lm5rm"
Nov 02 03:17:25.982604 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:25.982572    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05c9d201-5455-4a37-8382-8188a407e044-istio-token\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-lm5rm\" (UID: \"05c9d201-5455-4a37-8382-8188a407e044\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-lm5rm"
Nov 02 03:17:25.983226 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:25.983199    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5zjq\" (UniqueName: \"kubernetes.io/projected/05c9d201-5455-4a37-8382-8188a407e044-kube-api-access-c5zjq\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-lm5rm\" (UID: \"05c9d201-5455-4a37-8382-8188a407e044\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-lm5rm"
Nov 02 03:17:26.021438 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:26.021406    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-lm5rm"
Nov 02 03:17:26.455294 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:17:26.455262    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05c9d201_5455_4a37_8382_8188a407e044.slice/crio-769d29ff81e1b41a354008a53565d534fa59da400e7a847ead2675a7d0f701ef WatchSource:0}: Error finding container 769d29ff81e1b41a354008a53565d534fa59da400e7a847ead2675a7d0f701ef: Status 404 returned error can't find the container with id 769d29ff81e1b41a354008a53565d534fa59da400e7a847ead2675a7d0f701ef
Nov 02 03:17:26.456238 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:26.456219    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-lm5rm"]
Nov 02 03:17:26.458173 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:26.458136    4496 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 03:17:26.458315 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:26.458217    4496 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 03:17:26.458315 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:26.458269    4496 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 03:17:27.200693 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:27.200642    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-lm5rm" event={"ID":"05c9d201-5455-4a37-8382-8188a407e044","Type":"ContainerStarted","Data":"598a2e262bba3e5d2d25036e801009386eb99ca593fe30e905e2959a58ba48d1"}
Nov 02 03:17:27.201348 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:27.201300    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-lm5rm" event={"ID":"05c9d201-5455-4a37-8382-8188a407e044","Type":"ContainerStarted","Data":"769d29ff81e1b41a354008a53565d534fa59da400e7a847ead2675a7d0f701ef"}
Nov 02 03:17:27.474482 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:27.465381    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-lm5rm" podStartSLOduration=2.46536115 podStartE2EDuration="2.46536115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:17:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:17:27.382525881 +0000 UTC m=+1787.961133263" watchObservedRunningTime="2026-11-02 03:17:27.46536115 +0000 UTC m=+1788.043968532"
Nov 02 03:17:28.026413 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:28.025073    4496 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-lm5rm"
Nov 02 03:17:28.026727 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:28.026700    4496 patch_prober.go:29] interesting pod/e2e-worker-w4-398828-openshift-default-5647988965-lm5rm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body=
Nov 02 03:17:28.026919 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:28.026892    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-lm5rm" podUID="05c9d201-5455-4a37-8382-8188a407e044" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused"
Nov 02 03:17:28.172039 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:28.171248    4496 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-398828/simulator-w4-398828-kserve-6d5bd49c6f-9tw2c" podUID="5b6aecfa-3e3b-4fb3-a80c-35b77de60ce2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 03:17:28.577037 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:28.576291    4496 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xlbmw" podUID="2baeda96-f36e-49cb-bb8a-3389c9e8d61d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 03:17:28.777164 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:28.777116    4496 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-ca6546/simulator-w2-ca6546-kserve-5959bb649b-5x6tf" podUID="4fd6a8f7-0b10-4009-a6db-de0a01aca2b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/health\": context deadline exceeded"
Nov 02 03:17:28.886210 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:28.886101    4496 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-398828/distinct-2-w4-398828-kserve-545449d49b-6mg4b" podUID="36c1912c-2ab1-40c8-a09d-2425ab08ec8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.125:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 03:17:28.886788 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:28.886758    4496 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5ac93f/distinct-w1-5ac93f-kserve-64697cb8c9-vj9sk" podUID="75a1c221-38e8-47cf-a8b5-305396cee703" containerName="main" probeResult="failure" output="Get \"https://10.134.0.102:8000/ready\": context deadline exceeded"
Nov 02 03:17:28.987209 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:28.987108    4496 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-858236/distinct-w3-858236-kserve-df6dbd64b-qbk9g" podUID="719bb9e5-ea72-4782-bb17-034b909f874d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.126:8000/ready\": context deadline exceeded"
Nov 02 03:17:29.025407 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:29.025367    4496 patch_prober.go:29] interesting pod/e2e-worker-w4-398828-openshift-default-5647988965-lm5rm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body=
Nov 02 03:17:29.025705 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:29.025675    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-lm5rm" podUID="05c9d201-5455-4a37-8382-8188a407e044" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused"
Nov 02 03:17:29.052138 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:29.052100    4496 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-s9dfr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 03:17:29.052415 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:29.052382    4496 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-s9dfr" podUID="9e667364-fba5-4584-9251-773ae8e1f2e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 03:17:29.172434 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:29.172306    4496 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-398828/simulator-w4-398828-kserve-6d5bd49c6f-9tw2c" podUID="5b6aecfa-3e3b-4fb3-a80c-35b77de60ce2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded"
Nov 02 03:17:29.173233 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:29.173207    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-398828/simulator-w4-398828-kserve-6d5bd49c6f-9tw2c"
Nov 02 03:17:29.583623 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:29.583574    4496 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xlbmw" podUID="2baeda96-f36e-49cb-bb8a-3389c9e8d61d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 03:17:29.906207 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:29.906110    4496 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5ac93f/distinct-w1-5ac93f-kserve-64697cb8c9-vj9sk" podUID="75a1c221-38e8-47cf-a8b5-305396cee703" containerName="main" probeResult="failure" output="Get \"https://10.134.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 03:17:29.906552 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:29.906472    4496 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-398828/distinct-2-w4-398828-kserve-545449d49b-6mg4b" podUID="36c1912c-2ab1-40c8-a09d-2425ab08ec8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.125:8000/ready\": context deadline exceeded"
Nov 02 03:17:29.908293 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:29.906874    4496 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ca6546/simulator-w2-ca6546-kserve-5959bb649b-5x6tf" podUID="4fd6a8f7-0b10-4009-a6db-de0a01aca2b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 03:17:29.946663 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:29.912774    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xlbmw"
Nov 02 03:17:29.946663 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:29.912848    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5ac93f/distinct-w1-5ac93f-kserve-64697cb8c9-vj9sk"
Nov 02 03:17:29.946663 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:29.912881    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-398828/distinct-2-w4-398828-kserve-545449d49b-6mg4b"
Nov 02 03:17:29.946663 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:29.912907    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xlbmw"
Nov 02 03:17:29.946663 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:29.914590    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-398828/simulator-w4-398828-kserve-6d5bd49c6f-9tw2c"
Nov 02 03:17:30.004138 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:30.004106    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5ac93f/distinct-w1-5ac93f-kserve-64697cb8c9-vj9sk"
Nov 02 03:17:30.023153 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:30.022437    4496 patch_prober.go:29] interesting pod/e2e-worker-w4-398828-openshift-default-5647988965-lm5rm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body=
Nov 02 03:17:30.023153 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:30.022525    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-lm5rm" podUID="05c9d201-5455-4a37-8382-8188a407e044" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused"
Nov 02 03:17:30.023153 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:30.023086    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-398828/distinct-2-w4-398828-kserve-545449d49b-6mg4b"
Nov 02 03:17:30.680274 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:30.680242    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce9e7480-qhxqv"]
Nov 02 03:17:30.693037 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:30.687547    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce9e7480-qhxqv"
Nov 02 03:17:30.781668 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:30.781604    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce9e7480-qhxqv"]
Nov 02 03:17:30.802275 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:30.802242    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f61f56ac-c17c-4227-8d05-fadb7732cf42-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce9e7480-qhxqv\" (UID: \"f61f56ac-c17c-4227-8d05-fadb7732cf42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce9e7480-qhxqv"
Nov 02 03:17:30.802540 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:30.802518    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9fn7\" (UniqueName: \"kubernetes.io/projected/f61f56ac-c17c-4227-8d05-fadb7732cf42-kube-api-access-n9fn7\") pod \"maas-api-revoke-sub-ce9e7480-qhxqv\" (UID: \"f61f56ac-c17c-4227-8d05-fadb7732cf42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce9e7480-qhxqv"
Nov 02 03:17:30.802675 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:30.802660    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f61f56ac-c17c-4227-8d05-fadb7732cf42-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce9e7480-qhxqv\" (UID: \"f61f56ac-c17c-4227-8d05-fadb7732cf42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce9e7480-qhxqv"
Nov 02 03:17:30.813479 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:30.813442    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9z5bk\""
Nov 02 03:17:30.818898 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:30.818861    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 03:17:30.824503 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:30.824464    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 03:17:30.825140 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:30.824799    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 03:17:30.923519 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:30.923477    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f61f56ac-c17c-4227-8d05-fadb7732cf42-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce9e7480-qhxqv\" (UID: \"f61f56ac-c17c-4227-8d05-fadb7732cf42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce9e7480-qhxqv"
Nov 02 03:17:30.923886 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:30.923827    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9fn7\" (UniqueName: \"kubernetes.io/projected/f61f56ac-c17c-4227-8d05-fadb7732cf42-kube-api-access-n9fn7\") pod \"maas-api-revoke-sub-ce9e7480-qhxqv\" (UID: \"f61f56ac-c17c-4227-8d05-fadb7732cf42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce9e7480-qhxqv"
Nov 02 03:17:30.924107 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:30.924088    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f61f56ac-c17c-4227-8d05-fadb7732cf42-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce9e7480-qhxqv\" (UID: \"f61f56ac-c17c-4227-8d05-fadb7732cf42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce9e7480-qhxqv"
Nov 02 03:17:30.951476 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:30.951396    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f61f56ac-c17c-4227-8d05-fadb7732cf42-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce9e7480-qhxqv\" (UID: \"f61f56ac-c17c-4227-8d05-fadb7732cf42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce9e7480-qhxqv"
Nov 02 03:17:30.995088 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:30.994567    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f61f56ac-c17c-4227-8d05-fadb7732cf42-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce9e7480-qhxqv\" (UID: \"f61f56ac-c17c-4227-8d05-fadb7732cf42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce9e7480-qhxqv"
Nov 02 03:17:31.031456 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:31.028572    4496 patch_prober.go:29] interesting pod/e2e-worker-w4-398828-openshift-default-5647988965-lm5rm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body=
Nov 02 03:17:31.056994 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:31.056954    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-lm5rm" podUID="05c9d201-5455-4a37-8382-8188a407e044" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused"
Nov 02 03:17:31.089414 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:31.087697    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9fn7\" (UniqueName: \"kubernetes.io/projected/f61f56ac-c17c-4227-8d05-fadb7732cf42-kube-api-access-n9fn7\") pod \"maas-api-revoke-sub-ce9e7480-qhxqv\" (UID: \"f61f56ac-c17c-4227-8d05-fadb7732cf42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce9e7480-qhxqv"
Nov 02 03:17:31.124035 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:31.117185    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce9e7480-qhxqv"
Nov 02 03:17:31.612936 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:17:31.612893    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf61f56ac_c17c_4227_8d05_fadb7732cf42.slice/crio-36aa7e3d0455cb5075b8bfb9c6fb98da8245450db6ae0ad33b171b275580dc0c WatchSource:0}: Error finding container 36aa7e3d0455cb5075b8bfb9c6fb98da8245450db6ae0ad33b171b275580dc0c: Status 404 returned error can't find the container with id 36aa7e3d0455cb5075b8bfb9c6fb98da8245450db6ae0ad33b171b275580dc0c
Nov 02 03:17:31.613203 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:31.613180    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce9e7480-qhxqv"]
Nov 02 03:17:32.022199 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:32.022165    4496 patch_prober.go:29] interesting pod/e2e-worker-w4-398828-openshift-default-5647988965-lm5rm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body=
Nov 02 03:17:32.022674 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:32.022217    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-lm5rm" podUID="05c9d201-5455-4a37-8382-8188a407e044" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused"
Nov 02 03:17:32.407060 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:32.407010    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce9e7480-qhxqv" event={"ID":"f61f56ac-c17c-4227-8d05-fadb7732cf42","Type":"ContainerStarted","Data":"096d2cff54f511a2a2a8be39b0eadbedb16ff4024ff6b00e54cde9bc3c1d2ec2"}
Nov 02 03:17:32.407245 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:32.407108    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce9e7480-qhxqv" event={"ID":"f61f56ac-c17c-4227-8d05-fadb7732cf42","Type":"ContainerStarted","Data":"36aa7e3d0455cb5075b8bfb9c6fb98da8245450db6ae0ad33b171b275580dc0c"}
Nov 02 03:17:32.525707 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:32.525643    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce9e7480-qhxqv" podStartSLOduration=2.525626853 podStartE2EDuration="2.525626853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:17:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:17:32.522772311 +0000 UTC m=+1793.101379690" watchObservedRunningTime="2026-11-02 03:17:32.525626853 +0000 UTC m=+1793.104234234"
Nov 02 03:17:32.810855 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:32.810821    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a94dd74-6lwkz"]
Nov 02 03:17:32.815100 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:32.815082    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a94dd74-6lwkz"
Nov 02 03:17:32.872771 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:32.872735    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a94dd74-6lwkz"]
Nov 02 03:17:32.984329 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:32.984295    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dda7659a-4dd9-42bd-b072-902b9077062a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1a94dd74-6lwkz\" (UID: \"dda7659a-4dd9-42bd-b072-902b9077062a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a94dd74-6lwkz"
Nov 02 03:17:32.984570 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:32.984544    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dda7659a-4dd9-42bd-b072-902b9077062a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1a94dd74-6lwkz\" (UID: \"dda7659a-4dd9-42bd-b072-902b9077062a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a94dd74-6lwkz"
Nov 02 03:17:32.984621 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:32.984597    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdvq4\" (UniqueName: \"kubernetes.io/projected/dda7659a-4dd9-42bd-b072-902b9077062a-kube-api-access-tdvq4\") pod \"maas-api-revoke-sub-1a94dd74-6lwkz\" (UID: \"dda7659a-4dd9-42bd-b072-902b9077062a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a94dd74-6lwkz"
Nov 02 03:17:33.021691 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:33.021653    4496 patch_prober.go:29] interesting pod/e2e-worker-w4-398828-openshift-default-5647988965-lm5rm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body=
Nov 02 03:17:33.021881 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:33.021720    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-lm5rm" podUID="05c9d201-5455-4a37-8382-8188a407e044" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused"
Nov 02 03:17:33.085745 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:33.085648    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dda7659a-4dd9-42bd-b072-902b9077062a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1a94dd74-6lwkz\" (UID: \"dda7659a-4dd9-42bd-b072-902b9077062a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a94dd74-6lwkz"
Nov 02 03:17:33.085745 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:33.085709    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdvq4\" (UniqueName: \"kubernetes.io/projected/dda7659a-4dd9-42bd-b072-902b9077062a-kube-api-access-tdvq4\") pod \"maas-api-revoke-sub-1a94dd74-6lwkz\" (UID: \"dda7659a-4dd9-42bd-b072-902b9077062a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a94dd74-6lwkz"
Nov 02 03:17:33.086231 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:33.085760    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dda7659a-4dd9-42bd-b072-902b9077062a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1a94dd74-6lwkz\" (UID: \"dda7659a-4dd9-42bd-b072-902b9077062a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a94dd74-6lwkz"
Nov 02 03:17:33.086503 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:33.086476    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dda7659a-4dd9-42bd-b072-902b9077062a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1a94dd74-6lwkz\" (UID: \"dda7659a-4dd9-42bd-b072-902b9077062a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a94dd74-6lwkz"
Nov 02 03:17:33.087968 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:33.087944    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dda7659a-4dd9-42bd-b072-902b9077062a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1a94dd74-6lwkz\" (UID: \"dda7659a-4dd9-42bd-b072-902b9077062a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a94dd74-6lwkz"
Nov 02 03:17:33.158705 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:33.158670    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdvq4\" (UniqueName: \"kubernetes.io/projected/dda7659a-4dd9-42bd-b072-902b9077062a-kube-api-access-tdvq4\") pod \"maas-api-revoke-sub-1a94dd74-6lwkz\" (UID: \"dda7659a-4dd9-42bd-b072-902b9077062a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a94dd74-6lwkz"
Nov 02 03:17:33.416895 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:33.416811    4496 generic.go:356] "Generic (PLEG): container finished" podID="f61f56ac-c17c-4227-8d05-fadb7732cf42" containerID="096d2cff54f511a2a2a8be39b0eadbedb16ff4024ff6b00e54cde9bc3c1d2ec2" exitCode=0
Nov 02 03:17:33.417074 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:33.416899    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce9e7480-qhxqv" event={"ID":"f61f56ac-c17c-4227-8d05-fadb7732cf42","Type":"ContainerDied","Data":"096d2cff54f511a2a2a8be39b0eadbedb16ff4024ff6b00e54cde9bc3c1d2ec2"}
Nov 02 03:17:33.428948 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:33.428919    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a94dd74-6lwkz"
Nov 02 03:17:33.588463 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:17:33.588430    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 03:17:33.850038 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:33.846327    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a94dd74-6lwkz"]
Nov 02 03:17:33.866289 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:17:33.866246    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddda7659a_4dd9_42bd_b072_902b9077062a.slice/crio-ed29ad19bf5794da4e1ccecd9be99b3765f5a027fc4562d47dc7861051dd8ecb WatchSource:0}: Error finding container ed29ad19bf5794da4e1ccecd9be99b3765f5a027fc4562d47dc7861051dd8ecb: Status 404 returned error can't find the container with id ed29ad19bf5794da4e1ccecd9be99b3765f5a027fc4562d47dc7861051dd8ecb
Nov 02 03:17:34.022152 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:34.022110    4496 patch_prober.go:29] interesting pod/e2e-worker-w4-398828-openshift-default-5647988965-lm5rm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body=
Nov 02 03:17:34.022353 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:34.022168    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-lm5rm" podUID="05c9d201-5455-4a37-8382-8188a407e044" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused"
Nov 02 03:17:34.425908 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:34.425870    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a94dd74-6lwkz" event={"ID":"dda7659a-4dd9-42bd-b072-902b9077062a","Type":"ContainerStarted","Data":"e9e99f3ff4c8dc35a537cbb0487547198d8e7a55401dd90f435807b6acb76903"}
Nov 02 03:17:34.426332 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:34.425929    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a94dd74-6lwkz" event={"ID":"dda7659a-4dd9-42bd-b072-902b9077062a","Type":"ContainerStarted","Data":"ed29ad19bf5794da4e1ccecd9be99b3765f5a027fc4562d47dc7861051dd8ecb"}
Nov 02 03:17:34.559144 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:34.559095    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a94dd74-6lwkz" podStartSLOduration=2.559077159 podStartE2EDuration="2.559077159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:17:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:17:34.556448818 +0000 UTC m=+1795.135056200" watchObservedRunningTime="2026-11-02 03:17:34.559077159 +0000 UTC m=+1795.137684540"
Nov 02 03:17:34.587337 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:34.587312    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce9e7480-qhxqv"
Nov 02 03:17:34.709382 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:34.709333    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f61f56ac-c17c-4227-8d05-fadb7732cf42-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f61f56ac-c17c-4227-8d05-fadb7732cf42-maas-api-xks-ca\") pod \"f61f56ac-c17c-4227-8d05-fadb7732cf42\" (UID: \"f61f56ac-c17c-4227-8d05-fadb7732cf42\") "
Nov 02 03:17:34.709574 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:34.709393    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f61f56ac-c17c-4227-8d05-fadb7732cf42-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f61f56ac-c17c-4227-8d05-fadb7732cf42-maas-api-openshift-ca\") pod \"f61f56ac-c17c-4227-8d05-fadb7732cf42\" (UID: \"f61f56ac-c17c-4227-8d05-fadb7732cf42\") "
Nov 02 03:17:34.709658 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:34.709572    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f61f56ac-c17c-4227-8d05-fadb7732cf42-kube-api-access-n9fn7\" (UniqueName: \"kubernetes.io/projected/f61f56ac-c17c-4227-8d05-fadb7732cf42-kube-api-access-n9fn7\") pod \"f61f56ac-c17c-4227-8d05-fadb7732cf42\" (UID: \"f61f56ac-c17c-4227-8d05-fadb7732cf42\") "
Nov 02 03:17:34.711155 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:34.711126    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f61f56ac-c17c-4227-8d05-fadb7732cf42-maas-api-openshift-ca" pod "f61f56ac-c17c-4227-8d05-fadb7732cf42" (UID: "f61f56ac-c17c-4227-8d05-fadb7732cf42"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:17:34.713701 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:34.713671    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f61f56ac-c17c-4227-8d05-fadb7732cf42-maas-api-xks-ca" pod "f61f56ac-c17c-4227-8d05-fadb7732cf42" (UID: "f61f56ac-c17c-4227-8d05-fadb7732cf42"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 03:17:34.714171 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:34.714142    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f61f56ac-c17c-4227-8d05-fadb7732cf42-kube-api-access-n9fn7" pod "f61f56ac-c17c-4227-8d05-fadb7732cf42" (UID: "f61f56ac-c17c-4227-8d05-fadb7732cf42"). InnerVolumeSpecName "kube-api-access-n9fn7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:17:34.811719 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:34.811660    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9fn7\" (UniqueName: \"kubernetes.io/projected/f61f56ac-c17c-4227-8d05-fadb7732cf42-kube-api-access-n9fn7\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:17:34.811719 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:34.811687    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f61f56ac-c17c-4227-8d05-fadb7732cf42-maas-api-xks-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:17:34.811719 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:34.811698    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f61f56ac-c17c-4227-8d05-fadb7732cf42-maas-api-openshift-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:17:35.279575 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:35.279534    4496 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-lm5rm"
Nov 02 03:17:35.280159 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:35.280081    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-lm5rm"
Nov 02 03:17:35.281648 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:35.281622    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-lm5rm"
Nov 02 03:17:35.435552 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:35.435466    4496 generic.go:356] "Generic (PLEG): container finished" podID="dda7659a-4dd9-42bd-b072-902b9077062a" containerID="e9e99f3ff4c8dc35a537cbb0487547198d8e7a55401dd90f435807b6acb76903" exitCode=0
Nov 02 03:17:35.435939 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:35.435558    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a94dd74-6lwkz" event={"ID":"dda7659a-4dd9-42bd-b072-902b9077062a","Type":"ContainerDied","Data":"e9e99f3ff4c8dc35a537cbb0487547198d8e7a55401dd90f435807b6acb76903"}
Nov 02 03:17:35.437135 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:35.437117    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce9e7480-qhxqv"
Nov 02 03:17:35.437241 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:35.437162    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce9e7480-qhxqv" event={"ID":"f61f56ac-c17c-4227-8d05-fadb7732cf42","Type":"ContainerDied","Data":"36aa7e3d0455cb5075b8bfb9c6fb98da8245450db6ae0ad33b171b275580dc0c"}
Nov 02 03:17:35.437241 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:35.437210    4496 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="36aa7e3d0455cb5075b8bfb9c6fb98da8245450db6ae0ad33b171b275580dc0c"
Nov 02 03:17:35.828042 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:35.827991    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-x5gwb"]
Nov 02 03:17:35.828930 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:35.828901    4496 cpu_manager.go:414] "containerMap: removing container" podUID="f61f56ac-c17c-4227-8d05-fadb7732cf42" containerName="revoke-keys"
Nov 02 03:17:35.828930 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:35.828926    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f61f56ac-c17c-4227-8d05-fadb7732cf42" containerName="revoke-keys"
Nov 02 03:17:35.829163 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:35.829093    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="f61f56ac-c17c-4227-8d05-fadb7732cf42" containerName="revoke-keys"
Nov 02 03:17:35.832875 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:35.832855    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-x5gwb"
Nov 02 03:17:35.890868 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:35.890772    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-x5gwb"]
Nov 02 03:17:35.927517 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:35.927479    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/938e8e17-80ba-4b93-9f96-29e68f73f927-workload-socket\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-x5gwb\" (UID: \"938e8e17-80ba-4b93-9f96-29e68f73f927\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-x5gwb"
Nov 02 03:17:35.927668 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:35.927593    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/938e8e17-80ba-4b93-9f96-29e68f73f927-istio-envoy\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-x5gwb\" (UID: \"938e8e17-80ba-4b93-9f96-29e68f73f927\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-x5gwb"
Nov 02 03:17:35.927668 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:35.927637    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/938e8e17-80ba-4b93-9f96-29e68f73f927-credential-socket\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-x5gwb\" (UID: \"938e8e17-80ba-4b93-9f96-29e68f73f927\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-x5gwb"
Nov 02 03:17:35.927743 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:35.927675    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/938e8e17-80ba-4b93-9f96-29e68f73f927-istio-token\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-x5gwb\" (UID: \"938e8e17-80ba-4b93-9f96-29e68f73f927\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-x5gwb"
Nov 02 03:17:35.927743 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:35.927703    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/938e8e17-80ba-4b93-9f96-29e68f73f927-workload-certs\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-x5gwb\" (UID: \"938e8e17-80ba-4b93-9f96-29e68f73f927\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-x5gwb"
Nov 02 03:17:35.927825 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:35.927808    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/938e8e17-80ba-4b93-9f96-29e68f73f927-istio-data\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-x5gwb\" (UID: \"938e8e17-80ba-4b93-9f96-29e68f73f927\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-x5gwb"
Nov 02 03:17:35.927860 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:35.927838    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/938e8e17-80ba-4b93-9f96-29e68f73f927-istiod-ca-cert\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-x5gwb\" (UID: \"938e8e17-80ba-4b93-9f96-29e68f73f927\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-x5gwb"
Nov 02 03:17:35.927997 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:35.927976    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68b6p\" (UniqueName: \"kubernetes.io/projected/938e8e17-80ba-4b93-9f96-29e68f73f927-kube-api-access-68b6p\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-x5gwb\" (UID: \"938e8e17-80ba-4b93-9f96-29e68f73f927\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-x5gwb"
Nov 02 03:17:35.928079 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:35.928066    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/938e8e17-80ba-4b93-9f96-29e68f73f927-istio-podinfo\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-x5gwb\" (UID: \"938e8e17-80ba-4b93-9f96-29e68f73f927\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-x5gwb"
Nov 02 03:17:36.028938 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:36.028906    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/938e8e17-80ba-4b93-9f96-29e68f73f927-credential-socket\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-x5gwb\" (UID: \"938e8e17-80ba-4b93-9f96-29e68f73f927\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-x5gwb"
Nov 02 03:17:36.028938 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:36.028953    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/938e8e17-80ba-4b93-9f96-29e68f73f927-istio-token\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-x5gwb\" (UID: \"938e8e17-80ba-4b93-9f96-29e68f73f927\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-x5gwb"
Nov 02 03:17:36.029230 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:36.028978    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/938e8e17-80ba-4b93-9f96-29e68f73f927-workload-certs\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-x5gwb\" (UID: \"938e8e17-80ba-4b93-9f96-29e68f73f927\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-x5gwb"
Nov 02 03:17:36.029230 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:36.029054    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/938e8e17-80ba-4b93-9f96-29e68f73f927-istio-data\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-x5gwb\" (UID: \"938e8e17-80ba-4b93-9f96-29e68f73f927\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-x5gwb"
Nov 02 03:17:36.029230 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:36.029086    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/938e8e17-80ba-4b93-9f96-29e68f73f927-istiod-ca-cert\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-x5gwb\" (UID: \"938e8e17-80ba-4b93-9f96-29e68f73f927\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-x5gwb"
Nov 02 03:17:36.029230 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:36.029172    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68b6p\" (UniqueName: \"kubernetes.io/projected/938e8e17-80ba-4b93-9f96-29e68f73f927-kube-api-access-68b6p\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-x5gwb\" (UID: \"938e8e17-80ba-4b93-9f96-29e68f73f927\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-x5gwb"
Nov 02 03:17:36.029230 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:36.029219    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/938e8e17-80ba-4b93-9f96-29e68f73f927-istio-podinfo\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-x5gwb\" (UID: \"938e8e17-80ba-4b93-9f96-29e68f73f927\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-x5gwb"
Nov 02 03:17:36.029505 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:36.029324    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/938e8e17-80ba-4b93-9f96-29e68f73f927-workload-socket\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-x5gwb\" (UID: \"938e8e17-80ba-4b93-9f96-29e68f73f927\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-x5gwb"
Nov 02 03:17:36.029505 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:36.029370    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/938e8e17-80ba-4b93-9f96-29e68f73f927-istio-envoy\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-x5gwb\" (UID: \"938e8e17-80ba-4b93-9f96-29e68f73f927\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-x5gwb"
Nov 02 03:17:36.029505 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:36.029402    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/938e8e17-80ba-4b93-9f96-29e68f73f927-credential-socket\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-x5gwb\" (UID: \"938e8e17-80ba-4b93-9f96-29e68f73f927\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-x5gwb"
Nov 02 03:17:36.029505 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:36.029460    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/938e8e17-80ba-4b93-9f96-29e68f73f927-istio-data\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-x5gwb\" (UID: \"938e8e17-80ba-4b93-9f96-29e68f73f927\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-x5gwb"
Nov 02 03:17:36.030167 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:36.029681    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/938e8e17-80ba-4b93-9f96-29e68f73f927-workload-certs\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-x5gwb\" (UID: \"938e8e17-80ba-4b93-9f96-29e68f73f927\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-x5gwb"
Nov 02 03:17:36.030167 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:36.029729    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/938e8e17-80ba-4b93-9f96-29e68f73f927-workload-socket\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-x5gwb\" (UID: \"938e8e17-80ba-4b93-9f96-29e68f73f927\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-x5gwb"
Nov 02 03:17:36.030283 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:36.030259    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/938e8e17-80ba-4b93-9f96-29e68f73f927-istiod-ca-cert\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-x5gwb\" (UID: \"938e8e17-80ba-4b93-9f96-29e68f73f927\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-x5gwb"
Nov 02 03:17:36.031676 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:36.031649    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/938e8e17-80ba-4b93-9f96-29e68f73f927-istio-envoy\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-x5gwb\" (UID: \"938e8e17-80ba-4b93-9f96-29e68f73f927\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-x5gwb"
Nov 02 03:17:36.031785 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:36.031713    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/938e8e17-80ba-4b93-9f96-29e68f73f927-istio-podinfo\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-x5gwb\" (UID: \"938e8e17-80ba-4b93-9f96-29e68f73f927\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-x5gwb"
Nov 02 03:17:36.086660 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:36.086583    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/938e8e17-80ba-4b93-9f96-29e68f73f927-istio-token\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-x5gwb\" (UID: \"938e8e17-80ba-4b93-9f96-29e68f73f927\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-x5gwb"
Nov 02 03:17:36.086893 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:36.086867    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68b6p\" (UniqueName: \"kubernetes.io/projected/938e8e17-80ba-4b93-9f96-29e68f73f927-kube-api-access-68b6p\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-x5gwb\" (UID: \"938e8e17-80ba-4b93-9f96-29e68f73f927\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-x5gwb"
Nov 02 03:17:36.148199 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:36.148165    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-x5gwb"
Nov 02 03:17:36.586545 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:36.586512    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-x5gwb"]
Nov 02 03:17:36.588310 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:17:36.588280    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod938e8e17_80ba_4b93_9f96_29e68f73f927.slice/crio-4e6387c2c10ac5e347b2768084216de3d372c8c95bf3930ab5e250d7c4ed9ca5 WatchSource:0}: Error finding container 4e6387c2c10ac5e347b2768084216de3d372c8c95bf3930ab5e250d7c4ed9ca5: Status 404 returned error can't find the container with id 4e6387c2c10ac5e347b2768084216de3d372c8c95bf3930ab5e250d7c4ed9ca5
Nov 02 03:17:36.591214 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:36.591172    4496 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 03:17:36.591324 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:36.591252    4496 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 03:17:36.591324 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:36.591295    4496 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 03:17:36.694294 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:36.694269    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a94dd74-6lwkz"
Nov 02 03:17:36.838887 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:36.838847    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dda7659a-4dd9-42bd-b072-902b9077062a-kube-api-access-tdvq4\" (UniqueName: \"kubernetes.io/projected/dda7659a-4dd9-42bd-b072-902b9077062a-kube-api-access-tdvq4\") pod \"dda7659a-4dd9-42bd-b072-902b9077062a\" (UID: \"dda7659a-4dd9-42bd-b072-902b9077062a\") "
Nov 02 03:17:36.839114 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:36.839028    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dda7659a-4dd9-42bd-b072-902b9077062a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dda7659a-4dd9-42bd-b072-902b9077062a-maas-api-xks-ca\") pod \"dda7659a-4dd9-42bd-b072-902b9077062a\" (UID: \"dda7659a-4dd9-42bd-b072-902b9077062a\") "
Nov 02 03:17:36.839267 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:36.839246    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dda7659a-4dd9-42bd-b072-902b9077062a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dda7659a-4dd9-42bd-b072-902b9077062a-maas-api-openshift-ca\") pod \"dda7659a-4dd9-42bd-b072-902b9077062a\" (UID: \"dda7659a-4dd9-42bd-b072-902b9077062a\") "
Nov 02 03:17:36.840157 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:36.840132    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dda7659a-4dd9-42bd-b072-902b9077062a-maas-api-openshift-ca" pod "dda7659a-4dd9-42bd-b072-902b9077062a" (UID: "dda7659a-4dd9-42bd-b072-902b9077062a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:17:36.842616 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:36.842590    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dda7659a-4dd9-42bd-b072-902b9077062a-maas-api-xks-ca" pod "dda7659a-4dd9-42bd-b072-902b9077062a" (UID: "dda7659a-4dd9-42bd-b072-902b9077062a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 03:17:36.843390 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:36.843344    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dda7659a-4dd9-42bd-b072-902b9077062a-kube-api-access-tdvq4" pod "dda7659a-4dd9-42bd-b072-902b9077062a" (UID: "dda7659a-4dd9-42bd-b072-902b9077062a"). InnerVolumeSpecName "kube-api-access-tdvq4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:17:36.940798 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:36.940763    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dda7659a-4dd9-42bd-b072-902b9077062a-maas-api-openshift-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:17:36.940798 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:36.940793    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdvq4\" (UniqueName: \"kubernetes.io/projected/dda7659a-4dd9-42bd-b072-902b9077062a-kube-api-access-tdvq4\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:17:36.940798 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:36.940805    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dda7659a-4dd9-42bd-b072-902b9077062a-maas-api-xks-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:17:37.466967 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:37.466873    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-x5gwb" event={"ID":"938e8e17-80ba-4b93-9f96-29e68f73f927","Type":"ContainerStarted","Data":"843543e593513d43dd923e6a55ddc2f2bb1a95c51f7b86f5ab4ff1a6cae60e03"}
Nov 02 03:17:37.466967 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:37.466945    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-x5gwb" event={"ID":"938e8e17-80ba-4b93-9f96-29e68f73f927","Type":"ContainerStarted","Data":"4e6387c2c10ac5e347b2768084216de3d372c8c95bf3930ab5e250d7c4ed9ca5"}
Nov 02 03:17:37.469072 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:37.468954    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a94dd74-6lwkz"
Nov 02 03:17:37.469072 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:37.468952    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a94dd74-6lwkz" event={"ID":"dda7659a-4dd9-42bd-b072-902b9077062a","Type":"ContainerDied","Data":"ed29ad19bf5794da4e1ccecd9be99b3765f5a027fc4562d47dc7861051dd8ecb"}
Nov 02 03:17:37.469271 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:37.469108    4496 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed29ad19bf5794da4e1ccecd9be99b3765f5a027fc4562d47dc7861051dd8ecb"
Nov 02 03:17:37.640039 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:37.638874    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-x5gwb" podStartSLOduration=2.63885426 podStartE2EDuration="2.63885426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:17:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:17:37.637482606 +0000 UTC m=+1798.216089979" watchObservedRunningTime="2026-11-02 03:17:37.63885426 +0000 UTC m=+1798.217461640"
Nov 02 03:17:38.154524 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:38.154489    4496 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-x5gwb"
Nov 02 03:17:38.165888 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:38.165852    4496 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-x5gwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body=
Nov 02 03:17:38.166192 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:38.166157    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-x5gwb" podUID="938e8e17-80ba-4b93-9f96-29e68f73f927" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused"
Nov 02 03:17:39.151499 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:39.151454    4496 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-x5gwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body=
Nov 02 03:17:39.152140 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:39.152099    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-x5gwb" podUID="938e8e17-80ba-4b93-9f96-29e68f73f927" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused"
Nov 02 03:17:39.825785 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:39.825743    4496 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hww8q_5c797b08-a314-44ed-8026-8a0ce1f6c36f/ovn-acl-logging/0.log"
Nov 02 03:17:39.832854 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:39.832829    4496 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hww8q_5c797b08-a314-44ed-8026-8a0ce1f6c36f/ovn-acl-logging/0.log"
Nov 02 03:17:40.155720 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:40.155665    4496 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-x5gwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body=
Nov 02 03:17:40.170708 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:40.169096    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-x5gwb" podUID="938e8e17-80ba-4b93-9f96-29e68f73f927" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused"
Nov 02 03:17:41.150467 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:41.150428    4496 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-x5gwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body=
Nov 02 03:17:41.150773 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:41.150740    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-x5gwb" podUID="938e8e17-80ba-4b93-9f96-29e68f73f927" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused"
Nov 02 03:17:42.154039 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:42.153420    4496 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-x5gwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body=
Nov 02 03:17:42.154039 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:42.153495    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-x5gwb" podUID="938e8e17-80ba-4b93-9f96-29e68f73f927" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused"
Nov 02 03:17:42.988203 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:42.988152    4496 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-858236/distinct-w3-858236-kserve-df6dbd64b-qbk9g" podUID="719bb9e5-ea72-4782-bb17-034b909f874d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.126:8000/ready\": context deadline exceeded"
Nov 02 03:17:43.015891 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:43.015097    4496 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5ac93f/distinct-w1-5ac93f-kserve-64697cb8c9-vj9sk" podUID="75a1c221-38e8-47cf-a8b5-305396cee703" containerName="main" probeResult="failure" output="Get \"https://10.134.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 03:17:43.156042 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:43.150530    4496 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-x5gwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body=
Nov 02 03:17:43.156042 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:43.150593    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-x5gwb" podUID="938e8e17-80ba-4b93-9f96-29e68f73f927" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused"
Nov 02 03:17:43.705037 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:43.704983    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-813da41b-mx857"]
Nov 02 03:17:43.706247 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:43.706218    4496 cpu_manager.go:414] "containerMap: removing container" podUID="dda7659a-4dd9-42bd-b072-902b9077062a" containerName="revoke-keys"
Nov 02 03:17:43.706408 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:43.706394    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dda7659a-4dd9-42bd-b072-902b9077062a" containerName="revoke-keys"
Nov 02 03:17:43.706703 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:43.706690    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="dda7659a-4dd9-42bd-b072-902b9077062a" containerName="revoke-keys"
Nov 02 03:17:43.728038 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:43.726178    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-813da41b-mx857"
Nov 02 03:17:43.732038 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:43.728706    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-813da41b-mx857"]
Nov 02 03:17:43.838510 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:43.838477    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 03:17:43.838727 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:43.838539    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 03:17:43.838788 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:43.838536    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9z5bk\""
Nov 02 03:17:43.838788 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:43.838606    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 03:17:43.890167 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:43.890136    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f095bbfb-6214-411c-a1e1-6f59ddf05837-maas-api-xks-ca\") pod \"maas-api-revoke-sub-813da41b-mx857\" (UID: \"f095bbfb-6214-411c-a1e1-6f59ddf05837\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-813da41b-mx857"
Nov 02 03:17:43.890331 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:43.890258    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f095bbfb-6214-411c-a1e1-6f59ddf05837-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-813da41b-mx857\" (UID: \"f095bbfb-6214-411c-a1e1-6f59ddf05837\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-813da41b-mx857"
Nov 02 03:17:43.890416 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:43.890355    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fj54p\" (UniqueName: \"kubernetes.io/projected/f095bbfb-6214-411c-a1e1-6f59ddf05837-kube-api-access-fj54p\") pod \"maas-api-revoke-sub-813da41b-mx857\" (UID: \"f095bbfb-6214-411c-a1e1-6f59ddf05837\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-813da41b-mx857"
Nov 02 03:17:43.991412 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:43.991379    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fj54p\" (UniqueName: \"kubernetes.io/projected/f095bbfb-6214-411c-a1e1-6f59ddf05837-kube-api-access-fj54p\") pod \"maas-api-revoke-sub-813da41b-mx857\" (UID: \"f095bbfb-6214-411c-a1e1-6f59ddf05837\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-813da41b-mx857"
Nov 02 03:17:43.991597 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:43.991464    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f095bbfb-6214-411c-a1e1-6f59ddf05837-maas-api-xks-ca\") pod \"maas-api-revoke-sub-813da41b-mx857\" (UID: \"f095bbfb-6214-411c-a1e1-6f59ddf05837\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-813da41b-mx857"
Nov 02 03:17:43.991597 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:43.991516    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f095bbfb-6214-411c-a1e1-6f59ddf05837-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-813da41b-mx857\" (UID: \"f095bbfb-6214-411c-a1e1-6f59ddf05837\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-813da41b-mx857"
Nov 02 03:17:43.992181 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:43.992157    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f095bbfb-6214-411c-a1e1-6f59ddf05837-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-813da41b-mx857\" (UID: \"f095bbfb-6214-411c-a1e1-6f59ddf05837\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-813da41b-mx857"
Nov 02 03:17:43.994091 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:43.994057    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f095bbfb-6214-411c-a1e1-6f59ddf05837-maas-api-xks-ca\") pod \"maas-api-revoke-sub-813da41b-mx857\" (UID: \"f095bbfb-6214-411c-a1e1-6f59ddf05837\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-813da41b-mx857"
Nov 02 03:17:44.049183 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:44.049142    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fj54p\" (UniqueName: \"kubernetes.io/projected/f095bbfb-6214-411c-a1e1-6f59ddf05837-kube-api-access-fj54p\") pod \"maas-api-revoke-sub-813da41b-mx857\" (UID: \"f095bbfb-6214-411c-a1e1-6f59ddf05837\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-813da41b-mx857"
Nov 02 03:17:44.098338 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:44.098307    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-813da41b-mx857"
Nov 02 03:17:44.151076 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:44.149626    4496 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-x5gwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body=
Nov 02 03:17:44.151076 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:44.149685    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-x5gwb" podUID="938e8e17-80ba-4b93-9f96-29e68f73f927" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused"
Nov 02 03:17:44.514463 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:17:44.514434    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf095bbfb_6214_411c_a1e1_6f59ddf05837.slice/crio-b8a99c4bee0b89a9f425fae20f1881b6a88005458a87d5e6a9536a806edb3a7b WatchSource:0}: Error finding container b8a99c4bee0b89a9f425fae20f1881b6a88005458a87d5e6a9536a806edb3a7b: Status 404 returned error can't find the container with id b8a99c4bee0b89a9f425fae20f1881b6a88005458a87d5e6a9536a806edb3a7b
Nov 02 03:17:44.516354 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:44.516329    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-813da41b-mx857"]
Nov 02 03:17:44.645724 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:44.645684    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-813da41b-mx857" event={"ID":"f095bbfb-6214-411c-a1e1-6f59ddf05837","Type":"ContainerStarted","Data":"b8a99c4bee0b89a9f425fae20f1881b6a88005458a87d5e6a9536a806edb3a7b"}
Nov 02 03:17:45.658556 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:45.658521    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-813da41b-mx857" event={"ID":"f095bbfb-6214-411c-a1e1-6f59ddf05837","Type":"ContainerStarted","Data":"baafaace65f1db2d444fdf3fff77de463f2ae1fa71435e82d8d40f0e5c224503"}
Nov 02 03:17:45.776365 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:45.776300    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-813da41b-mx857" podStartSLOduration=2.776283134 podStartE2EDuration="2.776283134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:17:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:17:45.77567343 +0000 UTC m=+1806.354280809" watchObservedRunningTime="2026-11-02 03:17:45.776283134 +0000 UTC m=+1806.354890514"
Nov 02 03:17:46.189207 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:46.189154    4496 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-x5gwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 03:17:46.189423 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:46.189218    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-x5gwb" podUID="938e8e17-80ba-4b93-9f96-29e68f73f927" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 03:17:46.373763 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:46.373726    4496 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-x5gwb"
Nov 02 03:17:46.374680 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:46.374654    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-x5gwb"
Nov 02 03:17:46.376179 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:46.376154    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-x5gwb"
Nov 02 03:17:46.673420 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:46.673379    4496 generic.go:356] "Generic (PLEG): container finished" podID="f095bbfb-6214-411c-a1e1-6f59ddf05837" containerID="baafaace65f1db2d444fdf3fff77de463f2ae1fa71435e82d8d40f0e5c224503" exitCode=0
Nov 02 03:17:46.674129 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:46.674086    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-813da41b-mx857" event={"ID":"f095bbfb-6214-411c-a1e1-6f59ddf05837","Type":"ContainerDied","Data":"baafaace65f1db2d444fdf3fff77de463f2ae1fa71435e82d8d40f0e5c224503"}
Nov 02 03:17:47.000278 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:47.000230    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-tpj5j"]
Nov 02 03:17:47.004563 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:47.004545    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-tpj5j"
Nov 02 03:17:47.089163 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:47.089135    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-tpj5j"]
Nov 02 03:17:47.132936 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:47.132904    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ce597fb-f95c-4569-9498-1dda5305706b-workload-certs\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-tpj5j\" (UID: \"8ce597fb-f95c-4569-9498-1dda5305706b\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-tpj5j"
Nov 02 03:17:47.132936 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:47.132940    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ce597fb-f95c-4569-9498-1dda5305706b-istio-token\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-tpj5j\" (UID: \"8ce597fb-f95c-4569-9498-1dda5305706b\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-tpj5j"
Nov 02 03:17:47.133228 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:47.132964    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ce597fb-f95c-4569-9498-1dda5305706b-istio-podinfo\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-tpj5j\" (UID: \"8ce597fb-f95c-4569-9498-1dda5305706b\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-tpj5j"
Nov 02 03:17:47.133228 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:47.133153    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ce597fb-f95c-4569-9498-1dda5305706b-istio-envoy\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-tpj5j\" (UID: \"8ce597fb-f95c-4569-9498-1dda5305706b\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-tpj5j"
Nov 02 03:17:47.133228 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:47.133216    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ce597fb-f95c-4569-9498-1dda5305706b-istio-data\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-tpj5j\" (UID: \"8ce597fb-f95c-4569-9498-1dda5305706b\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-tpj5j"
Nov 02 03:17:47.133393 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:47.133269    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ce597fb-f95c-4569-9498-1dda5305706b-workload-socket\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-tpj5j\" (UID: \"8ce597fb-f95c-4569-9498-1dda5305706b\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-tpj5j"
Nov 02 03:17:47.133393 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:47.133380    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ce597fb-f95c-4569-9498-1dda5305706b-istiod-ca-cert\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-tpj5j\" (UID: \"8ce597fb-f95c-4569-9498-1dda5305706b\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-tpj5j"
Nov 02 03:17:47.133505 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:47.133488    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ce597fb-f95c-4569-9498-1dda5305706b-credential-socket\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-tpj5j\" (UID: \"8ce597fb-f95c-4569-9498-1dda5305706b\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-tpj5j"
Nov 02 03:17:47.133546 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:47.133514    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t56jz\" (UniqueName: \"kubernetes.io/projected/8ce597fb-f95c-4569-9498-1dda5305706b-kube-api-access-t56jz\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-tpj5j\" (UID: \"8ce597fb-f95c-4569-9498-1dda5305706b\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-tpj5j"
Nov 02 03:17:47.234980 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:47.234945    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ce597fb-f95c-4569-9498-1dda5305706b-istio-podinfo\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-tpj5j\" (UID: \"8ce597fb-f95c-4569-9498-1dda5305706b\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-tpj5j"
Nov 02 03:17:47.235187 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:47.235081    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ce597fb-f95c-4569-9498-1dda5305706b-istio-envoy\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-tpj5j\" (UID: \"8ce597fb-f95c-4569-9498-1dda5305706b\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-tpj5j"
Nov 02 03:17:47.235187 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:47.235133    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ce597fb-f95c-4569-9498-1dda5305706b-istio-data\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-tpj5j\" (UID: \"8ce597fb-f95c-4569-9498-1dda5305706b\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-tpj5j"
Nov 02 03:17:47.235187 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:47.235180    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ce597fb-f95c-4569-9498-1dda5305706b-workload-socket\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-tpj5j\" (UID: \"8ce597fb-f95c-4569-9498-1dda5305706b\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-tpj5j"
Nov 02 03:17:47.235361 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:47.235247    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ce597fb-f95c-4569-9498-1dda5305706b-istiod-ca-cert\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-tpj5j\" (UID: \"8ce597fb-f95c-4569-9498-1dda5305706b\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-tpj5j"
Nov 02 03:17:47.235361 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:47.235337    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ce597fb-f95c-4569-9498-1dda5305706b-credential-socket\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-tpj5j\" (UID: \"8ce597fb-f95c-4569-9498-1dda5305706b\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-tpj5j"
Nov 02 03:17:47.235463 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:47.235367    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t56jz\" (UniqueName: \"kubernetes.io/projected/8ce597fb-f95c-4569-9498-1dda5305706b-kube-api-access-t56jz\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-tpj5j\" (UID: \"8ce597fb-f95c-4569-9498-1dda5305706b\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-tpj5j"
Nov 02 03:17:47.235463 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:47.235435    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ce597fb-f95c-4569-9498-1dda5305706b-workload-certs\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-tpj5j\" (UID: \"8ce597fb-f95c-4569-9498-1dda5305706b\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-tpj5j"
Nov 02 03:17:47.235564 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:47.235463    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ce597fb-f95c-4569-9498-1dda5305706b-istio-token\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-tpj5j\" (UID: \"8ce597fb-f95c-4569-9498-1dda5305706b\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-tpj5j"
Nov 02 03:17:47.235649 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:47.235622    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ce597fb-f95c-4569-9498-1dda5305706b-istio-data\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-tpj5j\" (UID: \"8ce597fb-f95c-4569-9498-1dda5305706b\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-tpj5j"
Nov 02 03:17:47.235822 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:47.235755    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ce597fb-f95c-4569-9498-1dda5305706b-credential-socket\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-tpj5j\" (UID: \"8ce597fb-f95c-4569-9498-1dda5305706b\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-tpj5j"
Nov 02 03:17:47.235822 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:47.235807    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ce597fb-f95c-4569-9498-1dda5305706b-workload-certs\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-tpj5j\" (UID: \"8ce597fb-f95c-4569-9498-1dda5305706b\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-tpj5j"
Nov 02 03:17:47.236066 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:47.236040    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ce597fb-f95c-4569-9498-1dda5305706b-workload-socket\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-tpj5j\" (UID: \"8ce597fb-f95c-4569-9498-1dda5305706b\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-tpj5j"
Nov 02 03:17:47.236192 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:47.236090    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ce597fb-f95c-4569-9498-1dda5305706b-istiod-ca-cert\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-tpj5j\" (UID: \"8ce597fb-f95c-4569-9498-1dda5305706b\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-tpj5j"
Nov 02 03:17:47.237598 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:47.237573    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ce597fb-f95c-4569-9498-1dda5305706b-istio-envoy\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-tpj5j\" (UID: \"8ce597fb-f95c-4569-9498-1dda5305706b\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-tpj5j"
Nov 02 03:17:47.237858 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:47.237838    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ce597fb-f95c-4569-9498-1dda5305706b-istio-podinfo\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-tpj5j\" (UID: \"8ce597fb-f95c-4569-9498-1dda5305706b\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-tpj5j"
Nov 02 03:17:47.293598 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:47.293530    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ce597fb-f95c-4569-9498-1dda5305706b-istio-token\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-tpj5j\" (UID: \"8ce597fb-f95c-4569-9498-1dda5305706b\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-tpj5j"
Nov 02 03:17:47.293849 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:47.293823    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t56jz\" (UniqueName: \"kubernetes.io/projected/8ce597fb-f95c-4569-9498-1dda5305706b-kube-api-access-t56jz\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-tpj5j\" (UID: \"8ce597fb-f95c-4569-9498-1dda5305706b\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-tpj5j"
Nov 02 03:17:47.319184 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:47.319160    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-tpj5j"
Nov 02 03:17:47.759553 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:47.759513    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-tpj5j"]
Nov 02 03:17:47.770595 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:47.770555    4496 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 03:17:47.770726 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:47.770639    4496 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 03:17:47.770726 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:47.770678    4496 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 03:17:47.816350 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:47.816328    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-813da41b-mx857"
Nov 02 03:17:47.945844 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:47.944932    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f095bbfb-6214-411c-a1e1-6f59ddf05837-kube-api-access-fj54p\" (UniqueName: \"kubernetes.io/projected/f095bbfb-6214-411c-a1e1-6f59ddf05837-kube-api-access-fj54p\") pod \"f095bbfb-6214-411c-a1e1-6f59ddf05837\" (UID: \"f095bbfb-6214-411c-a1e1-6f59ddf05837\") "
Nov 02 03:17:47.945844 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:47.944999    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f095bbfb-6214-411c-a1e1-6f59ddf05837-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f095bbfb-6214-411c-a1e1-6f59ddf05837-maas-api-xks-ca\") pod \"f095bbfb-6214-411c-a1e1-6f59ddf05837\" (UID: \"f095bbfb-6214-411c-a1e1-6f59ddf05837\") "
Nov 02 03:17:47.945844 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:47.945062    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f095bbfb-6214-411c-a1e1-6f59ddf05837-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f095bbfb-6214-411c-a1e1-6f59ddf05837-maas-api-openshift-ca\") pod \"f095bbfb-6214-411c-a1e1-6f59ddf05837\" (UID: \"f095bbfb-6214-411c-a1e1-6f59ddf05837\") "
Nov 02 03:17:47.946335 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:47.946296    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f095bbfb-6214-411c-a1e1-6f59ddf05837-maas-api-openshift-ca" pod "f095bbfb-6214-411c-a1e1-6f59ddf05837" (UID: "f095bbfb-6214-411c-a1e1-6f59ddf05837"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:17:47.948427 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:47.948401    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f095bbfb-6214-411c-a1e1-6f59ddf05837-kube-api-access-fj54p" pod "f095bbfb-6214-411c-a1e1-6f59ddf05837" (UID: "f095bbfb-6214-411c-a1e1-6f59ddf05837"). InnerVolumeSpecName "kube-api-access-fj54p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:17:47.948567 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:47.948550    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f095bbfb-6214-411c-a1e1-6f59ddf05837-maas-api-xks-ca" pod "f095bbfb-6214-411c-a1e1-6f59ddf05837" (UID: "f095bbfb-6214-411c-a1e1-6f59ddf05837"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 03:17:48.046961 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:48.046924    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fj54p\" (UniqueName: \"kubernetes.io/projected/f095bbfb-6214-411c-a1e1-6f59ddf05837-kube-api-access-fj54p\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:17:48.046961 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:48.046956    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f095bbfb-6214-411c-a1e1-6f59ddf05837-maas-api-xks-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:17:48.046961 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:48.046966    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f095bbfb-6214-411c-a1e1-6f59ddf05837-maas-api-openshift-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:17:48.689792 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:48.689687    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-tpj5j" event={"ID":"8ce597fb-f95c-4569-9498-1dda5305706b","Type":"ContainerStarted","Data":"bf5fabd88371ac16996a9a47b2f5e345b449fa80635798c1fa0283481d4bb783"}
Nov 02 03:17:48.689792 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:48.689761    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-tpj5j" event={"ID":"8ce597fb-f95c-4569-9498-1dda5305706b","Type":"ContainerStarted","Data":"83caa44067cd07e0070e9378d74b7b9d90d89671585194984088f51766f1cfef"}
Nov 02 03:17:48.691230 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:48.691201    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-813da41b-mx857"
Nov 02 03:17:48.691370 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:48.691248    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-813da41b-mx857" event={"ID":"f095bbfb-6214-411c-a1e1-6f59ddf05837","Type":"ContainerDied","Data":"b8a99c4bee0b89a9f425fae20f1881b6a88005458a87d5e6a9536a806edb3a7b"}
Nov 02 03:17:48.691370 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:48.691307    4496 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b8a99c4bee0b89a9f425fae20f1881b6a88005458a87d5e6a9536a806edb3a7b"
Nov 02 03:17:48.864149 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:48.864117    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7d14541-sfbvq"]
Nov 02 03:17:48.865242 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:48.865218    4496 cpu_manager.go:414] "containerMap: removing container" podUID="f095bbfb-6214-411c-a1e1-6f59ddf05837" containerName="revoke-keys"
Nov 02 03:17:48.865371 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:48.865360    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f095bbfb-6214-411c-a1e1-6f59ddf05837" containerName="revoke-keys"
Nov 02 03:17:48.865557 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:48.865546    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="f095bbfb-6214-411c-a1e1-6f59ddf05837" containerName="revoke-keys"
Nov 02 03:17:48.869546 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:48.869526    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7d14541-sfbvq"
Nov 02 03:17:48.887606 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:48.887566    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-tpj5j" podStartSLOduration=2.887554234 podStartE2EDuration="2.887554234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:17:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:17:48.871385182 +0000 UTC m=+1809.449992562" watchObservedRunningTime="2026-11-02 03:17:48.887554234 +0000 UTC m=+1809.466161614"
Nov 02 03:17:48.897678 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:48.897649    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7d14541-sfbvq"]
Nov 02 03:17:48.923160 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:48.923138    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 03:17:48.923308 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:48.923223    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 03:17:48.923308 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:48.923307    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 03:17:48.923451 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:48.923399    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9z5bk\""
Nov 02 03:17:49.066184 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:49.066134    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fc3e00d-f565-40cc-8c0d-2d409da9bffc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e7d14541-sfbvq\" (UID: \"4fc3e00d-f565-40cc-8c0d-2d409da9bffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7d14541-sfbvq"
Nov 02 03:17:49.066395 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:49.066295    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-th56g\" (UniqueName: \"kubernetes.io/projected/4fc3e00d-f565-40cc-8c0d-2d409da9bffc-kube-api-access-th56g\") pod \"maas-api-revoke-sub-e7d14541-sfbvq\" (UID: \"4fc3e00d-f565-40cc-8c0d-2d409da9bffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7d14541-sfbvq"
Nov 02 03:17:49.066499 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:49.066479    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fc3e00d-f565-40cc-8c0d-2d409da9bffc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e7d14541-sfbvq\" (UID: \"4fc3e00d-f565-40cc-8c0d-2d409da9bffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7d14541-sfbvq"
Nov 02 03:17:49.069948 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:49.069925    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec07dc13-jlg6j"]
Nov 02 03:17:49.073704 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:49.073684    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-181e39b9-mhtxr"]
Nov 02 03:17:49.077233 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:49.077219    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec07dc13-jlg6j"]
Nov 02 03:17:49.080727 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:49.080709    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-181e39b9-mhtxr"]
Nov 02 03:17:49.167855 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:49.167817    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fc3e00d-f565-40cc-8c0d-2d409da9bffc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e7d14541-sfbvq\" (UID: \"4fc3e00d-f565-40cc-8c0d-2d409da9bffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7d14541-sfbvq"
Nov 02 03:17:49.168063 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:49.167893    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-th56g\" (UniqueName: \"kubernetes.io/projected/4fc3e00d-f565-40cc-8c0d-2d409da9bffc-kube-api-access-th56g\") pod \"maas-api-revoke-sub-e7d14541-sfbvq\" (UID: \"4fc3e00d-f565-40cc-8c0d-2d409da9bffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7d14541-sfbvq"
Nov 02 03:17:49.168063 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:49.168051    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fc3e00d-f565-40cc-8c0d-2d409da9bffc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e7d14541-sfbvq\" (UID: \"4fc3e00d-f565-40cc-8c0d-2d409da9bffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7d14541-sfbvq"
Nov 02 03:17:49.168676 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:49.168656    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fc3e00d-f565-40cc-8c0d-2d409da9bffc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e7d14541-sfbvq\" (UID: \"4fc3e00d-f565-40cc-8c0d-2d409da9bffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7d14541-sfbvq"
Nov 02 03:17:49.170259 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:49.170235    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fc3e00d-f565-40cc-8c0d-2d409da9bffc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e7d14541-sfbvq\" (UID: \"4fc3e00d-f565-40cc-8c0d-2d409da9bffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7d14541-sfbvq"
Nov 02 03:17:49.225540 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:49.225510    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-th56g\" (UniqueName: \"kubernetes.io/projected/4fc3e00d-f565-40cc-8c0d-2d409da9bffc-kube-api-access-th56g\") pod \"maas-api-revoke-sub-e7d14541-sfbvq\" (UID: \"4fc3e00d-f565-40cc-8c0d-2d409da9bffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7d14541-sfbvq"
Nov 02 03:17:49.320354 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:49.320273    4496 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-tpj5j"
Nov 02 03:17:49.321865 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:49.321841    4496 patch_prober.go:29] interesting pod/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-tpj5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body=
Nov 02 03:17:49.321961 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:49.321919    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-tpj5j" podUID="8ce597fb-f95c-4569-9498-1dda5305706b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused"
Nov 02 03:17:49.485555 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:49.485525    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7d14541-sfbvq"
Nov 02 03:17:49.581061 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:49.580985    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c2c1ce9-e204-4204-9793-7d46595601ba" path="/var/lib/kubelet/pods/9c2c1ce9-e204-4204-9793-7d46595601ba/volumes"
Nov 02 03:17:49.581741 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:49.581682    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7b60965-efe4-4d29-baa5-2e14bc094e1b" path="/var/lib/kubelet/pods/e7b60965-efe4-4d29-baa5-2e14bc094e1b/volumes"
Nov 02 03:17:49.904117 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:17:49.894960    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fc3e00d_f565_40cc_8c0d_2d409da9bffc.slice/crio-ca7624918df2b085d80ec968deb3b48b2188cce6a4e5f92faae48cf571a15331 WatchSource:0}: Error finding container ca7624918df2b085d80ec968deb3b48b2188cce6a4e5f92faae48cf571a15331: Status 404 returned error can't find the container with id ca7624918df2b085d80ec968deb3b48b2188cce6a4e5f92faae48cf571a15331
Nov 02 03:17:49.904117 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:49.895263    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7d14541-sfbvq"]
Nov 02 03:17:50.319997 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:50.319960    4496 patch_prober.go:29] interesting pod/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-tpj5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body=
Nov 02 03:17:50.320158 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:50.320043    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-tpj5j" podUID="8ce597fb-f95c-4569-9498-1dda5305706b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused"
Nov 02 03:17:50.709496 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:50.709447    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7d14541-sfbvq" event={"ID":"4fc3e00d-f565-40cc-8c0d-2d409da9bffc","Type":"ContainerStarted","Data":"369be73cdc34cca261cf0982987e2e7dc485000865fd2ac9cdf7847bc5d9bdbd"}
Nov 02 03:17:50.709676 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:50.709545    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7d14541-sfbvq" event={"ID":"4fc3e00d-f565-40cc-8c0d-2d409da9bffc","Type":"ContainerStarted","Data":"ca7624918df2b085d80ec968deb3b48b2188cce6a4e5f92faae48cf571a15331"}
Nov 02 03:17:50.828223 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:50.828174    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7d14541-sfbvq" podStartSLOduration=2.828159181 podStartE2EDuration="2.828159181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:17:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:17:50.823454375 +0000 UTC m=+1811.402061758" watchObservedRunningTime="2026-11-02 03:17:50.828159181 +0000 UTC m=+1811.406766591"
Nov 02 03:17:51.319714 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:51.319635    4496 patch_prober.go:29] interesting pod/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-tpj5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body=
Nov 02 03:17:51.319714 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:51.319691    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-tpj5j" podUID="8ce597fb-f95c-4569-9498-1dda5305706b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused"
Nov 02 03:17:51.484482 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:51.484448    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a96d142e-8ck27"]
Nov 02 03:17:51.488931 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:51.488905    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a96d142e-8ck27"
Nov 02 03:17:51.494691 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:51.494664    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdaadbc3-63c0-4353-b3b6-576f63617326-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a96d142e-8ck27\" (UID: \"fdaadbc3-63c0-4353-b3b6-576f63617326\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a96d142e-8ck27"
Nov 02 03:17:51.494805 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:51.494704    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdaadbc3-63c0-4353-b3b6-576f63617326-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a96d142e-8ck27\" (UID: \"fdaadbc3-63c0-4353-b3b6-576f63617326\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a96d142e-8ck27"
Nov 02 03:17:51.494861 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:51.494845    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xq97c\" (UniqueName: \"kubernetes.io/projected/fdaadbc3-63c0-4353-b3b6-576f63617326-kube-api-access-xq97c\") pod \"maas-api-revoke-sub-a96d142e-8ck27\" (UID: \"fdaadbc3-63c0-4353-b3b6-576f63617326\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a96d142e-8ck27"
Nov 02 03:17:51.585670 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:51.585594    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a96d142e-8ck27"]
Nov 02 03:17:51.596677 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:51.596644    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdaadbc3-63c0-4353-b3b6-576f63617326-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a96d142e-8ck27\" (UID: \"fdaadbc3-63c0-4353-b3b6-576f63617326\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a96d142e-8ck27"
Nov 02 03:17:51.596933 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:51.596913    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdaadbc3-63c0-4353-b3b6-576f63617326-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a96d142e-8ck27\" (UID: \"fdaadbc3-63c0-4353-b3b6-576f63617326\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a96d142e-8ck27"
Nov 02 03:17:51.597238 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:51.597215    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xq97c\" (UniqueName: \"kubernetes.io/projected/fdaadbc3-63c0-4353-b3b6-576f63617326-kube-api-access-xq97c\") pod \"maas-api-revoke-sub-a96d142e-8ck27\" (UID: \"fdaadbc3-63c0-4353-b3b6-576f63617326\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a96d142e-8ck27"
Nov 02 03:17:51.598661 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:51.598578    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdaadbc3-63c0-4353-b3b6-576f63617326-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a96d142e-8ck27\" (UID: \"fdaadbc3-63c0-4353-b3b6-576f63617326\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a96d142e-8ck27"
Nov 02 03:17:51.600669 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:51.600642    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdaadbc3-63c0-4353-b3b6-576f63617326-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a96d142e-8ck27\" (UID: \"fdaadbc3-63c0-4353-b3b6-576f63617326\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a96d142e-8ck27"
Nov 02 03:17:51.666717 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:51.666680    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xq97c\" (UniqueName: \"kubernetes.io/projected/fdaadbc3-63c0-4353-b3b6-576f63617326-kube-api-access-xq97c\") pod \"maas-api-revoke-sub-a96d142e-8ck27\" (UID: \"fdaadbc3-63c0-4353-b3b6-576f63617326\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a96d142e-8ck27"
Nov 02 03:17:51.719087 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:51.719047    4496 generic.go:356] "Generic (PLEG): container finished" podID="4fc3e00d-f565-40cc-8c0d-2d409da9bffc" containerID="369be73cdc34cca261cf0982987e2e7dc485000865fd2ac9cdf7847bc5d9bdbd" exitCode=0
Nov 02 03:17:51.719237 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:51.719116    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7d14541-sfbvq" event={"ID":"4fc3e00d-f565-40cc-8c0d-2d409da9bffc","Type":"ContainerDied","Data":"369be73cdc34cca261cf0982987e2e7dc485000865fd2ac9cdf7847bc5d9bdbd"}
Nov 02 03:17:51.804067 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:51.804032    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a96d142e-8ck27"
Nov 02 03:17:52.216953 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:52.216924    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a96d142e-8ck27"]
Nov 02 03:17:52.217470 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:17:52.217444    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdaadbc3_63c0_4353_b3b6_576f63617326.slice/crio-7948c4db6db98b61d2d65c44fbb53c42f7e74acd0989673111babd9b36c620b3 WatchSource:0}: Error finding container 7948c4db6db98b61d2d65c44fbb53c42f7e74acd0989673111babd9b36c620b3: Status 404 returned error can't find the container with id 7948c4db6db98b61d2d65c44fbb53c42f7e74acd0989673111babd9b36c620b3
Nov 02 03:17:52.320217 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:52.320191    4496 patch_prober.go:29] interesting pod/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-tpj5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body=
Nov 02 03:17:52.320521 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:52.320244    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-tpj5j" podUID="8ce597fb-f95c-4569-9498-1dda5305706b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused"
Nov 02 03:17:52.728397 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:52.728364    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a96d142e-8ck27" event={"ID":"fdaadbc3-63c0-4353-b3b6-576f63617326","Type":"ContainerStarted","Data":"8959e0d0edc25e0de08cecda67df9b27d7cbcc08ab09affc05dc8e21bb17bb25"}
Nov 02 03:17:52.728541 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:52.728422    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a96d142e-8ck27" event={"ID":"fdaadbc3-63c0-4353-b3b6-576f63617326","Type":"ContainerStarted","Data":"7948c4db6db98b61d2d65c44fbb53c42f7e74acd0989673111babd9b36c620b3"}
Nov 02 03:17:52.863591 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:52.863526    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a96d142e-8ck27" podStartSLOduration=1.8635042419999999 podStartE2EDuration="1.863504242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:17:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:17:52.850251278 +0000 UTC m=+1813.428858662" watchObservedRunningTime="2026-11-02 03:17:52.863504242 +0000 UTC m=+1813.442111623"
Nov 02 03:17:52.965070 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:52.963756    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7d14541-sfbvq"
Nov 02 03:17:53.015272 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:53.015194    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4fc3e00d-f565-40cc-8c0d-2d409da9bffc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fc3e00d-f565-40cc-8c0d-2d409da9bffc-maas-api-openshift-ca\") pod \"4fc3e00d-f565-40cc-8c0d-2d409da9bffc\" (UID: \"4fc3e00d-f565-40cc-8c0d-2d409da9bffc\") "
Nov 02 03:17:53.015434 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:53.015300    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fc3e00d-f565-40cc-8c0d-2d409da9bffc-kube-api-access-th56g\" (UniqueName: \"kubernetes.io/projected/4fc3e00d-f565-40cc-8c0d-2d409da9bffc-kube-api-access-th56g\") pod \"4fc3e00d-f565-40cc-8c0d-2d409da9bffc\" (UID: \"4fc3e00d-f565-40cc-8c0d-2d409da9bffc\") "
Nov 02 03:17:53.015434 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:53.015355    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4fc3e00d-f565-40cc-8c0d-2d409da9bffc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fc3e00d-f565-40cc-8c0d-2d409da9bffc-maas-api-xks-ca\") pod \"4fc3e00d-f565-40cc-8c0d-2d409da9bffc\" (UID: \"4fc3e00d-f565-40cc-8c0d-2d409da9bffc\") "
Nov 02 03:17:53.015613 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:53.015588    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fc3e00d-f565-40cc-8c0d-2d409da9bffc-maas-api-openshift-ca" pod "4fc3e00d-f565-40cc-8c0d-2d409da9bffc" (UID: "4fc3e00d-f565-40cc-8c0d-2d409da9bffc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:17:53.015836 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:53.015816    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fc3e00d-f565-40cc-8c0d-2d409da9bffc-maas-api-openshift-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:17:53.017484 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:53.017452    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fc3e00d-f565-40cc-8c0d-2d409da9bffc-maas-api-xks-ca" pod "4fc3e00d-f565-40cc-8c0d-2d409da9bffc" (UID: "4fc3e00d-f565-40cc-8c0d-2d409da9bffc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 03:17:53.017708 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:53.017686    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fc3e00d-f565-40cc-8c0d-2d409da9bffc-kube-api-access-th56g" pod "4fc3e00d-f565-40cc-8c0d-2d409da9bffc" (UID: "4fc3e00d-f565-40cc-8c0d-2d409da9bffc"). InnerVolumeSpecName "kube-api-access-th56g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:17:53.116992 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:53.116949    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-th56g\" (UniqueName: \"kubernetes.io/projected/4fc3e00d-f565-40cc-8c0d-2d409da9bffc-kube-api-access-th56g\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:17:53.116992 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:53.116985    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fc3e00d-f565-40cc-8c0d-2d409da9bffc-maas-api-xks-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:17:53.319579 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:53.319502    4496 patch_prober.go:29] interesting pod/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-tpj5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body=
Nov 02 03:17:53.319579 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:53.319557    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-tpj5j" podUID="8ce597fb-f95c-4569-9498-1dda5305706b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused"
Nov 02 03:17:53.760797 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:53.760769    4496 generic.go:356] "Generic (PLEG): container finished" podID="fdaadbc3-63c0-4353-b3b6-576f63617326" containerID="8959e0d0edc25e0de08cecda67df9b27d7cbcc08ab09affc05dc8e21bb17bb25" exitCode=0
Nov 02 03:17:53.761297 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:53.760929    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a96d142e-8ck27" event={"ID":"fdaadbc3-63c0-4353-b3b6-576f63617326","Type":"ContainerDied","Data":"8959e0d0edc25e0de08cecda67df9b27d7cbcc08ab09affc05dc8e21bb17bb25"}
Nov 02 03:17:53.763272 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:53.763248    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7d14541-sfbvq"
Nov 02 03:17:53.763374 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:53.763268    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7d14541-sfbvq" event={"ID":"4fc3e00d-f565-40cc-8c0d-2d409da9bffc","Type":"ContainerDied","Data":"ca7624918df2b085d80ec968deb3b48b2188cce6a4e5f92faae48cf571a15331"}
Nov 02 03:17:53.763374 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:53.763310    4496 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca7624918df2b085d80ec968deb3b48b2188cce6a4e5f92faae48cf571a15331"
Nov 02 03:17:54.320401 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:54.320360    4496 patch_prober.go:29] interesting pod/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-tpj5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body=
Nov 02 03:17:54.320588 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:54.320424    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-tpj5j" podUID="8ce597fb-f95c-4569-9498-1dda5305706b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused"
Nov 02 03:17:54.963765 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:54.963739    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a96d142e-8ck27"
Nov 02 03:17:55.041231 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:55.041154    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdaadbc3-63c0-4353-b3b6-576f63617326-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdaadbc3-63c0-4353-b3b6-576f63617326-maas-api-openshift-ca\") pod \"fdaadbc3-63c0-4353-b3b6-576f63617326\" (UID: \"fdaadbc3-63c0-4353-b3b6-576f63617326\") "
Nov 02 03:17:55.041379 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:55.041259    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fdaadbc3-63c0-4353-b3b6-576f63617326-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdaadbc3-63c0-4353-b3b6-576f63617326-maas-api-xks-ca\") pod \"fdaadbc3-63c0-4353-b3b6-576f63617326\" (UID: \"fdaadbc3-63c0-4353-b3b6-576f63617326\") "
Nov 02 03:17:55.041379 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:55.041283    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdaadbc3-63c0-4353-b3b6-576f63617326-kube-api-access-xq97c\" (UniqueName: \"kubernetes.io/projected/fdaadbc3-63c0-4353-b3b6-576f63617326-kube-api-access-xq97c\") pod \"fdaadbc3-63c0-4353-b3b6-576f63617326\" (UID: \"fdaadbc3-63c0-4353-b3b6-576f63617326\") "
Nov 02 03:17:55.041610 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:55.041582    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdaadbc3-63c0-4353-b3b6-576f63617326-maas-api-openshift-ca" pod "fdaadbc3-63c0-4353-b3b6-576f63617326" (UID: "fdaadbc3-63c0-4353-b3b6-576f63617326"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:17:55.041806 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:55.041778    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdaadbc3-63c0-4353-b3b6-576f63617326-maas-api-openshift-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:17:55.043504 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:55.043480    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdaadbc3-63c0-4353-b3b6-576f63617326-kube-api-access-xq97c" pod "fdaadbc3-63c0-4353-b3b6-576f63617326" (UID: "fdaadbc3-63c0-4353-b3b6-576f63617326"). InnerVolumeSpecName "kube-api-access-xq97c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:17:55.043623 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:55.043486    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdaadbc3-63c0-4353-b3b6-576f63617326-maas-api-xks-ca" pod "fdaadbc3-63c0-4353-b3b6-576f63617326" (UID: "fdaadbc3-63c0-4353-b3b6-576f63617326"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 03:17:55.143038 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:55.142983    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdaadbc3-63c0-4353-b3b6-576f63617326-maas-api-xks-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:17:55.143038 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:55.143027    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xq97c\" (UniqueName: \"kubernetes.io/projected/fdaadbc3-63c0-4353-b3b6-576f63617326-kube-api-access-xq97c\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:17:55.326325 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:55.326232    4496 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-tpj5j"
Nov 02 03:17:55.326791 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:55.326735    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-tpj5j"
Nov 02 03:17:55.327886 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:55.327849    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-tpj5j"
Nov 02 03:17:55.782534 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:55.782505    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a96d142e-8ck27"
Nov 02 03:17:55.782689 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:55.782552    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a96d142e-8ck27" event={"ID":"fdaadbc3-63c0-4353-b3b6-576f63617326","Type":"ContainerDied","Data":"7948c4db6db98b61d2d65c44fbb53c42f7e74acd0989673111babd9b36c620b3"}
Nov 02 03:17:55.782689 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:17:55.782592    4496 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7948c4db6db98b61d2d65c44fbb53c42f7e74acd0989673111babd9b36c620b3"
Nov 02 03:18:03.531744 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:18:03.531700    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 03:18:04.089037 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:04.080273    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ce700f8-fc6fc"]
Nov 02 03:18:04.162041 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:04.161467    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ce700f8-fc6fc"]
Nov 02 03:18:04.910389 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:04.909152    4496 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ca6546/simulator-w2-ca6546-kserve-5959bb649b-5x6tf" podUID="4fd6a8f7-0b10-4009-a6db-de0a01aca2b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 03:18:05.436039 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:05.424204    4496 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5ac93f/distinct-w1-5ac93f-kserve-64697cb8c9-vj9sk" podUID="75a1c221-38e8-47cf-a8b5-305396cee703" containerName="main" probeResult="failure" output="Get \"https://10.134.0.102:8000/health\": context deadline exceeded"
Nov 02 03:18:05.588040 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:05.584625    4496 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xlbmw" podUID="2baeda96-f36e-49cb-bb8a-3389c9e8d61d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 03:18:05.932694 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:05.932645    4496 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-398828/simulator-w4-398828-kserve-6d5bd49c6f-9tw2c" podUID="5b6aecfa-3e3b-4fb3-a80c-35b77de60ce2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 03:18:05.935929 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:05.935895    4496 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ca6546/simulator-w2-ca6546-kserve-5959bb649b-5x6tf" podUID="4fd6a8f7-0b10-4009-a6db-de0a01aca2b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": context deadline exceeded"
Nov 02 03:18:05.963035 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:05.952260    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a95cfe25-614a-4e18-a401-5ff9c2b2772a" path="/var/lib/kubelet/pods/a95cfe25-614a-4e18-a401-5ff9c2b2772a/volumes"
Nov 02 03:18:05.963756 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:05.963732    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d5dcca72-846bb99569-wldb8"]
Nov 02 03:18:05.964825 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:05.964799    4496 cpu_manager.go:414] "containerMap: removing container" podUID="4fc3e00d-f565-40cc-8c0d-2d409da9bffc" containerName="revoke-keys"
Nov 02 03:18:05.964965 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:05.964951    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fc3e00d-f565-40cc-8c0d-2d409da9bffc" containerName="revoke-keys"
Nov 02 03:18:05.965251 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:05.965235    4496 cpu_manager.go:414] "containerMap: removing container" podUID="fdaadbc3-63c0-4353-b3b6-576f63617326" containerName="revoke-keys"
Nov 02 03:18:05.965359 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:05.965345    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdaadbc3-63c0-4353-b3b6-576f63617326" containerName="revoke-keys"
Nov 02 03:18:05.965662 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:05.965645    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fc3e00d-f565-40cc-8c0d-2d409da9bffc" containerName="revoke-keys"
Nov 02 03:18:05.965846 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:05.965830    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdaadbc3-63c0-4353-b3b6-576f63617326" containerName="revoke-keys"
Nov 02 03:18:06.001581 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:06.001553    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d5dcca72-846bb99569-wldb8"]
Nov 02 03:18:06.001965 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:06.001945    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-ca6546/simulator-w2-ca6546-kserve-5959bb649b-5x6tf"
Nov 02 03:18:06.002630 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:06.001852    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d5dcca72-846bb99569-wldb8"
Nov 02 03:18:06.020037 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:06.009335    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d5dcca72-7746cbb699-h8cx9"]
Nov 02 03:18:06.057453 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:06.057425    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d5dcca72-7746cbb699-h8cx9"]
Nov 02 03:18:06.057760 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:06.057740    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d5dcca72-7746cbb699-h8cx9"
Nov 02 03:18:06.081038 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:06.076645    4496 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-398828/distinct-2-w4-398828-kserve-545449d49b-6mg4b" podUID="36c1912c-2ab1-40c8-a09d-2425ab08ec8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.125:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 03:18:06.081038 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:06.080922    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvsvd\" (UniqueName: \"kubernetes.io/projected/e54dff57-76c5-4a6f-bd9f-b59e6444b307-kube-api-access-jvsvd\") pod \"payload-pre-processing-e2e-mt-d5dcca72-7746cbb699-h8cx9\" (UID: \"e54dff57-76c5-4a6f-bd9f-b59e6444b307\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d5dcca72-7746cbb699-h8cx9"
Nov 02 03:18:06.081038 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:06.080991    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgwwj\" (UniqueName: \"kubernetes.io/projected/473be601-bbd9-4dcb-89ec-e43b66619766-kube-api-access-wgwwj\") pod \"payload-processing-e2e-mt-d5dcca72-846bb99569-wldb8\" (UID: \"473be601-bbd9-4dcb-89ec-e43b66619766\") " pod="openshift-ingress/payload-processing-e2e-mt-d5dcca72-846bb99569-wldb8"
Nov 02 03:18:06.081322 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:06.081073    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e54dff57-76c5-4a6f-bd9f-b59e6444b307-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d5dcca72-7746cbb699-h8cx9\" (UID: \"e54dff57-76c5-4a6f-bd9f-b59e6444b307\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d5dcca72-7746cbb699-h8cx9"
Nov 02 03:18:06.081322 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:06.081241    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/473be601-bbd9-4dcb-89ec-e43b66619766-plugins-config-volume\") pod \"payload-processing-e2e-mt-d5dcca72-846bb99569-wldb8\" (UID: \"473be601-bbd9-4dcb-89ec-e43b66619766\") " pod="openshift-ingress/payload-processing-e2e-mt-d5dcca72-846bb99569-wldb8"
Nov 02 03:18:06.105038 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:06.095262    4496 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5ac93f/distinct-w1-5ac93f-kserve-64697cb8c9-vj9sk" podUID="75a1c221-38e8-47cf-a8b5-305396cee703" containerName="main" probeResult="failure" output="Get \"https://10.134.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 03:18:06.113037 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:06.111191    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-d5dcca72\""
Nov 02 03:18:06.113037 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:06.111301    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-d5dcca72-dockercfg-2qhx9\""
Nov 02 03:18:06.137979 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:06.137940    4496 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-858236/distinct-w3-858236-kserve-df6dbd64b-qbk9g" podUID="719bb9e5-ea72-4782-bb17-034b909f874d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.126:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 03:18:06.138832 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:06.138799    4496 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-398828/distinct-2-w4-398828-kserve-545449d49b-6mg4b" podUID="36c1912c-2ab1-40c8-a09d-2425ab08ec8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.125:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 03:18:06.255787 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:06.255755    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/473be601-bbd9-4dcb-89ec-e43b66619766-plugins-config-volume\") pod \"payload-processing-e2e-mt-d5dcca72-846bb99569-wldb8\" (UID: \"473be601-bbd9-4dcb-89ec-e43b66619766\") " pod="openshift-ingress/payload-processing-e2e-mt-d5dcca72-846bb99569-wldb8"
Nov 02 03:18:06.256175 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:06.256149    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvsvd\" (UniqueName: \"kubernetes.io/projected/e54dff57-76c5-4a6f-bd9f-b59e6444b307-kube-api-access-jvsvd\") pod \"payload-pre-processing-e2e-mt-d5dcca72-7746cbb699-h8cx9\" (UID: \"e54dff57-76c5-4a6f-bd9f-b59e6444b307\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d5dcca72-7746cbb699-h8cx9"
Nov 02 03:18:06.260091 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:06.260065    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgwwj\" (UniqueName: \"kubernetes.io/projected/473be601-bbd9-4dcb-89ec-e43b66619766-kube-api-access-wgwwj\") pod \"payload-processing-e2e-mt-d5dcca72-846bb99569-wldb8\" (UID: \"473be601-bbd9-4dcb-89ec-e43b66619766\") " pod="openshift-ingress/payload-processing-e2e-mt-d5dcca72-846bb99569-wldb8"
Nov 02 03:18:06.268036 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:06.264132    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e54dff57-76c5-4a6f-bd9f-b59e6444b307-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d5dcca72-7746cbb699-h8cx9\" (UID: \"e54dff57-76c5-4a6f-bd9f-b59e6444b307\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d5dcca72-7746cbb699-h8cx9"
Nov 02 03:18:06.272076 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:06.263638    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/473be601-bbd9-4dcb-89ec-e43b66619766-plugins-config-volume\") pod \"payload-processing-e2e-mt-d5dcca72-846bb99569-wldb8\" (UID: \"473be601-bbd9-4dcb-89ec-e43b66619766\") " pod="openshift-ingress/payload-processing-e2e-mt-d5dcca72-846bb99569-wldb8"
Nov 02 03:18:06.288120 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:06.266302    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e54dff57-76c5-4a6f-bd9f-b59e6444b307-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d5dcca72-7746cbb699-h8cx9\" (UID: \"e54dff57-76c5-4a6f-bd9f-b59e6444b307\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d5dcca72-7746cbb699-h8cx9"
Nov 02 03:18:06.806041 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:06.789222    4496 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xlbmw" podUID="2baeda96-f36e-49cb-bb8a-3389c9e8d61d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 02 03:18:06.825037 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:06.808371    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xlbmw"
Nov 02 03:18:06.970114 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:06.968602    4496 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-398828/simulator-w4-398828-kserve-6d5bd49c6f-9tw2c" podUID="5b6aecfa-3e3b-4fb3-a80c-35b77de60ce2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 03:18:07.208107 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:07.207768    4496 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5ac93f/distinct-w1-5ac93f-kserve-64697cb8c9-vj9sk" podUID="75a1c221-38e8-47cf-a8b5-305396cee703" containerName="main" probeResult="failure" output="Get \"https://10.134.0.102:8000/ready\": context deadline exceeded"
Nov 02 03:18:07.415039 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:07.412992    4496 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-398828/distinct-2-w4-398828-kserve-545449d49b-6mg4b" podUID="36c1912c-2ab1-40c8-a09d-2425ab08ec8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.125:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 03:18:07.415238 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:07.415115    4496 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ca6546/simulator-w2-ca6546-kserve-5959bb649b-5x6tf" podUID="4fd6a8f7-0b10-4009-a6db-de0a01aca2b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": context deadline exceeded"
Nov 02 03:18:07.419035 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:07.415481    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvsvd\" (UniqueName: \"kubernetes.io/projected/e54dff57-76c5-4a6f-bd9f-b59e6444b307-kube-api-access-jvsvd\") pod \"payload-pre-processing-e2e-mt-d5dcca72-7746cbb699-h8cx9\" (UID: \"e54dff57-76c5-4a6f-bd9f-b59e6444b307\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d5dcca72-7746cbb699-h8cx9"
Nov 02 03:18:07.419035 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:07.416811    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-398828/simulator-w4-398828-kserve-6d5bd49c6f-9tw2c"
Nov 02 03:18:07.444064 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:07.444032    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5ac93f/distinct-w1-5ac93f-kserve-64697cb8c9-vj9sk"
Nov 02 03:18:07.444356 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:07.444341    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-398828/distinct-2-w4-398828-kserve-545449d49b-6mg4b"
Nov 02 03:18:07.467086 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:07.447823    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgwwj\" (UniqueName: \"kubernetes.io/projected/473be601-bbd9-4dcb-89ec-e43b66619766-kube-api-access-wgwwj\") pod \"payload-processing-e2e-mt-d5dcca72-846bb99569-wldb8\" (UID: \"473be601-bbd9-4dcb-89ec-e43b66619766\") " pod="openshift-ingress/payload-processing-e2e-mt-d5dcca72-846bb99569-wldb8"
Nov 02 03:18:07.470664 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:07.470617    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d5dcca72-846bb99569-wldb8"
Nov 02 03:18:07.476331 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:07.476294    4496 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-858236/distinct-w3-858236-kserve-df6dbd64b-qbk9g" podUID="719bb9e5-ea72-4782-bb17-034b909f874d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.126:8000/ready\": context deadline exceeded"
Nov 02 03:18:07.491053 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:07.488911    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-858236/distinct-w3-858236-kserve-df6dbd64b-qbk9g"
Nov 02 03:18:07.628599 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:07.628566    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d5dcca72-7746cbb699-h8cx9"
Nov 02 03:18:07.918583 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:07.904781    4496 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xlbmw" podUID="2baeda96-f36e-49cb-bb8a-3389c9e8d61d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 03:18:08.003182 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:08.003137    4496 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-398828/simulator-w4-398828-kserve-6d5bd49c6f-9tw2c" podUID="5b6aecfa-3e3b-4fb3-a80c-35b77de60ce2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 03:18:08.379819 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:08.379781    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xlbmw"
Nov 02 03:18:08.482036 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:08.466133    4496 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5ac93f/distinct-w1-5ac93f-kserve-64697cb8c9-vj9sk" podUID="75a1c221-38e8-47cf-a8b5-305396cee703" containerName="main" probeResult="failure" output="Get \"https://10.134.0.102:8000/ready\": context deadline exceeded"
Nov 02 03:18:08.482036 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:08.469186    4496 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ca6546/simulator-w2-ca6546-kserve-5959bb649b-5x6tf" podUID="4fd6a8f7-0b10-4009-a6db-de0a01aca2b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 03:18:08.482036 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:08.469254    4496 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-398828/distinct-2-w4-398828-kserve-545449d49b-6mg4b" podUID="36c1912c-2ab1-40c8-a09d-2425ab08ec8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.125:8000/ready\": context deadline exceeded"
Nov 02 03:18:08.482036 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:08.469824    4496 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vnxgf" podUID="bebc74da-01c1-4547-ab5d-90a0a8ec1ad7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 03:18:08.562097 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:08.537290    4496 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-858236/distinct-w3-858236-kserve-df6dbd64b-qbk9g" podUID="719bb9e5-ea72-4782-bb17-034b909f874d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.126:8000/ready\": context deadline exceeded"
Nov 02 03:18:08.771214 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:08.771165    4496 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-ca6546/simulator-w2-ca6546-kserve-5959bb649b-5x6tf" podUID="4fd6a8f7-0b10-4009-a6db-de0a01aca2b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/health\": context deadline exceeded"
Nov 02 03:18:09.139167 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:09.139124    4496 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-398828/simulator-w4-398828-kserve-6d5bd49c6f-9tw2c" podUID="5b6aecfa-3e3b-4fb3-a80c-35b77de60ce2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded"
Nov 02 03:18:09.200088 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:09.200057    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5ac93f/distinct-w1-5ac93f-kserve-64697cb8c9-vj9sk"
Nov 02 03:18:09.214524 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:09.214487    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-ca6546/simulator-w2-ca6546-kserve-5959bb649b-5x6tf"
Nov 02 03:18:09.220409 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:09.217373    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-398828/distinct-2-w4-398828-kserve-545449d49b-6mg4b"
Nov 02 03:18:09.300578 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:09.300549    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-858236/distinct-w3-858236-kserve-df6dbd64b-qbk9g"
Nov 02 03:18:09.302534 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:09.302448    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-398828/simulator-w4-398828-kserve-6d5bd49c6f-9tw2c"
Nov 02 03:18:09.570404 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:09.570350    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1744e3a8-bsqhh"]
Nov 02 03:18:09.570705 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:09.570672    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1744e3a8-bsqhh"]
Nov 02 03:18:09.570881 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:09.570865    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d5dcca72-7746cbb699-h8cx9"]
Nov 02 03:18:09.703269 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:09.703234    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d15dd94-66d5-475e-9ec8-55b043fe3597" path="/var/lib/kubelet/pods/4d15dd94-66d5-475e-9ec8-55b043fe3597/volumes"
Nov 02 03:18:09.708323 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:09.708292    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d5dcca72-7746cbb699-h8cx9" event={"ID":"e54dff57-76c5-4a6f-bd9f-b59e6444b307","Type":"ContainerStarted","Data":"cd703a792c2b8ab031b4fbf3bd8a57b5ded0c374ee55dbeb7974664eb4f49e96"}
Nov 02 03:18:09.794412 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:18:09.794364    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod473be601_bbd9_4dcb_89ec_e43b66619766.slice/crio-647e0619114b94b09ef98e908c59d13b89c4c0b632cc0b342ea5a626c1227c7c WatchSource:0}: Error finding container 647e0619114b94b09ef98e908c59d13b89c4c0b632cc0b342ea5a626c1227c7c: Status 404 returned error can't find the container with id 647e0619114b94b09ef98e908c59d13b89c4c0b632cc0b342ea5a626c1227c7c
Nov 02 03:18:09.802355 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:09.795415    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d5dcca72-846bb99569-wldb8"]
Nov 02 03:18:10.627506 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:10.627472    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1606467d-4g8s2"]
Nov 02 03:18:10.631934 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:10.631913    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1606467d-4g8s2"
Nov 02 03:18:10.685071 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:10.685040    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 03:18:10.685222 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:10.685091    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 03:18:10.685222 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:10.685155    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 03:18:10.688537 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:10.688508    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1606467d-4g8s2"]
Nov 02 03:18:10.719121 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:10.719081    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d5dcca72-846bb99569-wldb8" event={"ID":"473be601-bbd9-4dcb-89ec-e43b66619766","Type":"ContainerStarted","Data":"67c8426e350846de1fd54ad4b63f2156d324c9a6a10a6d69677671639670881b"}
Nov 02 03:18:10.719313 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:10.719143    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d5dcca72-846bb99569-wldb8" event={"ID":"473be601-bbd9-4dcb-89ec-e43b66619766","Type":"ContainerStarted","Data":"647e0619114b94b09ef98e908c59d13b89c4c0b632cc0b342ea5a626c1227c7c"}
Nov 02 03:18:10.719313 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:10.719212    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-d5dcca72-846bb99569-wldb8"
Nov 02 03:18:10.721173 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:10.721150    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d5dcca72-7746cbb699-h8cx9" event={"ID":"e54dff57-76c5-4a6f-bd9f-b59e6444b307","Type":"ContainerStarted","Data":"3d08984050e3663631649ec4e390035e89aed6fad6574495fe8d5b42d47bf05a"}
Nov 02 03:18:10.721294 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:10.721199    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-d5dcca72-7746cbb699-h8cx9"
Nov 02 03:18:10.724583 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:10.724560    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/789eba8d-594a-4f33-bc35-1d9c93281593-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1606467d-4g8s2\" (UID: \"789eba8d-594a-4f33-bc35-1d9c93281593\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1606467d-4g8s2"
Nov 02 03:18:10.724663 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:10.724591    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/789eba8d-594a-4f33-bc35-1d9c93281593-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1606467d-4g8s2\" (UID: \"789eba8d-594a-4f33-bc35-1d9c93281593\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1606467d-4g8s2"
Nov 02 03:18:10.724663 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:10.724623    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n97n8\" (UniqueName: \"kubernetes.io/projected/789eba8d-594a-4f33-bc35-1d9c93281593-kube-api-access-n97n8\") pod \"maas-api-revoke-sub-1606467d-4g8s2\" (UID: \"789eba8d-594a-4f33-bc35-1d9c93281593\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1606467d-4g8s2"
Nov 02 03:18:10.732197 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:10.732179    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9z5bk\""
Nov 02 03:18:10.825789 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:10.825760    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/789eba8d-594a-4f33-bc35-1d9c93281593-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1606467d-4g8s2\" (UID: \"789eba8d-594a-4f33-bc35-1d9c93281593\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1606467d-4g8s2"
Nov 02 03:18:10.825789 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:10.825793    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/789eba8d-594a-4f33-bc35-1d9c93281593-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1606467d-4g8s2\" (UID: \"789eba8d-594a-4f33-bc35-1d9c93281593\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1606467d-4g8s2"
Nov 02 03:18:10.826160 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:10.825973    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n97n8\" (UniqueName: \"kubernetes.io/projected/789eba8d-594a-4f33-bc35-1d9c93281593-kube-api-access-n97n8\") pod \"maas-api-revoke-sub-1606467d-4g8s2\" (UID: \"789eba8d-594a-4f33-bc35-1d9c93281593\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1606467d-4g8s2"
Nov 02 03:18:10.826527 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:10.826503    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/789eba8d-594a-4f33-bc35-1d9c93281593-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1606467d-4g8s2\" (UID: \"789eba8d-594a-4f33-bc35-1d9c93281593\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1606467d-4g8s2"
Nov 02 03:18:10.828271 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:10.828250    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/789eba8d-594a-4f33-bc35-1d9c93281593-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1606467d-4g8s2\" (UID: \"789eba8d-594a-4f33-bc35-1d9c93281593\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1606467d-4g8s2"
Nov 02 03:18:10.863360 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:10.863305    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-d5dcca72-846bb99569-wldb8" podStartSLOduration=5.863293239 podStartE2EDuration="5.863293239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:18:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:18:10.860873289 +0000 UTC m=+1831.439480669" watchObservedRunningTime="2026-11-02 03:18:10.863293239 +0000 UTC m=+1831.441900616"
Nov 02 03:18:10.884335 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:10.884281    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n97n8\" (UniqueName: \"kubernetes.io/projected/789eba8d-594a-4f33-bc35-1d9c93281593-kube-api-access-n97n8\") pod \"maas-api-revoke-sub-1606467d-4g8s2\" (UID: \"789eba8d-594a-4f33-bc35-1d9c93281593\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1606467d-4g8s2"
Nov 02 03:18:10.946153 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:10.946129    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1606467d-4g8s2"
Nov 02 03:18:10.986585 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:10.986512    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-d5dcca72-7746cbb699-h8cx9" podStartSLOduration=5.986491877 podStartE2EDuration="5.986491877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:18:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:18:10.981439219 +0000 UTC m=+1831.560046600" watchObservedRunningTime="2026-11-02 03:18:10.986491877 +0000 UTC m=+1831.565099260"
Nov 02 03:18:11.337252 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:18:11.337219    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod789eba8d_594a_4f33_bc35_1d9c93281593.slice/crio-462cd6f963af4bbd6bfd3552b8459e529313e4c2fae330d64e07e8168bd28155 WatchSource:0}: Error finding container 462cd6f963af4bbd6bfd3552b8459e529313e4c2fae330d64e07e8168bd28155: Status 404 returned error can't find the container with id 462cd6f963af4bbd6bfd3552b8459e529313e4c2fae330d64e07e8168bd28155
Nov 02 03:18:11.338252 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:11.338178    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1606467d-4g8s2"]
Nov 02 03:18:11.731218 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:11.731181    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1606467d-4g8s2" event={"ID":"789eba8d-594a-4f33-bc35-1d9c93281593","Type":"ContainerStarted","Data":"eb86f9650e2ab1c0cd3176ca5ea0aca8f0d5aa494a72ebb231a616615f33a576"}
Nov 02 03:18:11.731634 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:11.731243    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1606467d-4g8s2" event={"ID":"789eba8d-594a-4f33-bc35-1d9c93281593","Type":"ContainerStarted","Data":"462cd6f963af4bbd6bfd3552b8459e529313e4c2fae330d64e07e8168bd28155"}
Nov 02 03:18:11.857149 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:11.857087    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1606467d-4g8s2" podStartSLOduration=1.857068618 podStartE2EDuration="1.857068618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:18:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:18:11.854911562 +0000 UTC m=+1832.433518943" watchObservedRunningTime="2026-11-02 03:18:11.857068618 +0000 UTC m=+1832.435676001"
Nov 02 03:18:12.740678 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:12.740643    4496 generic.go:356] "Generic (PLEG): container finished" podID="789eba8d-594a-4f33-bc35-1d9c93281593" containerID="eb86f9650e2ab1c0cd3176ca5ea0aca8f0d5aa494a72ebb231a616615f33a576" exitCode=0
Nov 02 03:18:12.741127 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:12.740730    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1606467d-4g8s2" event={"ID":"789eba8d-594a-4f33-bc35-1d9c93281593","Type":"ContainerDied","Data":"eb86f9650e2ab1c0cd3176ca5ea0aca8f0d5aa494a72ebb231a616615f33a576"}
Nov 02 03:18:13.908057 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:13.908031    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1606467d-4g8s2"
Nov 02 03:18:13.966411 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:13.966381    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/789eba8d-594a-4f33-bc35-1d9c93281593-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/789eba8d-594a-4f33-bc35-1d9c93281593-maas-api-xks-ca\") pod \"789eba8d-594a-4f33-bc35-1d9c93281593\" (UID: \"789eba8d-594a-4f33-bc35-1d9c93281593\") "
Nov 02 03:18:13.966560 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:13.966468    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/789eba8d-594a-4f33-bc35-1d9c93281593-kube-api-access-n97n8\" (UniqueName: \"kubernetes.io/projected/789eba8d-594a-4f33-bc35-1d9c93281593-kube-api-access-n97n8\") pod \"789eba8d-594a-4f33-bc35-1d9c93281593\" (UID: \"789eba8d-594a-4f33-bc35-1d9c93281593\") "
Nov 02 03:18:13.966602 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:13.966577    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/789eba8d-594a-4f33-bc35-1d9c93281593-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/789eba8d-594a-4f33-bc35-1d9c93281593-maas-api-openshift-ca\") pod \"789eba8d-594a-4f33-bc35-1d9c93281593\" (UID: \"789eba8d-594a-4f33-bc35-1d9c93281593\") "
Nov 02 03:18:13.966926 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:13.966893    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/789eba8d-594a-4f33-bc35-1d9c93281593-maas-api-openshift-ca" pod "789eba8d-594a-4f33-bc35-1d9c93281593" (UID: "789eba8d-594a-4f33-bc35-1d9c93281593"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:18:13.968521 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:13.968499    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/789eba8d-594a-4f33-bc35-1d9c93281593-maas-api-xks-ca" pod "789eba8d-594a-4f33-bc35-1d9c93281593" (UID: "789eba8d-594a-4f33-bc35-1d9c93281593"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 03:18:13.968670 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:13.968649    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/789eba8d-594a-4f33-bc35-1d9c93281593-kube-api-access-n97n8" pod "789eba8d-594a-4f33-bc35-1d9c93281593" (UID: "789eba8d-594a-4f33-bc35-1d9c93281593"). InnerVolumeSpecName "kube-api-access-n97n8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:18:14.068404 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:14.068307    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/789eba8d-594a-4f33-bc35-1d9c93281593-maas-api-openshift-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:18:14.068404 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:14.068344    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/789eba8d-594a-4f33-bc35-1d9c93281593-maas-api-xks-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:18:14.068404 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:14.068359    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n97n8\" (UniqueName: \"kubernetes.io/projected/789eba8d-594a-4f33-bc35-1d9c93281593-kube-api-access-n97n8\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:18:14.759112 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:14.759083    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1606467d-4g8s2"
Nov 02 03:18:14.759338 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:14.759113    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1606467d-4g8s2" event={"ID":"789eba8d-594a-4f33-bc35-1d9c93281593","Type":"ContainerDied","Data":"462cd6f963af4bbd6bfd3552b8459e529313e4c2fae330d64e07e8168bd28155"}
Nov 02 03:18:14.759338 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:14.759169    4496 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="462cd6f963af4bbd6bfd3552b8459e529313e4c2fae330d64e07e8168bd28155"
Nov 02 03:18:19.048055 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:19.048001    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b3a374f-mclzj"]
Nov 02 03:18:19.098950 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:19.098917    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b3a374f-mclzj"]
Nov 02 03:18:19.576028 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:19.575971    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="901d77d5-60d5-4708-b970-62d5411c2f3f" path="/var/lib/kubelet/pods/901d77d5-60d5-4708-b970-62d5411c2f3f/volumes"
Nov 02 03:18:21.735089 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:21.735057    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-d5dcca72-846bb99569-wldb8"
Nov 02 03:18:21.735526 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:21.735132    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-d5dcca72-7746cbb699-h8cx9"
Nov 02 03:18:26.143753 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:26.143665    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-csm6n"]
Nov 02 03:18:26.144564 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:26.144542    4496 cpu_manager.go:414] "containerMap: removing container" podUID="789eba8d-594a-4f33-bc35-1d9c93281593" containerName="revoke-keys"
Nov 02 03:18:26.144667 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:26.144564    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="789eba8d-594a-4f33-bc35-1d9c93281593" containerName="revoke-keys"
Nov 02 03:18:26.144777 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:26.144764    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="789eba8d-594a-4f33-bc35-1d9c93281593" containerName="revoke-keys"
Nov 02 03:18:26.149944 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:26.149918    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-csm6n"
Nov 02 03:18:26.203494 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:26.203458    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Nov 02 03:18:26.203663 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:26.203535    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Nov 02 03:18:26.203663 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:26.203597    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Nov 02 03:18:26.203663 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:26.203608    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-9w67j\""
Nov 02 03:18:26.220419 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:26.220393    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2cfcd892-9dc1-4853-bb63-3b79e5c70942-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-csm6n\" (UID: \"2cfcd892-9dc1-4853-bb63-3b79e5c70942\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-csm6n"
Nov 02 03:18:26.220565 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:26.220439    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2cfcd892-9dc1-4853-bb63-3b79e5c70942-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-csm6n\" (UID: \"2cfcd892-9dc1-4853-bb63-3b79e5c70942\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-csm6n"
Nov 02 03:18:26.220628 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:26.220606    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cfcd892-9dc1-4853-bb63-3b79e5c70942-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-csm6n\" (UID: \"2cfcd892-9dc1-4853-bb63-3b79e5c70942\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-csm6n"
Nov 02 03:18:26.220675 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:26.220651    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wzxs\" (UniqueName: \"kubernetes.io/projected/2cfcd892-9dc1-4853-bb63-3b79e5c70942-kube-api-access-2wzxs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-csm6n\" (UID: \"2cfcd892-9dc1-4853-bb63-3b79e5c70942\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-csm6n"
Nov 02 03:18:26.220778 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:26.220759    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cfcd892-9dc1-4853-bb63-3b79e5c70942-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-csm6n\" (UID: \"2cfcd892-9dc1-4853-bb63-3b79e5c70942\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-csm6n"
Nov 02 03:18:26.220867 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:26.220852    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2cfcd892-9dc1-4853-bb63-3b79e5c70942-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-csm6n\" (UID: \"2cfcd892-9dc1-4853-bb63-3b79e5c70942\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-csm6n"
Nov 02 03:18:26.220928 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:26.220915    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2cfcd892-9dc1-4853-bb63-3b79e5c70942-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-csm6n\" (UID: \"2cfcd892-9dc1-4853-bb63-3b79e5c70942\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-csm6n"
Nov 02 03:18:26.260666 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:26.260628    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-csm6n"]
Nov 02 03:18:26.322438 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:26.322404    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cfcd892-9dc1-4853-bb63-3b79e5c70942-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-csm6n\" (UID: \"2cfcd892-9dc1-4853-bb63-3b79e5c70942\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-csm6n"
Nov 02 03:18:26.322641 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:26.322463    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2cfcd892-9dc1-4853-bb63-3b79e5c70942-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-csm6n\" (UID: \"2cfcd892-9dc1-4853-bb63-3b79e5c70942\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-csm6n"
Nov 02 03:18:26.322641 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:26.322515    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2cfcd892-9dc1-4853-bb63-3b79e5c70942-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-csm6n\" (UID: \"2cfcd892-9dc1-4853-bb63-3b79e5c70942\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-csm6n"
Nov 02 03:18:26.322641 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:26.322601    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2cfcd892-9dc1-4853-bb63-3b79e5c70942-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-csm6n\" (UID: \"2cfcd892-9dc1-4853-bb63-3b79e5c70942\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-csm6n"
Nov 02 03:18:26.322641 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:26.322636    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2cfcd892-9dc1-4853-bb63-3b79e5c70942-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-csm6n\" (UID: \"2cfcd892-9dc1-4853-bb63-3b79e5c70942\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-csm6n"
Nov 02 03:18:26.322871 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:26.322741    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cfcd892-9dc1-4853-bb63-3b79e5c70942-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-csm6n\" (UID: \"2cfcd892-9dc1-4853-bb63-3b79e5c70942\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-csm6n"
Nov 02 03:18:26.322871 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:26.322778    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wzxs\" (UniqueName: \"kubernetes.io/projected/2cfcd892-9dc1-4853-bb63-3b79e5c70942-kube-api-access-2wzxs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-csm6n\" (UID: \"2cfcd892-9dc1-4853-bb63-3b79e5c70942\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-csm6n"
Nov 02 03:18:26.323360 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:26.323322    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2cfcd892-9dc1-4853-bb63-3b79e5c70942-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-csm6n\" (UID: \"2cfcd892-9dc1-4853-bb63-3b79e5c70942\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-csm6n"
Nov 02 03:18:26.325231 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:26.325202    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cfcd892-9dc1-4853-bb63-3b79e5c70942-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-csm6n\" (UID: \"2cfcd892-9dc1-4853-bb63-3b79e5c70942\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-csm6n"
Nov 02 03:18:26.325331 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:26.325284    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2cfcd892-9dc1-4853-bb63-3b79e5c70942-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-csm6n\" (UID: \"2cfcd892-9dc1-4853-bb63-3b79e5c70942\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-csm6n"
Nov 02 03:18:26.325848 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:26.325829    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2cfcd892-9dc1-4853-bb63-3b79e5c70942-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-csm6n\" (UID: \"2cfcd892-9dc1-4853-bb63-3b79e5c70942\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-csm6n"
Nov 02 03:18:26.326001 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:26.325980    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2cfcd892-9dc1-4853-bb63-3b79e5c70942-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-csm6n\" (UID: \"2cfcd892-9dc1-4853-bb63-3b79e5c70942\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-csm6n"
Nov 02 03:18:26.382774 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:26.382698    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wzxs\" (UniqueName: \"kubernetes.io/projected/2cfcd892-9dc1-4853-bb63-3b79e5c70942-kube-api-access-2wzxs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-csm6n\" (UID: \"2cfcd892-9dc1-4853-bb63-3b79e5c70942\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-csm6n"
Nov 02 03:18:26.383619 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:26.383589    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cfcd892-9dc1-4853-bb63-3b79e5c70942-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-csm6n\" (UID: \"2cfcd892-9dc1-4853-bb63-3b79e5c70942\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-csm6n"
Nov 02 03:18:26.465692 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:26.465600    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-csm6n"
Nov 02 03:18:26.907634 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:26.907592    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-csm6n"]
Nov 02 03:18:26.909375 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:18:26.909339    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cfcd892_9dc1_4853_bb63_3b79e5c70942.slice/crio-116109002c1f187a5e8e0d8720febb8e59faff1c12ce7dce33900cd5f655a7a1 WatchSource:0}: Error finding container 116109002c1f187a5e8e0d8720febb8e59faff1c12ce7dce33900cd5f655a7a1: Status 404 returned error can't find the container with id 116109002c1f187a5e8e0d8720febb8e59faff1c12ce7dce33900cd5f655a7a1
Nov 02 03:18:26.912184 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:26.912158    4496 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 03:18:27.889913 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:27.889128    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-csm6n" event={"ID":"2cfcd892-9dc1-4853-bb63-3b79e5c70942","Type":"ContainerStarted","Data":"116109002c1f187a5e8e0d8720febb8e59faff1c12ce7dce33900cd5f655a7a1"}
Nov 02 03:18:31.984421 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:31.984379    4496 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 03:18:31.984856 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:31.984459    4496 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 03:18:32.964142 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:32.964108    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-csm6n" event={"ID":"2cfcd892-9dc1-4853-bb63-3b79e5c70942","Type":"ContainerStarted","Data":"5c38de8389b0b433309dde41f9ecd53dae948d17ffaf26c51ec089d1c9158986"}
Nov 02 03:18:32.964307 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:32.964255    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-csm6n"
Nov 02 03:18:33.187349 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:33.187288    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-csm6n" podStartSLOduration=2.116901093 podStartE2EDuration="7.187273534s" totalImagesPullingTime="5.070372441s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:18:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:18:33.133867758 +0000 UTC m=+1853.712475139" watchObservedRunningTime="2026-11-02 03:18:33.187273534 +0000 UTC m=+1853.765880914"
Nov 02 03:18:33.556863 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:18:33.553946    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 03:18:33.975657 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:33.975622    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-csm6n"
Nov 02 03:18:40.136915 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:40.136879    4496 scope.go:122] "RemoveContainer" containerID="15cfb6c0f9c57eb6b52062aae2fd3f738e23cee30f38cc83d9eb6ae55524bafb"
Nov 02 03:18:40.147554 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:40.147533    4496 scope.go:122] "RemoveContainer" containerID="0b99df6099e2b1f6703db46b3c34de2ab0b75c354bded6ecfcb1fee97818b1f0"
Nov 02 03:18:41.050766 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:41.050738    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89ceec26-hwnls"]
Nov 02 03:18:41.101449 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:41.101414    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89ceec26-hwnls"]
Nov 02 03:18:41.263529 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:41.263489    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-kj8qb"]
Nov 02 03:18:41.269467 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:41.269439    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-kj8qb"
Nov 02 03:18:41.326284 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:41.326224    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-kj8qb"]
Nov 02 03:18:41.376265 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:41.376225    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a41eaed2-344b-405c-86cb-631055ace710-workload-socket\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-kj8qb\" (UID: \"a41eaed2-344b-405c-86cb-631055ace710\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-kj8qb"
Nov 02 03:18:41.376429 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:41.376270    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a41eaed2-344b-405c-86cb-631055ace710-istio-data\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-kj8qb\" (UID: \"a41eaed2-344b-405c-86cb-631055ace710\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-kj8qb"
Nov 02 03:18:41.376429 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:41.376411    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a41eaed2-344b-405c-86cb-631055ace710-credential-socket\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-kj8qb\" (UID: \"a41eaed2-344b-405c-86cb-631055ace710\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-kj8qb"
Nov 02 03:18:41.376555 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:41.376533    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a41eaed2-344b-405c-86cb-631055ace710-istio-envoy\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-kj8qb\" (UID: \"a41eaed2-344b-405c-86cb-631055ace710\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-kj8qb"
Nov 02 03:18:41.376615 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:41.376562    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a41eaed2-344b-405c-86cb-631055ace710-istiod-ca-cert\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-kj8qb\" (UID: \"a41eaed2-344b-405c-86cb-631055ace710\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-kj8qb"
Nov 02 03:18:41.376615 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:41.376601    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a41eaed2-344b-405c-86cb-631055ace710-istio-token\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-kj8qb\" (UID: \"a41eaed2-344b-405c-86cb-631055ace710\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-kj8qb"
Nov 02 03:18:41.376788 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:41.376758    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfhgf\" (UniqueName: \"kubernetes.io/projected/a41eaed2-344b-405c-86cb-631055ace710-kube-api-access-cfhgf\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-kj8qb\" (UID: \"a41eaed2-344b-405c-86cb-631055ace710\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-kj8qb"
Nov 02 03:18:41.376830 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:41.376807    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a41eaed2-344b-405c-86cb-631055ace710-workload-certs\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-kj8qb\" (UID: \"a41eaed2-344b-405c-86cb-631055ace710\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-kj8qb"
Nov 02 03:18:41.376894 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:41.376879    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a41eaed2-344b-405c-86cb-631055ace710-istio-podinfo\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-kj8qb\" (UID: \"a41eaed2-344b-405c-86cb-631055ace710\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-kj8qb"
Nov 02 03:18:41.478065 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:41.478008    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a41eaed2-344b-405c-86cb-631055ace710-credential-socket\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-kj8qb\" (UID: \"a41eaed2-344b-405c-86cb-631055ace710\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-kj8qb"
Nov 02 03:18:41.478294 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:41.478138    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a41eaed2-344b-405c-86cb-631055ace710-istio-envoy\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-kj8qb\" (UID: \"a41eaed2-344b-405c-86cb-631055ace710\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-kj8qb"
Nov 02 03:18:41.478294 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:41.478165    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a41eaed2-344b-405c-86cb-631055ace710-istiod-ca-cert\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-kj8qb\" (UID: \"a41eaed2-344b-405c-86cb-631055ace710\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-kj8qb"
Nov 02 03:18:41.478443 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:41.478299    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a41eaed2-344b-405c-86cb-631055ace710-istio-token\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-kj8qb\" (UID: \"a41eaed2-344b-405c-86cb-631055ace710\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-kj8qb"
Nov 02 03:18:41.478512 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:41.478448    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfhgf\" (UniqueName: \"kubernetes.io/projected/a41eaed2-344b-405c-86cb-631055ace710-kube-api-access-cfhgf\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-kj8qb\" (UID: \"a41eaed2-344b-405c-86cb-631055ace710\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-kj8qb"
Nov 02 03:18:41.478568 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:41.478504    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a41eaed2-344b-405c-86cb-631055ace710-credential-socket\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-kj8qb\" (UID: \"a41eaed2-344b-405c-86cb-631055ace710\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-kj8qb"
Nov 02 03:18:41.478624 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:41.478480    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a41eaed2-344b-405c-86cb-631055ace710-workload-certs\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-kj8qb\" (UID: \"a41eaed2-344b-405c-86cb-631055ace710\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-kj8qb"
Nov 02 03:18:41.478752 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:41.478731    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a41eaed2-344b-405c-86cb-631055ace710-istio-podinfo\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-kj8qb\" (UID: \"a41eaed2-344b-405c-86cb-631055ace710\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-kj8qb"
Nov 02 03:18:41.478887 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:41.478858    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a41eaed2-344b-405c-86cb-631055ace710-istiod-ca-cert\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-kj8qb\" (UID: \"a41eaed2-344b-405c-86cb-631055ace710\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-kj8qb"
Nov 02 03:18:41.478976 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:41.478942    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a41eaed2-344b-405c-86cb-631055ace710-workload-socket\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-kj8qb\" (UID: \"a41eaed2-344b-405c-86cb-631055ace710\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-kj8qb"
Nov 02 03:18:41.478976 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:41.478969    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a41eaed2-344b-405c-86cb-631055ace710-istio-data\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-kj8qb\" (UID: \"a41eaed2-344b-405c-86cb-631055ace710\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-kj8qb"
Nov 02 03:18:41.479442 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:41.479388    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a41eaed2-344b-405c-86cb-631055ace710-workload-socket\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-kj8qb\" (UID: \"a41eaed2-344b-405c-86cb-631055ace710\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-kj8qb"
Nov 02 03:18:41.479442 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:41.479404    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a41eaed2-344b-405c-86cb-631055ace710-istio-data\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-kj8qb\" (UID: \"a41eaed2-344b-405c-86cb-631055ace710\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-kj8qb"
Nov 02 03:18:41.479679 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:41.479534    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a41eaed2-344b-405c-86cb-631055ace710-workload-certs\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-kj8qb\" (UID: \"a41eaed2-344b-405c-86cb-631055ace710\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-kj8qb"
Nov 02 03:18:41.481093 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:41.481064    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a41eaed2-344b-405c-86cb-631055ace710-istio-envoy\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-kj8qb\" (UID: \"a41eaed2-344b-405c-86cb-631055ace710\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-kj8qb"
Nov 02 03:18:41.481345 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:41.481326    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a41eaed2-344b-405c-86cb-631055ace710-istio-podinfo\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-kj8qb\" (UID: \"a41eaed2-344b-405c-86cb-631055ace710\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-kj8qb"
Nov 02 03:18:41.536299 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:41.536269    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a41eaed2-344b-405c-86cb-631055ace710-istio-token\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-kj8qb\" (UID: \"a41eaed2-344b-405c-86cb-631055ace710\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-kj8qb"
Nov 02 03:18:41.536489 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:41.536465    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfhgf\" (UniqueName: \"kubernetes.io/projected/a41eaed2-344b-405c-86cb-631055ace710-kube-api-access-cfhgf\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-kj8qb\" (UID: \"a41eaed2-344b-405c-86cb-631055ace710\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-kj8qb"
Nov 02 03:18:41.574094 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:41.574066    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97d3e35e-8ce3-4a2e-8f1e-9370367b4a1a" path="/var/lib/kubelet/pods/97d3e35e-8ce3-4a2e-8f1e-9370367b4a1a/volumes"
Nov 02 03:18:41.583356 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:41.583318    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-kj8qb"
Nov 02 03:18:42.013940 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:18:42.013902    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda41eaed2_344b_405c_86cb_631055ace710.slice/crio-82339e375f1e036d70b6bc66adf53c4818cc2f14b3a432840647fd65c068d4fc WatchSource:0}: Error finding container 82339e375f1e036d70b6bc66adf53c4818cc2f14b3a432840647fd65c068d4fc: Status 404 returned error can't find the container with id 82339e375f1e036d70b6bc66adf53c4818cc2f14b3a432840647fd65c068d4fc
Nov 02 03:18:42.014678 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:42.014653    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-kj8qb"]
Nov 02 03:18:42.016413 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:42.016383    4496 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 03:18:42.016493 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:42.016447    4496 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 03:18:42.016493 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:42.016473    4496 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 03:18:42.049515 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:42.049491    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-kj8qb" event={"ID":"a41eaed2-344b-405c-86cb-631055ace710","Type":"ContainerStarted","Data":"82339e375f1e036d70b6bc66adf53c4818cc2f14b3a432840647fd65c068d4fc"}
Nov 02 03:18:43.067272 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:43.067233    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-kj8qb" event={"ID":"a41eaed2-344b-405c-86cb-631055ace710","Type":"ContainerStarted","Data":"e8f062428768b2fd4e15c6b5823e3b230b07f39789d001da76df900c686c6d21"}
Nov 02 03:18:43.259605 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:43.259486    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-kj8qb" podStartSLOduration=2.259462876 podStartE2EDuration="2.259462876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:18:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:18:43.255380331 +0000 UTC m=+1863.833987712" watchObservedRunningTime="2026-11-02 03:18:43.259462876 +0000 UTC m=+1863.838070260"
Nov 02 03:18:43.584424 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:43.584382    4496 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-kj8qb"
Nov 02 03:18:43.586026 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:43.585993    4496 patch_prober.go:29] interesting pod/e2e-worker-w4-398828-openshift-default-5647988965-kj8qb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body=
Nov 02 03:18:43.586144 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:43.586090    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-kj8qb" podUID="a41eaed2-344b-405c-86cb-631055ace710" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused"
Nov 02 03:18:44.584596 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:44.584560    4496 patch_prober.go:29] interesting pod/e2e-worker-w4-398828-openshift-default-5647988965-kj8qb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body=
Nov 02 03:18:44.585000 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:44.584626    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-kj8qb" podUID="a41eaed2-344b-405c-86cb-631055ace710" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused"
Nov 02 03:18:45.583644 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:45.583608    4496 patch_prober.go:29] interesting pod/e2e-worker-w4-398828-openshift-default-5647988965-kj8qb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body=
Nov 02 03:18:45.583850 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:45.583657    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-kj8qb" podUID="a41eaed2-344b-405c-86cb-631055ace710" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused"
Nov 02 03:18:46.584075 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:46.584005    4496 patch_prober.go:29] interesting pod/e2e-worker-w4-398828-openshift-default-5647988965-kj8qb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body=
Nov 02 03:18:46.584472 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:46.584087    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-kj8qb" podUID="a41eaed2-344b-405c-86cb-631055ace710" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused"
Nov 02 03:18:47.584125 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:47.584090    4496 patch_prober.go:29] interesting pod/e2e-worker-w4-398828-openshift-default-5647988965-kj8qb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body=
Nov 02 03:18:47.584624 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:47.584148    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-kj8qb" podUID="a41eaed2-344b-405c-86cb-631055ace710" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused"
Nov 02 03:18:48.584605 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:48.584567    4496 patch_prober.go:29] interesting pod/e2e-worker-w4-398828-openshift-default-5647988965-kj8qb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body=
Nov 02 03:18:48.585126 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:48.584622    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-kj8qb" podUID="a41eaed2-344b-405c-86cb-631055ace710" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused"
Nov 02 03:18:49.587784 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:49.587751    4496 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-kj8qb"
Nov 02 03:18:49.588276 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:49.588224    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-kj8qb"
Nov 02 03:18:49.589006 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:18:49.588989    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-kj8qb"
Nov 02 03:19:03.528782 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:19:03.528745    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 03:19:05.589772 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:05.589737    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-5fp6v"]
Nov 02 03:19:05.594165 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:05.594142    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-5fp6v"
Nov 02 03:19:05.647317 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:05.647286    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-858236-openshift-default-dockercfg-79wxp\""
Nov 02 03:19:05.670957 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:05.670925    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b2e2446-f387-4ea4-8787-1fa5f47e18e6-istio-envoy\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-5fp6v\" (UID: \"6b2e2446-f387-4ea4-8787-1fa5f47e18e6\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-5fp6v"
Nov 02 03:19:05.671110 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:05.670990    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b2e2446-f387-4ea4-8787-1fa5f47e18e6-istio-data\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-5fp6v\" (UID: \"6b2e2446-f387-4ea4-8787-1fa5f47e18e6\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-5fp6v"
Nov 02 03:19:05.671157 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:05.671139    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b2e2446-f387-4ea4-8787-1fa5f47e18e6-workload-socket\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-5fp6v\" (UID: \"6b2e2446-f387-4ea4-8787-1fa5f47e18e6\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-5fp6v"
Nov 02 03:19:05.671193 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:05.671171    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ggms\" (UniqueName: \"kubernetes.io/projected/6b2e2446-f387-4ea4-8787-1fa5f47e18e6-kube-api-access-9ggms\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-5fp6v\" (UID: \"6b2e2446-f387-4ea4-8787-1fa5f47e18e6\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-5fp6v"
Nov 02 03:19:05.671289 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:05.671272    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b2e2446-f387-4ea4-8787-1fa5f47e18e6-istio-podinfo\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-5fp6v\" (UID: \"6b2e2446-f387-4ea4-8787-1fa5f47e18e6\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-5fp6v"
Nov 02 03:19:05.671418 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:05.671402    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b2e2446-f387-4ea4-8787-1fa5f47e18e6-istiod-ca-cert\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-5fp6v\" (UID: \"6b2e2446-f387-4ea4-8787-1fa5f47e18e6\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-5fp6v"
Nov 02 03:19:05.671512 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:05.671497    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b2e2446-f387-4ea4-8787-1fa5f47e18e6-credential-socket\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-5fp6v\" (UID: \"6b2e2446-f387-4ea4-8787-1fa5f47e18e6\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-5fp6v"
Nov 02 03:19:05.671587 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:05.671565    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b2e2446-f387-4ea4-8787-1fa5f47e18e6-workload-certs\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-5fp6v\" (UID: \"6b2e2446-f387-4ea4-8787-1fa5f47e18e6\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-5fp6v"
Nov 02 03:19:05.671686 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:05.671665    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b2e2446-f387-4ea4-8787-1fa5f47e18e6-istio-token\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-5fp6v\" (UID: \"6b2e2446-f387-4ea4-8787-1fa5f47e18e6\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-5fp6v"
Nov 02 03:19:05.706277 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:05.706241    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-5fp6v"]
Nov 02 03:19:05.772483 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:05.772442    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b2e2446-f387-4ea4-8787-1fa5f47e18e6-credential-socket\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-5fp6v\" (UID: \"6b2e2446-f387-4ea4-8787-1fa5f47e18e6\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-5fp6v"
Nov 02 03:19:05.772483 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:05.772480    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b2e2446-f387-4ea4-8787-1fa5f47e18e6-workload-certs\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-5fp6v\" (UID: \"6b2e2446-f387-4ea4-8787-1fa5f47e18e6\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-5fp6v"
Nov 02 03:19:05.772725 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:05.772507    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b2e2446-f387-4ea4-8787-1fa5f47e18e6-istio-token\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-5fp6v\" (UID: \"6b2e2446-f387-4ea4-8787-1fa5f47e18e6\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-5fp6v"
Nov 02 03:19:05.772725 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:05.772572    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b2e2446-f387-4ea4-8787-1fa5f47e18e6-istio-envoy\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-5fp6v\" (UID: \"6b2e2446-f387-4ea4-8787-1fa5f47e18e6\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-5fp6v"
Nov 02 03:19:05.772725 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:05.772626    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b2e2446-f387-4ea4-8787-1fa5f47e18e6-istio-data\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-5fp6v\" (UID: \"6b2e2446-f387-4ea4-8787-1fa5f47e18e6\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-5fp6v"
Nov 02 03:19:05.772725 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:05.772681    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b2e2446-f387-4ea4-8787-1fa5f47e18e6-workload-socket\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-5fp6v\" (UID: \"6b2e2446-f387-4ea4-8787-1fa5f47e18e6\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-5fp6v"
Nov 02 03:19:05.772725 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:05.772706    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ggms\" (UniqueName: \"kubernetes.io/projected/6b2e2446-f387-4ea4-8787-1fa5f47e18e6-kube-api-access-9ggms\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-5fp6v\" (UID: \"6b2e2446-f387-4ea4-8787-1fa5f47e18e6\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-5fp6v"
Nov 02 03:19:05.772987 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:05.772761    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b2e2446-f387-4ea4-8787-1fa5f47e18e6-istio-podinfo\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-5fp6v\" (UID: \"6b2e2446-f387-4ea4-8787-1fa5f47e18e6\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-5fp6v"
Nov 02 03:19:05.772987 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:05.772875    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b2e2446-f387-4ea4-8787-1fa5f47e18e6-workload-certs\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-5fp6v\" (UID: \"6b2e2446-f387-4ea4-8787-1fa5f47e18e6\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-5fp6v"
Nov 02 03:19:05.772987 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:05.772887    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b2e2446-f387-4ea4-8787-1fa5f47e18e6-istiod-ca-cert\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-5fp6v\" (UID: \"6b2e2446-f387-4ea4-8787-1fa5f47e18e6\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-5fp6v"
Nov 02 03:19:05.772987 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:05.772920    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b2e2446-f387-4ea4-8787-1fa5f47e18e6-credential-socket\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-5fp6v\" (UID: \"6b2e2446-f387-4ea4-8787-1fa5f47e18e6\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-5fp6v"
Nov 02 03:19:05.773236 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:05.773041    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b2e2446-f387-4ea4-8787-1fa5f47e18e6-istio-data\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-5fp6v\" (UID: \"6b2e2446-f387-4ea4-8787-1fa5f47e18e6\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-5fp6v"
Nov 02 03:19:05.773236 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:05.773134    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b2e2446-f387-4ea4-8787-1fa5f47e18e6-workload-socket\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-5fp6v\" (UID: \"6b2e2446-f387-4ea4-8787-1fa5f47e18e6\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-5fp6v"
Nov 02 03:19:05.773511 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:05.773491    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b2e2446-f387-4ea4-8787-1fa5f47e18e6-istiod-ca-cert\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-5fp6v\" (UID: \"6b2e2446-f387-4ea4-8787-1fa5f47e18e6\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-5fp6v"
Nov 02 03:19:05.775123 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:05.775100    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b2e2446-f387-4ea4-8787-1fa5f47e18e6-istio-envoy\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-5fp6v\" (UID: \"6b2e2446-f387-4ea4-8787-1fa5f47e18e6\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-5fp6v"
Nov 02 03:19:05.775397 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:05.775377    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b2e2446-f387-4ea4-8787-1fa5f47e18e6-istio-podinfo\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-5fp6v\" (UID: \"6b2e2446-f387-4ea4-8787-1fa5f47e18e6\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-5fp6v"
Nov 02 03:19:05.830540 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:05.830516    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b2e2446-f387-4ea4-8787-1fa5f47e18e6-istio-token\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-5fp6v\" (UID: \"6b2e2446-f387-4ea4-8787-1fa5f47e18e6\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-5fp6v"
Nov 02 03:19:05.830829 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:05.830806    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ggms\" (UniqueName: \"kubernetes.io/projected/6b2e2446-f387-4ea4-8787-1fa5f47e18e6-kube-api-access-9ggms\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-5fp6v\" (UID: \"6b2e2446-f387-4ea4-8787-1fa5f47e18e6\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-5fp6v"
Nov 02 03:19:05.909485 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:05.909382    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-5fp6v"
Nov 02 03:19:06.342669 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:19:06.342640    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b2e2446_f387_4ea4_8787_1fa5f47e18e6.slice/crio-d04c141255b9dbdeba41491fa0251e0081f9db3c4a9ad11d46fab49bcbb711b3 WatchSource:0}: Error finding container d04c141255b9dbdeba41491fa0251e0081f9db3c4a9ad11d46fab49bcbb711b3: Status 404 returned error can't find the container with id d04c141255b9dbdeba41491fa0251e0081f9db3c4a9ad11d46fab49bcbb711b3
Nov 02 03:19:06.344137 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:06.344106    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-5fp6v"]
Nov 02 03:19:06.345086 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:06.345057    4496 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 03:19:06.345158 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:06.345123    4496 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 03:19:06.345158 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:06.345154    4496 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 03:19:07.370085 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:07.369998    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-5fp6v" event={"ID":"6b2e2446-f387-4ea4-8787-1fa5f47e18e6","Type":"ContainerStarted","Data":"61a831abd73b508ee30c4060eacb78400bc5818568d8c7e9517c12efadf55d88"}
Nov 02 03:19:07.370520 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:07.370105    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-5fp6v" event={"ID":"6b2e2446-f387-4ea4-8787-1fa5f47e18e6","Type":"ContainerStarted","Data":"d04c141255b9dbdeba41491fa0251e0081f9db3c4a9ad11d46fab49bcbb711b3"}
Nov 02 03:19:07.553264 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:07.553190    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-5fp6v" podStartSLOduration=2.553166134 podStartE2EDuration="2.553166134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:19:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:19:07.53856028 +0000 UTC m=+1888.117167661" watchObservedRunningTime="2026-11-02 03:19:07.553166134 +0000 UTC m=+1888.131773515"
Nov 02 03:19:07.910204 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:07.910168    4496 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-5fp6v"
Nov 02 03:19:07.911697 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:07.911669    4496 patch_prober.go:29] interesting pod/e2e-worker-w3-858236-openshift-default-74fcd64bf5-5fp6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body=
Nov 02 03:19:07.911816 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:07.911724    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-5fp6v" podUID="6b2e2446-f387-4ea4-8787-1fa5f47e18e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused"
Nov 02 03:19:08.910721 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:08.910683    4496 patch_prober.go:29] interesting pod/e2e-worker-w3-858236-openshift-default-74fcd64bf5-5fp6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body=
Nov 02 03:19:08.911331 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:08.911297    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-5fp6v" podUID="6b2e2446-f387-4ea4-8787-1fa5f47e18e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused"
Nov 02 03:19:09.910372 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:09.910333    4496 patch_prober.go:29] interesting pod/e2e-worker-w3-858236-openshift-default-74fcd64bf5-5fp6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body=
Nov 02 03:19:09.910556 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:09.910397    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-5fp6v" podUID="6b2e2446-f387-4ea4-8787-1fa5f47e18e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused"
Nov 02 03:19:10.910703 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:10.910668    4496 patch_prober.go:29] interesting pod/e2e-worker-w3-858236-openshift-default-74fcd64bf5-5fp6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body=
Nov 02 03:19:10.911112 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:10.910722    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-5fp6v" podUID="6b2e2446-f387-4ea4-8787-1fa5f47e18e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused"
Nov 02 03:19:11.910047 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:11.909979    4496 patch_prober.go:29] interesting pod/e2e-worker-w3-858236-openshift-default-74fcd64bf5-5fp6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body=
Nov 02 03:19:11.910263 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:11.910089    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-5fp6v" podUID="6b2e2446-f387-4ea4-8787-1fa5f47e18e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused"
Nov 02 03:19:12.910523 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:12.910481    4496 patch_prober.go:29] interesting pod/e2e-worker-w3-858236-openshift-default-74fcd64bf5-5fp6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body=
Nov 02 03:19:12.910889 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:12.910553    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-5fp6v" podUID="6b2e2446-f387-4ea4-8787-1fa5f47e18e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused"
Nov 02 03:19:13.918703 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:13.918668    4496 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-5fp6v"
Nov 02 03:19:13.921453 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:13.921319    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-5fp6v"
Nov 02 03:19:13.922876 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:13.922854    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-5fp6v"
Nov 02 03:19:23.574190 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:23.574159    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5165bfa2-s22rx"]
Nov 02 03:19:23.579272 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:23.579252    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5165bfa2-s22rx"
Nov 02 03:19:23.614789 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:23.614762    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69be064c-f48a-40b8-8f62-011b916949ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5165bfa2-s22rx\" (UID: \"69be064c-f48a-40b8-8f62-011b916949ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5165bfa2-s22rx"
Nov 02 03:19:23.614910 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:23.614813    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lkqj\" (UniqueName: \"kubernetes.io/projected/69be064c-f48a-40b8-8f62-011b916949ee-kube-api-access-6lkqj\") pod \"maas-api-revoke-sub-5165bfa2-s22rx\" (UID: \"69be064c-f48a-40b8-8f62-011b916949ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5165bfa2-s22rx"
Nov 02 03:19:23.614973 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:23.614919    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69be064c-f48a-40b8-8f62-011b916949ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5165bfa2-s22rx\" (UID: \"69be064c-f48a-40b8-8f62-011b916949ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5165bfa2-s22rx"
Nov 02 03:19:23.629305 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:23.629280    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5165bfa2-s22rx"]
Nov 02 03:19:23.674297 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:23.674251    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 03:19:23.674297 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:23.674273    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 03:19:23.674297 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:23.674272    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 03:19:23.682702 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:23.682678    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9z5bk\""
Nov 02 03:19:23.716263 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:23.716240    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69be064c-f48a-40b8-8f62-011b916949ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5165bfa2-s22rx\" (UID: \"69be064c-f48a-40b8-8f62-011b916949ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5165bfa2-s22rx"
Nov 02 03:19:23.716389 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:23.716286    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lkqj\" (UniqueName: \"kubernetes.io/projected/69be064c-f48a-40b8-8f62-011b916949ee-kube-api-access-6lkqj\") pod \"maas-api-revoke-sub-5165bfa2-s22rx\" (UID: \"69be064c-f48a-40b8-8f62-011b916949ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5165bfa2-s22rx"
Nov 02 03:19:23.716389 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:23.716339    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69be064c-f48a-40b8-8f62-011b916949ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5165bfa2-s22rx\" (UID: \"69be064c-f48a-40b8-8f62-011b916949ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5165bfa2-s22rx"
Nov 02 03:19:23.716988 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:23.716965    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69be064c-f48a-40b8-8f62-011b916949ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5165bfa2-s22rx\" (UID: \"69be064c-f48a-40b8-8f62-011b916949ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5165bfa2-s22rx"
Nov 02 03:19:23.718622 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:23.718595    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69be064c-f48a-40b8-8f62-011b916949ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5165bfa2-s22rx\" (UID: \"69be064c-f48a-40b8-8f62-011b916949ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5165bfa2-s22rx"
Nov 02 03:19:23.774703 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:23.774669    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lkqj\" (UniqueName: \"kubernetes.io/projected/69be064c-f48a-40b8-8f62-011b916949ee-kube-api-access-6lkqj\") pod \"maas-api-revoke-sub-5165bfa2-s22rx\" (UID: \"69be064c-f48a-40b8-8f62-011b916949ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5165bfa2-s22rx"
Nov 02 03:19:23.894158 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:23.894070    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5165bfa2-s22rx"
Nov 02 03:19:24.297930 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:19:24.297894    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69be064c_f48a_40b8_8f62_011b916949ee.slice/crio-340981c25d086ddaf635381a5fdfa825c4edd45cb20c9c46303ca7c2b1be46ad WatchSource:0}: Error finding container 340981c25d086ddaf635381a5fdfa825c4edd45cb20c9c46303ca7c2b1be46ad: Status 404 returned error can't find the container with id 340981c25d086ddaf635381a5fdfa825c4edd45cb20c9c46303ca7c2b1be46ad
Nov 02 03:19:24.299720 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:24.299701    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5165bfa2-s22rx"]
Nov 02 03:19:24.574931 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:24.574897    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5165bfa2-s22rx" event={"ID":"69be064c-f48a-40b8-8f62-011b916949ee","Type":"ContainerStarted","Data":"96c1606d177840247f80a4817ac84aa3611ff96f3ef679fca688a4cd87eedeb6"}
Nov 02 03:19:24.575293 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:24.574964    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5165bfa2-s22rx" event={"ID":"69be064c-f48a-40b8-8f62-011b916949ee","Type":"ContainerStarted","Data":"340981c25d086ddaf635381a5fdfa825c4edd45cb20c9c46303ca7c2b1be46ad"}
Nov 02 03:19:24.690464 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:24.690412    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5165bfa2-s22rx" podStartSLOduration=1.690398564 podStartE2EDuration="1.690398564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:19:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:19:24.689617486 +0000 UTC m=+1905.268224928" watchObservedRunningTime="2026-11-02 03:19:24.690398564 +0000 UTC m=+1905.269005943"
Nov 02 03:19:25.596139 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:25.596097    4496 generic.go:356] "Generic (PLEG): container finished" podID="69be064c-f48a-40b8-8f62-011b916949ee" containerID="96c1606d177840247f80a4817ac84aa3611ff96f3ef679fca688a4cd87eedeb6" exitCode=0
Nov 02 03:19:25.596575 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:25.596170    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5165bfa2-s22rx" event={"ID":"69be064c-f48a-40b8-8f62-011b916949ee","Type":"ContainerDied","Data":"96c1606d177840247f80a4817ac84aa3611ff96f3ef679fca688a4cd87eedeb6"}
Nov 02 03:19:26.791647 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:26.791623    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5165bfa2-s22rx"
Nov 02 03:19:26.859563 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:26.859489    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69be064c-f48a-40b8-8f62-011b916949ee-kube-api-access-6lkqj\" (UniqueName: \"kubernetes.io/projected/69be064c-f48a-40b8-8f62-011b916949ee-kube-api-access-6lkqj\") pod \"69be064c-f48a-40b8-8f62-011b916949ee\" (UID: \"69be064c-f48a-40b8-8f62-011b916949ee\") "
Nov 02 03:19:26.859714 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:26.859658    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69be064c-f48a-40b8-8f62-011b916949ee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69be064c-f48a-40b8-8f62-011b916949ee-maas-api-xks-ca\") pod \"69be064c-f48a-40b8-8f62-011b916949ee\" (UID: \"69be064c-f48a-40b8-8f62-011b916949ee\") "
Nov 02 03:19:26.859780 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:26.859752    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69be064c-f48a-40b8-8f62-011b916949ee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69be064c-f48a-40b8-8f62-011b916949ee-maas-api-openshift-ca\") pod \"69be064c-f48a-40b8-8f62-011b916949ee\" (UID: \"69be064c-f48a-40b8-8f62-011b916949ee\") "
Nov 02 03:19:26.860167 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:26.860140    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69be064c-f48a-40b8-8f62-011b916949ee-maas-api-openshift-ca" pod "69be064c-f48a-40b8-8f62-011b916949ee" (UID: "69be064c-f48a-40b8-8f62-011b916949ee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:19:26.860356 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:26.860334    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69be064c-f48a-40b8-8f62-011b916949ee-maas-api-openshift-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:19:26.861814 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:26.861790    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69be064c-f48a-40b8-8f62-011b916949ee-kube-api-access-6lkqj" pod "69be064c-f48a-40b8-8f62-011b916949ee" (UID: "69be064c-f48a-40b8-8f62-011b916949ee"). InnerVolumeSpecName "kube-api-access-6lkqj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:19:26.862189 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:26.862170    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69be064c-f48a-40b8-8f62-011b916949ee-maas-api-xks-ca" pod "69be064c-f48a-40b8-8f62-011b916949ee" (UID: "69be064c-f48a-40b8-8f62-011b916949ee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 03:19:26.961401 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:26.961368    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lkqj\" (UniqueName: \"kubernetes.io/projected/69be064c-f48a-40b8-8f62-011b916949ee-kube-api-access-6lkqj\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:19:26.961401 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:26.961404    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69be064c-f48a-40b8-8f62-011b916949ee-maas-api-xks-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:19:27.615376 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:27.615337    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5165bfa2-s22rx" event={"ID":"69be064c-f48a-40b8-8f62-011b916949ee","Type":"ContainerDied","Data":"340981c25d086ddaf635381a5fdfa825c4edd45cb20c9c46303ca7c2b1be46ad"}
Nov 02 03:19:27.615532 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:27.615397    4496 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="340981c25d086ddaf635381a5fdfa825c4edd45cb20c9c46303ca7c2b1be46ad"
Nov 02 03:19:27.615532 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:27.615348    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5165bfa2-s22rx"
Nov 02 03:19:28.849827 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:28.849790    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f9a40ef-qrxrd"]
Nov 02 03:19:28.850807 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:28.850787    4496 cpu_manager.go:414] "containerMap: removing container" podUID="69be064c-f48a-40b8-8f62-011b916949ee" containerName="revoke-keys"
Nov 02 03:19:28.850857 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:28.850812    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69be064c-f48a-40b8-8f62-011b916949ee" containerName="revoke-keys"
Nov 02 03:19:28.851054 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:28.851039    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="69be064c-f48a-40b8-8f62-011b916949ee" containerName="revoke-keys"
Nov 02 03:19:28.855578 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:28.855558    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f9a40ef-qrxrd"
Nov 02 03:19:28.907931 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:28.907900    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f9a40ef-qrxrd"]
Nov 02 03:19:28.951958 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:28.951925    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 03:19:28.951958 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:28.951954    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 03:19:28.952165 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:28.951933    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 03:19:28.959338 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:28.959319    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9z5bk\""
Nov 02 03:19:28.988629 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:28.988606    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7x5pw\" (UniqueName: \"kubernetes.io/projected/08f82f8d-8213-423d-8df8-08ae031d0f7c-kube-api-access-7x5pw\") pod \"maas-api-revoke-sub-8f9a40ef-qrxrd\" (UID: \"08f82f8d-8213-423d-8df8-08ae031d0f7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f9a40ef-qrxrd"
Nov 02 03:19:28.988747 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:28.988651    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08f82f8d-8213-423d-8df8-08ae031d0f7c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f9a40ef-qrxrd\" (UID: \"08f82f8d-8213-423d-8df8-08ae031d0f7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f9a40ef-qrxrd"
Nov 02 03:19:28.988801 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:28.988786    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08f82f8d-8213-423d-8df8-08ae031d0f7c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f9a40ef-qrxrd\" (UID: \"08f82f8d-8213-423d-8df8-08ae031d0f7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f9a40ef-qrxrd"
Nov 02 03:19:29.090434 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:29.090394    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7x5pw\" (UniqueName: \"kubernetes.io/projected/08f82f8d-8213-423d-8df8-08ae031d0f7c-kube-api-access-7x5pw\") pod \"maas-api-revoke-sub-8f9a40ef-qrxrd\" (UID: \"08f82f8d-8213-423d-8df8-08ae031d0f7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f9a40ef-qrxrd"
Nov 02 03:19:29.090627 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:29.090482    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08f82f8d-8213-423d-8df8-08ae031d0f7c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f9a40ef-qrxrd\" (UID: \"08f82f8d-8213-423d-8df8-08ae031d0f7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f9a40ef-qrxrd"
Nov 02 03:19:29.090627 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:29.090601    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08f82f8d-8213-423d-8df8-08ae031d0f7c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f9a40ef-qrxrd\" (UID: \"08f82f8d-8213-423d-8df8-08ae031d0f7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f9a40ef-qrxrd"
Nov 02 03:19:29.091305 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:29.091278    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08f82f8d-8213-423d-8df8-08ae031d0f7c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f9a40ef-qrxrd\" (UID: \"08f82f8d-8213-423d-8df8-08ae031d0f7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f9a40ef-qrxrd"
Nov 02 03:19:29.093219 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:29.093191    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08f82f8d-8213-423d-8df8-08ae031d0f7c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f9a40ef-qrxrd\" (UID: \"08f82f8d-8213-423d-8df8-08ae031d0f7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f9a40ef-qrxrd"
Nov 02 03:19:29.148099 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:29.148032    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7x5pw\" (UniqueName: \"kubernetes.io/projected/08f82f8d-8213-423d-8df8-08ae031d0f7c-kube-api-access-7x5pw\") pod \"maas-api-revoke-sub-8f9a40ef-qrxrd\" (UID: \"08f82f8d-8213-423d-8df8-08ae031d0f7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f9a40ef-qrxrd"
Nov 02 03:19:29.171822 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:29.171792    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f9a40ef-qrxrd"
Nov 02 03:19:29.554984 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:19:29.554952    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08f82f8d_8213_423d_8df8_08ae031d0f7c.slice/crio-07981ce43af55123ca6175f3099fbefbaec1134a9c9d6b98e5eb2318e66310a5 WatchSource:0}: Error finding container 07981ce43af55123ca6175f3099fbefbaec1134a9c9d6b98e5eb2318e66310a5: Status 404 returned error can't find the container with id 07981ce43af55123ca6175f3099fbefbaec1134a9c9d6b98e5eb2318e66310a5
Nov 02 03:19:29.555604 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:29.555578    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f9a40ef-qrxrd"]
Nov 02 03:19:29.642715 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:29.642675    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f9a40ef-qrxrd" event={"ID":"08f82f8d-8213-423d-8df8-08ae031d0f7c","Type":"ContainerStarted","Data":"07981ce43af55123ca6175f3099fbefbaec1134a9c9d6b98e5eb2318e66310a5"}
Nov 02 03:19:30.652778 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:30.652740    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f9a40ef-qrxrd" event={"ID":"08f82f8d-8213-423d-8df8-08ae031d0f7c","Type":"ContainerStarted","Data":"0211042c283a59769404ae5580a0facf4b51b060b6f8b7f0bfcd091dc3bdc67a"}
Nov 02 03:19:30.774876 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:30.774811    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f9a40ef-qrxrd" podStartSLOduration=2.774792786 podStartE2EDuration="2.774792786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:19:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:19:30.76750709 +0000 UTC m=+1911.346114471" watchObservedRunningTime="2026-11-02 03:19:30.774792786 +0000 UTC m=+1911.353400167"
Nov 02 03:19:31.662974 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:31.662938    4496 generic.go:356] "Generic (PLEG): container finished" podID="08f82f8d-8213-423d-8df8-08ae031d0f7c" containerID="0211042c283a59769404ae5580a0facf4b51b060b6f8b7f0bfcd091dc3bdc67a" exitCode=0
Nov 02 03:19:31.663437 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:31.663043    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f9a40ef-qrxrd" event={"ID":"08f82f8d-8213-423d-8df8-08ae031d0f7c","Type":"ContainerDied","Data":"0211042c283a59769404ae5580a0facf4b51b060b6f8b7f0bfcd091dc3bdc67a"}
Nov 02 03:19:32.830038 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:32.829999    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f9a40ef-qrxrd"
Nov 02 03:19:32.960524 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:32.960432    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08f82f8d-8213-423d-8df8-08ae031d0f7c-kube-api-access-7x5pw\" (UniqueName: \"kubernetes.io/projected/08f82f8d-8213-423d-8df8-08ae031d0f7c-kube-api-access-7x5pw\") pod \"08f82f8d-8213-423d-8df8-08ae031d0f7c\" (UID: \"08f82f8d-8213-423d-8df8-08ae031d0f7c\") "
Nov 02 03:19:32.960696 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:32.960590    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08f82f8d-8213-423d-8df8-08ae031d0f7c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08f82f8d-8213-423d-8df8-08ae031d0f7c-maas-api-openshift-ca\") pod \"08f82f8d-8213-423d-8df8-08ae031d0f7c\" (UID: \"08f82f8d-8213-423d-8df8-08ae031d0f7c\") "
Nov 02 03:19:32.960696 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:32.960630    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08f82f8d-8213-423d-8df8-08ae031d0f7c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08f82f8d-8213-423d-8df8-08ae031d0f7c-maas-api-xks-ca\") pod \"08f82f8d-8213-423d-8df8-08ae031d0f7c\" (UID: \"08f82f8d-8213-423d-8df8-08ae031d0f7c\") "
Nov 02 03:19:32.961331 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:32.960954    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08f82f8d-8213-423d-8df8-08ae031d0f7c-maas-api-openshift-ca" pod "08f82f8d-8213-423d-8df8-08ae031d0f7c" (UID: "08f82f8d-8213-423d-8df8-08ae031d0f7c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:19:32.961331 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:32.961217    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08f82f8d-8213-423d-8df8-08ae031d0f7c-maas-api-openshift-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:19:32.969930 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:32.963236    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08f82f8d-8213-423d-8df8-08ae031d0f7c-kube-api-access-7x5pw" pod "08f82f8d-8213-423d-8df8-08ae031d0f7c" (UID: "08f82f8d-8213-423d-8df8-08ae031d0f7c"). InnerVolumeSpecName "kube-api-access-7x5pw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:19:32.969930 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:32.963523    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08f82f8d-8213-423d-8df8-08ae031d0f7c-maas-api-xks-ca" pod "08f82f8d-8213-423d-8df8-08ae031d0f7c" (UID: "08f82f8d-8213-423d-8df8-08ae031d0f7c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 03:19:33.062868 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:33.062830    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7x5pw\" (UniqueName: \"kubernetes.io/projected/08f82f8d-8213-423d-8df8-08ae031d0f7c-kube-api-access-7x5pw\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:19:33.062868 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:33.062865    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08f82f8d-8213-423d-8df8-08ae031d0f7c-maas-api-xks-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:19:33.529760 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:19:33.529720    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 03:19:33.685204 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:33.685163    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f9a40ef-qrxrd"
Nov 02 03:19:33.685387 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:33.685173    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f9a40ef-qrxrd" event={"ID":"08f82f8d-8213-423d-8df8-08ae031d0f7c","Type":"ContainerDied","Data":"07981ce43af55123ca6175f3099fbefbaec1134a9c9d6b98e5eb2318e66310a5"}
Nov 02 03:19:33.685387 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:33.685330    4496 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="07981ce43af55123ca6175f3099fbefbaec1134a9c9d6b98e5eb2318e66310a5"
Nov 02 03:19:39.080428 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:39.080397    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-330c7d4b-7jkx4"]
Nov 02 03:19:39.081230 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:39.081212    4496 cpu_manager.go:414] "containerMap: removing container" podUID="08f82f8d-8213-423d-8df8-08ae031d0f7c" containerName="revoke-keys"
Nov 02 03:19:39.081230 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:39.081231    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08f82f8d-8213-423d-8df8-08ae031d0f7c" containerName="revoke-keys"
Nov 02 03:19:39.081400 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:39.081390    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="08f82f8d-8213-423d-8df8-08ae031d0f7c" containerName="revoke-keys"
Nov 02 03:19:39.085501 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:39.085473    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-330c7d4b-7jkx4"
Nov 02 03:19:39.142217 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:39.142185    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-330c7d4b-7jkx4"]
Nov 02 03:19:39.142964 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:39.142932    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50032077-80e1-4c33-8dfd-faf513f9ecc8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-330c7d4b-7jkx4\" (UID: \"50032077-80e1-4c33-8dfd-faf513f9ecc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-330c7d4b-7jkx4"
Nov 02 03:19:39.143208 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:39.143186    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zthc\" (UniqueName: \"kubernetes.io/projected/50032077-80e1-4c33-8dfd-faf513f9ecc8-kube-api-access-6zthc\") pod \"maas-api-revoke-sub-330c7d4b-7jkx4\" (UID: \"50032077-80e1-4c33-8dfd-faf513f9ecc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-330c7d4b-7jkx4"
Nov 02 03:19:39.143634 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:39.143227    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50032077-80e1-4c33-8dfd-faf513f9ecc8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-330c7d4b-7jkx4\" (UID: \"50032077-80e1-4c33-8dfd-faf513f9ecc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-330c7d4b-7jkx4"
Nov 02 03:19:39.185663 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:39.185638    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 03:19:39.185785 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:39.185676    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 03:19:39.185785 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:39.185710    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 03:19:39.189000 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:39.188975    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9z5bk\""
Nov 02 03:19:39.244160 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:39.244137    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50032077-80e1-4c33-8dfd-faf513f9ecc8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-330c7d4b-7jkx4\" (UID: \"50032077-80e1-4c33-8dfd-faf513f9ecc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-330c7d4b-7jkx4"
Nov 02 03:19:39.244280 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:39.244234    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zthc\" (UniqueName: \"kubernetes.io/projected/50032077-80e1-4c33-8dfd-faf513f9ecc8-kube-api-access-6zthc\") pod \"maas-api-revoke-sub-330c7d4b-7jkx4\" (UID: \"50032077-80e1-4c33-8dfd-faf513f9ecc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-330c7d4b-7jkx4"
Nov 02 03:19:39.244280 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:39.244259    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50032077-80e1-4c33-8dfd-faf513f9ecc8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-330c7d4b-7jkx4\" (UID: \"50032077-80e1-4c33-8dfd-faf513f9ecc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-330c7d4b-7jkx4"
Nov 02 03:19:39.244871 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:39.244847    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50032077-80e1-4c33-8dfd-faf513f9ecc8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-330c7d4b-7jkx4\" (UID: \"50032077-80e1-4c33-8dfd-faf513f9ecc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-330c7d4b-7jkx4"
Nov 02 03:19:39.246507 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:39.246480    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50032077-80e1-4c33-8dfd-faf513f9ecc8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-330c7d4b-7jkx4\" (UID: \"50032077-80e1-4c33-8dfd-faf513f9ecc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-330c7d4b-7jkx4"
Nov 02 03:19:39.301590 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:39.301563    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zthc\" (UniqueName: \"kubernetes.io/projected/50032077-80e1-4c33-8dfd-faf513f9ecc8-kube-api-access-6zthc\") pod \"maas-api-revoke-sub-330c7d4b-7jkx4\" (UID: \"50032077-80e1-4c33-8dfd-faf513f9ecc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-330c7d4b-7jkx4"
Nov 02 03:19:39.399598 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:39.399506    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-330c7d4b-7jkx4"
Nov 02 03:19:39.585303 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:39.585269    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fde0b86e-pvbq5"]
Nov 02 03:19:39.590854 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:39.590828    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fde0b86e-pvbq5"
Nov 02 03:19:39.647101 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:39.647067    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fde0b86e-pvbq5"]
Nov 02 03:19:39.649111 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:39.649082    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e174f62e-606b-4970-aa05-32df9c3e7fbd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fde0b86e-pvbq5\" (UID: \"e174f62e-606b-4970-aa05-32df9c3e7fbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fde0b86e-pvbq5"
Nov 02 03:19:39.649251 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:39.649143    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmh26\" (UniqueName: \"kubernetes.io/projected/e174f62e-606b-4970-aa05-32df9c3e7fbd-kube-api-access-vmh26\") pod \"maas-api-revoke-sub-fde0b86e-pvbq5\" (UID: \"e174f62e-606b-4970-aa05-32df9c3e7fbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fde0b86e-pvbq5"
Nov 02 03:19:39.649370 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:39.649345    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e174f62e-606b-4970-aa05-32df9c3e7fbd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fde0b86e-pvbq5\" (UID: \"e174f62e-606b-4970-aa05-32df9c3e7fbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fde0b86e-pvbq5"
Nov 02 03:19:39.751405 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:39.751373    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e174f62e-606b-4970-aa05-32df9c3e7fbd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fde0b86e-pvbq5\" (UID: \"e174f62e-606b-4970-aa05-32df9c3e7fbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fde0b86e-pvbq5"
Nov 02 03:19:39.751625 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:39.751430    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmh26\" (UniqueName: \"kubernetes.io/projected/e174f62e-606b-4970-aa05-32df9c3e7fbd-kube-api-access-vmh26\") pod \"maas-api-revoke-sub-fde0b86e-pvbq5\" (UID: \"e174f62e-606b-4970-aa05-32df9c3e7fbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fde0b86e-pvbq5"
Nov 02 03:19:39.751625 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:39.751524    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e174f62e-606b-4970-aa05-32df9c3e7fbd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fde0b86e-pvbq5\" (UID: \"e174f62e-606b-4970-aa05-32df9c3e7fbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fde0b86e-pvbq5"
Nov 02 03:19:39.752263 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:39.752235    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e174f62e-606b-4970-aa05-32df9c3e7fbd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fde0b86e-pvbq5\" (UID: \"e174f62e-606b-4970-aa05-32df9c3e7fbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fde0b86e-pvbq5"
Nov 02 03:19:39.753938 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:39.753910    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e174f62e-606b-4970-aa05-32df9c3e7fbd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fde0b86e-pvbq5\" (UID: \"e174f62e-606b-4970-aa05-32df9c3e7fbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fde0b86e-pvbq5"
Nov 02 03:19:39.808849 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:39.808813    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmh26\" (UniqueName: \"kubernetes.io/projected/e174f62e-606b-4970-aa05-32df9c3e7fbd-kube-api-access-vmh26\") pod \"maas-api-revoke-sub-fde0b86e-pvbq5\" (UID: \"e174f62e-606b-4970-aa05-32df9c3e7fbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fde0b86e-pvbq5"
Nov 02 03:19:39.820002 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:19:39.819973    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50032077_80e1_4c33_8dfd_faf513f9ecc8.slice/crio-7da125d073c50b207aba19096d6387071bda7e00f925b423580e13da06d88118 WatchSource:0}: Error finding container 7da125d073c50b207aba19096d6387071bda7e00f925b423580e13da06d88118: Status 404 returned error can't find the container with id 7da125d073c50b207aba19096d6387071bda7e00f925b423580e13da06d88118
Nov 02 03:19:39.821258 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:39.821235    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-330c7d4b-7jkx4"]
Nov 02 03:19:39.906564 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:39.906531    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fde0b86e-pvbq5"
Nov 02 03:19:40.183348 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:40.183319    4496 scope.go:122] "RemoveContainer" containerID="01e2fddb4f0e48331e4632319437e0b1273743466a9685ebcdc4236be094f502"
Nov 02 03:19:40.194087 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:40.194063    4496 scope.go:122] "RemoveContainer" containerID="617bacde990926e12001bd1e6dbf2856dd1f58c6d4c9b85277bbdf94411f3314"
Nov 02 03:19:40.203792 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:40.203771    4496 scope.go:122] "RemoveContainer" containerID="1b0a1c915f45520f1ba81be8c1d45c6f07c4f9590450fd6506c3fc9e027c6aff"
Nov 02 03:19:40.213338 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:40.213317    4496 scope.go:122] "RemoveContainer" containerID="c773fe88a57ed083ce65d5420da0bd5cfa2905e082de5d0bf9105aae6ff0070f"
Nov 02 03:19:40.222292 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:40.222269    4496 scope.go:122] "RemoveContainer" containerID="3349496a244a764722e55b12fd44d4731a331c34f40853527e4b1161529dc196"
Nov 02 03:19:40.231637 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:40.231617    4496 scope.go:122] "RemoveContainer" containerID="6a8eae85c0e24b7350fc89dc3b43b487e6786840dc69056689ee8059e7d63fea"
Nov 02 03:19:40.289693 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:40.289669    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fde0b86e-pvbq5"]
Nov 02 03:19:40.294830 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:19:40.294796    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode174f62e_606b_4970_aa05_32df9c3e7fbd.slice/crio-bb966b8b6485bdb563e3f4ea297251a0e32d651490857bc7032e966052b7e05a WatchSource:0}: Error finding container bb966b8b6485bdb563e3f4ea297251a0e32d651490857bc7032e966052b7e05a: Status 404 returned error can't find the container with id bb966b8b6485bdb563e3f4ea297251a0e32d651490857bc7032e966052b7e05a
Nov 02 03:19:40.746862 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:40.746824    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fde0b86e-pvbq5" event={"ID":"e174f62e-606b-4970-aa05-32df9c3e7fbd","Type":"ContainerStarted","Data":"15661fcf126c68ab7b9a91035d9b2482f4d0245deb28495677d9be78e2374c67"}
Nov 02 03:19:40.747074 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:40.746884    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fde0b86e-pvbq5" event={"ID":"e174f62e-606b-4970-aa05-32df9c3e7fbd","Type":"ContainerStarted","Data":"bb966b8b6485bdb563e3f4ea297251a0e32d651490857bc7032e966052b7e05a"}
Nov 02 03:19:40.748763 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:40.748738    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-330c7d4b-7jkx4" event={"ID":"50032077-80e1-4c33-8dfd-faf513f9ecc8","Type":"ContainerStarted","Data":"2b44f9e64c155f55ed911023b843f0631b20a8eb36843c324fa38f85827de085"}
Nov 02 03:19:40.748872 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:40.748778    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-330c7d4b-7jkx4" event={"ID":"50032077-80e1-4c33-8dfd-faf513f9ecc8","Type":"ContainerStarted","Data":"7da125d073c50b207aba19096d6387071bda7e00f925b423580e13da06d88118"}
Nov 02 03:19:40.865288 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:40.865193    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fde0b86e-pvbq5" podStartSLOduration=1.865179447 podStartE2EDuration="1.865179447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:19:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:19:40.86072377 +0000 UTC m=+1921.439331150" watchObservedRunningTime="2026-11-02 03:19:40.865179447 +0000 UTC m=+1921.443786828"
Nov 02 03:19:41.758235 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:41.758200    4496 generic.go:356] "Generic (PLEG): container finished" podID="50032077-80e1-4c33-8dfd-faf513f9ecc8" containerID="2b44f9e64c155f55ed911023b843f0631b20a8eb36843c324fa38f85827de085" exitCode=0
Nov 02 03:19:41.758633 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:41.758282    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-330c7d4b-7jkx4" event={"ID":"50032077-80e1-4c33-8dfd-faf513f9ecc8","Type":"ContainerDied","Data":"2b44f9e64c155f55ed911023b843f0631b20a8eb36843c324fa38f85827de085"}
Nov 02 03:19:41.759965 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:41.759945    4496 generic.go:356] "Generic (PLEG): container finished" podID="e174f62e-606b-4970-aa05-32df9c3e7fbd" containerID="15661fcf126c68ab7b9a91035d9b2482f4d0245deb28495677d9be78e2374c67" exitCode=0
Nov 02 03:19:41.760090 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:41.760042    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fde0b86e-pvbq5" event={"ID":"e174f62e-606b-4970-aa05-32df9c3e7fbd","Type":"ContainerDied","Data":"15661fcf126c68ab7b9a91035d9b2482f4d0245deb28495677d9be78e2374c67"}
Nov 02 03:19:42.954705 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:42.954678    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-330c7d4b-7jkx4"
Nov 02 03:19:42.979726 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:42.979702    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fde0b86e-pvbq5"
Nov 02 03:19:42.995609 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:42.995586    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50032077-80e1-4c33-8dfd-faf513f9ecc8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50032077-80e1-4c33-8dfd-faf513f9ecc8-maas-api-xks-ca\") pod \"50032077-80e1-4c33-8dfd-faf513f9ecc8\" (UID: \"50032077-80e1-4c33-8dfd-faf513f9ecc8\") "
Nov 02 03:19:42.995752 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:42.995734    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50032077-80e1-4c33-8dfd-faf513f9ecc8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50032077-80e1-4c33-8dfd-faf513f9ecc8-maas-api-openshift-ca\") pod \"50032077-80e1-4c33-8dfd-faf513f9ecc8\" (UID: \"50032077-80e1-4c33-8dfd-faf513f9ecc8\") "
Nov 02 03:19:42.995950 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:42.995931    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50032077-80e1-4c33-8dfd-faf513f9ecc8-kube-api-access-6zthc\" (UniqueName: \"kubernetes.io/projected/50032077-80e1-4c33-8dfd-faf513f9ecc8-kube-api-access-6zthc\") pod \"50032077-80e1-4c33-8dfd-faf513f9ecc8\" (UID: \"50032077-80e1-4c33-8dfd-faf513f9ecc8\") "
Nov 02 03:19:42.996140 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:42.996114    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50032077-80e1-4c33-8dfd-faf513f9ecc8-maas-api-openshift-ca" pod "50032077-80e1-4c33-8dfd-faf513f9ecc8" (UID: "50032077-80e1-4c33-8dfd-faf513f9ecc8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:19:42.996509 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:42.996487    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50032077-80e1-4c33-8dfd-faf513f9ecc8-maas-api-openshift-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:19:42.998103 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:42.998077    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50032077-80e1-4c33-8dfd-faf513f9ecc8-kube-api-access-6zthc" pod "50032077-80e1-4c33-8dfd-faf513f9ecc8" (UID: "50032077-80e1-4c33-8dfd-faf513f9ecc8"). InnerVolumeSpecName "kube-api-access-6zthc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:19:42.998243 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:42.998222    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50032077-80e1-4c33-8dfd-faf513f9ecc8-maas-api-xks-ca" pod "50032077-80e1-4c33-8dfd-faf513f9ecc8" (UID: "50032077-80e1-4c33-8dfd-faf513f9ecc8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 03:19:43.097261 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:43.097167    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e174f62e-606b-4970-aa05-32df9c3e7fbd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e174f62e-606b-4970-aa05-32df9c3e7fbd-maas-api-xks-ca\") pod \"e174f62e-606b-4970-aa05-32df9c3e7fbd\" (UID: \"e174f62e-606b-4970-aa05-32df9c3e7fbd\") "
Nov 02 03:19:43.097437 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:43.097314    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e174f62e-606b-4970-aa05-32df9c3e7fbd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e174f62e-606b-4970-aa05-32df9c3e7fbd-maas-api-openshift-ca\") pod \"e174f62e-606b-4970-aa05-32df9c3e7fbd\" (UID: \"e174f62e-606b-4970-aa05-32df9c3e7fbd\") "
Nov 02 03:19:43.097519 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:43.097456    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e174f62e-606b-4970-aa05-32df9c3e7fbd-kube-api-access-vmh26\" (UniqueName: \"kubernetes.io/projected/e174f62e-606b-4970-aa05-32df9c3e7fbd-kube-api-access-vmh26\") pod \"e174f62e-606b-4970-aa05-32df9c3e7fbd\" (UID: \"e174f62e-606b-4970-aa05-32df9c3e7fbd\") "
Nov 02 03:19:43.097773 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:43.097738    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e174f62e-606b-4970-aa05-32df9c3e7fbd-maas-api-openshift-ca" pod "e174f62e-606b-4970-aa05-32df9c3e7fbd" (UID: "e174f62e-606b-4970-aa05-32df9c3e7fbd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:19:43.098146 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:43.098120    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zthc\" (UniqueName: \"kubernetes.io/projected/50032077-80e1-4c33-8dfd-faf513f9ecc8-kube-api-access-6zthc\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:19:43.098261 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:43.098151    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50032077-80e1-4c33-8dfd-faf513f9ecc8-maas-api-xks-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:19:43.098261 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:43.098167    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e174f62e-606b-4970-aa05-32df9c3e7fbd-maas-api-openshift-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:19:43.100086 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:43.100052    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e174f62e-606b-4970-aa05-32df9c3e7fbd-maas-api-xks-ca" pod "e174f62e-606b-4970-aa05-32df9c3e7fbd" (UID: "e174f62e-606b-4970-aa05-32df9c3e7fbd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 03:19:43.100194 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:43.100167    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e174f62e-606b-4970-aa05-32df9c3e7fbd-kube-api-access-vmh26" pod "e174f62e-606b-4970-aa05-32df9c3e7fbd" (UID: "e174f62e-606b-4970-aa05-32df9c3e7fbd"). InnerVolumeSpecName "kube-api-access-vmh26". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:19:43.199956 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:43.199919    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmh26\" (UniqueName: \"kubernetes.io/projected/e174f62e-606b-4970-aa05-32df9c3e7fbd-kube-api-access-vmh26\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:19:43.199956 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:43.199955    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e174f62e-606b-4970-aa05-32df9c3e7fbd-maas-api-xks-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:19:43.778284 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:43.778258    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fde0b86e-pvbq5"
Nov 02 03:19:43.778430 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:43.778300    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fde0b86e-pvbq5" event={"ID":"e174f62e-606b-4970-aa05-32df9c3e7fbd","Type":"ContainerDied","Data":"bb966b8b6485bdb563e3f4ea297251a0e32d651490857bc7032e966052b7e05a"}
Nov 02 03:19:43.778430 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:43.778350    4496 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bb966b8b6485bdb563e3f4ea297251a0e32d651490857bc7032e966052b7e05a"
Nov 02 03:19:43.779848 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:43.779826    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-330c7d4b-7jkx4"
Nov 02 03:19:43.779963 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:43.779850    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-330c7d4b-7jkx4" event={"ID":"50032077-80e1-4c33-8dfd-faf513f9ecc8","Type":"ContainerDied","Data":"7da125d073c50b207aba19096d6387071bda7e00f925b423580e13da06d88118"}
Nov 02 03:19:43.779963 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:43.779888    4496 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7da125d073c50b207aba19096d6387071bda7e00f925b423580e13da06d88118"
Nov 02 03:19:52.380805 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:52.364842    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61cf3d87-jjgxb"]
Nov 02 03:19:52.388905 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:52.385163    4496 cpu_manager.go:414] "containerMap: removing container" podUID="50032077-80e1-4c33-8dfd-faf513f9ecc8" containerName="revoke-keys"
Nov 02 03:19:52.388905 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:52.385194    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50032077-80e1-4c33-8dfd-faf513f9ecc8" containerName="revoke-keys"
Nov 02 03:19:52.388905 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:52.385217    4496 cpu_manager.go:414] "containerMap: removing container" podUID="e174f62e-606b-4970-aa05-32df9c3e7fbd" containerName="revoke-keys"
Nov 02 03:19:52.388905 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:52.385224    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e174f62e-606b-4970-aa05-32df9c3e7fbd" containerName="revoke-keys"
Nov 02 03:19:52.388905 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:52.385396    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="50032077-80e1-4c33-8dfd-faf513f9ecc8" containerName="revoke-keys"
Nov 02 03:19:52.388905 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:52.385409    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="e174f62e-606b-4970-aa05-32df9c3e7fbd" containerName="revoke-keys"
Nov 02 03:19:52.443044 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:52.434237    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61cf3d87-jjgxb"]
Nov 02 03:19:52.443044 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:52.434364    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61cf3d87-jjgxb"
Nov 02 03:19:52.495734 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:52.493276    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 03:19:52.517359 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:52.517323    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rrs9\" (UniqueName: \"kubernetes.io/projected/27f971ed-34be-47b3-ac97-ab4710e5a412-kube-api-access-8rrs9\") pod \"maas-api-revoke-sub-61cf3d87-jjgxb\" (UID: \"27f971ed-34be-47b3-ac97-ab4710e5a412\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61cf3d87-jjgxb"
Nov 02 03:19:52.517542 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:52.517522    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27f971ed-34be-47b3-ac97-ab4710e5a412-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-61cf3d87-jjgxb\" (UID: \"27f971ed-34be-47b3-ac97-ab4710e5a412\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61cf3d87-jjgxb"
Nov 02 03:19:52.517600 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:52.517581    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27f971ed-34be-47b3-ac97-ab4710e5a412-maas-api-xks-ca\") pod \"maas-api-revoke-sub-61cf3d87-jjgxb\" (UID: \"27f971ed-34be-47b3-ac97-ab4710e5a412\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61cf3d87-jjgxb"
Nov 02 03:19:52.546202 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:52.543661    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 03:19:52.546202 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:52.543751    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 03:19:52.546202 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:52.543774    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9z5bk\""
Nov 02 03:19:52.618747 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:52.618716    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27f971ed-34be-47b3-ac97-ab4710e5a412-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-61cf3d87-jjgxb\" (UID: \"27f971ed-34be-47b3-ac97-ab4710e5a412\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61cf3d87-jjgxb"
Nov 02 03:19:52.618911 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:52.618769    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27f971ed-34be-47b3-ac97-ab4710e5a412-maas-api-xks-ca\") pod \"maas-api-revoke-sub-61cf3d87-jjgxb\" (UID: \"27f971ed-34be-47b3-ac97-ab4710e5a412\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61cf3d87-jjgxb"
Nov 02 03:19:52.618953 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:52.618916    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rrs9\" (UniqueName: \"kubernetes.io/projected/27f971ed-34be-47b3-ac97-ab4710e5a412-kube-api-access-8rrs9\") pod \"maas-api-revoke-sub-61cf3d87-jjgxb\" (UID: \"27f971ed-34be-47b3-ac97-ab4710e5a412\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61cf3d87-jjgxb"
Nov 02 03:19:52.619490 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:52.619466    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27f971ed-34be-47b3-ac97-ab4710e5a412-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-61cf3d87-jjgxb\" (UID: \"27f971ed-34be-47b3-ac97-ab4710e5a412\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61cf3d87-jjgxb"
Nov 02 03:19:52.625331 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:52.625302    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27f971ed-34be-47b3-ac97-ab4710e5a412-maas-api-xks-ca\") pod \"maas-api-revoke-sub-61cf3d87-jjgxb\" (UID: \"27f971ed-34be-47b3-ac97-ab4710e5a412\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61cf3d87-jjgxb"
Nov 02 03:19:52.678006 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:52.677923    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rrs9\" (UniqueName: \"kubernetes.io/projected/27f971ed-34be-47b3-ac97-ab4710e5a412-kube-api-access-8rrs9\") pod \"maas-api-revoke-sub-61cf3d87-jjgxb\" (UID: \"27f971ed-34be-47b3-ac97-ab4710e5a412\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61cf3d87-jjgxb"
Nov 02 03:19:52.810801 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:52.810766    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61cf3d87-jjgxb"
Nov 02 03:19:53.223729 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:19:53.223693    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27f971ed_34be_47b3_ac97_ab4710e5a412.slice/crio-a8adf33e9d76aa9a5d42231c4ba7f7121aa6225253e080a19635a0012ef48223 WatchSource:0}: Error finding container a8adf33e9d76aa9a5d42231c4ba7f7121aa6225253e080a19635a0012ef48223: Status 404 returned error can't find the container with id a8adf33e9d76aa9a5d42231c4ba7f7121aa6225253e080a19635a0012ef48223
Nov 02 03:19:53.224690 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:53.224665    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61cf3d87-jjgxb"]
Nov 02 03:19:54.174689 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:54.174655    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61cf3d87-jjgxb" event={"ID":"27f971ed-34be-47b3-ac97-ab4710e5a412","Type":"ContainerStarted","Data":"ad3a7ba90784f96b6884f905ba1cfec591dea359924d401d0f24f418d26166ab"}
Nov 02 03:19:54.175175 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:54.174714    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61cf3d87-jjgxb" event={"ID":"27f971ed-34be-47b3-ac97-ab4710e5a412","Type":"ContainerStarted","Data":"a8adf33e9d76aa9a5d42231c4ba7f7121aa6225253e080a19635a0012ef48223"}
Nov 02 03:19:54.296991 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:54.296915    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61cf3d87-jjgxb" podStartSLOduration=2.296894465 podStartE2EDuration="2.296894465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:19:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:19:54.288600478 +0000 UTC m=+1934.867207860" watchObservedRunningTime="2026-11-02 03:19:54.296894465 +0000 UTC m=+1934.875501846"
Nov 02 03:19:55.185932 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:55.185895    4496 generic.go:356] "Generic (PLEG): container finished" podID="27f971ed-34be-47b3-ac97-ab4710e5a412" containerID="ad3a7ba90784f96b6884f905ba1cfec591dea359924d401d0f24f418d26166ab" exitCode=0
Nov 02 03:19:55.186369 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:55.185971    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61cf3d87-jjgxb" event={"ID":"27f971ed-34be-47b3-ac97-ab4710e5a412","Type":"ContainerDied","Data":"ad3a7ba90784f96b6884f905ba1cfec591dea359924d401d0f24f418d26166ab"}
Nov 02 03:19:56.367674 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:56.367644    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61cf3d87-jjgxb"
Nov 02 03:19:56.482360 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:56.482325    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27f971ed-34be-47b3-ac97-ab4710e5a412-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27f971ed-34be-47b3-ac97-ab4710e5a412-maas-api-openshift-ca\") pod \"27f971ed-34be-47b3-ac97-ab4710e5a412\" (UID: \"27f971ed-34be-47b3-ac97-ab4710e5a412\") "
Nov 02 03:19:56.482529 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:56.482382    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27f971ed-34be-47b3-ac97-ab4710e5a412-kube-api-access-8rrs9\" (UniqueName: \"kubernetes.io/projected/27f971ed-34be-47b3-ac97-ab4710e5a412-kube-api-access-8rrs9\") pod \"27f971ed-34be-47b3-ac97-ab4710e5a412\" (UID: \"27f971ed-34be-47b3-ac97-ab4710e5a412\") "
Nov 02 03:19:56.482585 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:56.482527    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27f971ed-34be-47b3-ac97-ab4710e5a412-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27f971ed-34be-47b3-ac97-ab4710e5a412-maas-api-xks-ca\") pod \"27f971ed-34be-47b3-ac97-ab4710e5a412\" (UID: \"27f971ed-34be-47b3-ac97-ab4710e5a412\") "
Nov 02 03:19:56.483585 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:56.483554    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27f971ed-34be-47b3-ac97-ab4710e5a412-maas-api-openshift-ca" pod "27f971ed-34be-47b3-ac97-ab4710e5a412" (UID: "27f971ed-34be-47b3-ac97-ab4710e5a412"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:19:56.487408 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:56.487383    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27f971ed-34be-47b3-ac97-ab4710e5a412-maas-api-xks-ca" pod "27f971ed-34be-47b3-ac97-ab4710e5a412" (UID: "27f971ed-34be-47b3-ac97-ab4710e5a412"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 03:19:56.487613 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:56.487588    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27f971ed-34be-47b3-ac97-ab4710e5a412-kube-api-access-8rrs9" pod "27f971ed-34be-47b3-ac97-ab4710e5a412" (UID: "27f971ed-34be-47b3-ac97-ab4710e5a412"). InnerVolumeSpecName "kube-api-access-8rrs9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:19:56.584648 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:56.584620    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27f971ed-34be-47b3-ac97-ab4710e5a412-maas-api-xks-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:19:56.584648 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:56.584647    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27f971ed-34be-47b3-ac97-ab4710e5a412-maas-api-openshift-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:19:56.584809 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:56.584657    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rrs9\" (UniqueName: \"kubernetes.io/projected/27f971ed-34be-47b3-ac97-ab4710e5a412-kube-api-access-8rrs9\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:19:57.203675 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:57.203640    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61cf3d87-jjgxb"
Nov 02 03:19:57.203831 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:57.203671    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61cf3d87-jjgxb" event={"ID":"27f971ed-34be-47b3-ac97-ab4710e5a412","Type":"ContainerDied","Data":"a8adf33e9d76aa9a5d42231c4ba7f7121aa6225253e080a19635a0012ef48223"}
Nov 02 03:19:57.203831 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:19:57.203723    4496 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a8adf33e9d76aa9a5d42231c4ba7f7121aa6225253e080a19635a0012ef48223"
Nov 02 03:20:00.729897 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:00.729853    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ed15c1-57bddc8787-l7kbv"]
Nov 02 03:20:00.731329 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:00.731299    4496 cpu_manager.go:414] "containerMap: removing container" podUID="27f971ed-34be-47b3-ac97-ab4710e5a412" containerName="revoke-keys"
Nov 02 03:20:00.731329 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:00.731327    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27f971ed-34be-47b3-ac97-ab4710e5a412" containerName="revoke-keys"
Nov 02 03:20:00.731543 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:00.731529    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="27f971ed-34be-47b3-ac97-ab4710e5a412" containerName="revoke-keys"
Nov 02 03:20:00.736081 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:00.736054    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ed15c1-57bddc8787-l7kbv"
Nov 02 03:20:00.788222 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:00.788189    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ed15c1-57bddc8787-l7kbv"]
Nov 02 03:20:00.832718 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:00.832685    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-ed15c1\""
Nov 02 03:20:00.880677 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:00.880641    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ed15c1-56c5d6557c-g5df2"]
Nov 02 03:20:00.885322 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:00.885298    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-ed15c1-56c5d6557c-g5df2"
Nov 02 03:20:00.885459 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:00.885355    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xtkw\" (UniqueName: \"kubernetes.io/projected/4cf48f92-69f5-47aa-8163-26f2d6df1921-kube-api-access-7xtkw\") pod \"payload-pre-processing-e2e-api-a-ed15c1-57bddc8787-l7kbv\" (UID: \"4cf48f92-69f5-47aa-8163-26f2d6df1921\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ed15c1-57bddc8787-l7kbv"
Nov 02 03:20:00.885517 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:00.885461    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cf48f92-69f5-47aa-8163-26f2d6df1921-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-ed15c1-57bddc8787-l7kbv\" (UID: \"4cf48f92-69f5-47aa-8163-26f2d6df1921\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ed15c1-57bddc8787-l7kbv"
Nov 02 03:20:00.946399 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:00.946366    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-ed15c1-dockercfg-6lqvl\""
Nov 02 03:20:00.970721 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:00.970663    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ed15c1-56c5d6557c-g5df2"]
Nov 02 03:20:00.991083 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:00.986934    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ac8f265-8646-430c-bca6-36655877def2-plugins-config-volume\") pod \"payload-processing-e2e-api-a-ed15c1-56c5d6557c-g5df2\" (UID: \"5ac8f265-8646-430c-bca6-36655877def2\") " pod="openshift-ingress/payload-processing-e2e-api-a-ed15c1-56c5d6557c-g5df2"
Nov 02 03:20:00.991083 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:00.986982    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jq4wn\" (UniqueName: \"kubernetes.io/projected/5ac8f265-8646-430c-bca6-36655877def2-kube-api-access-jq4wn\") pod \"payload-processing-e2e-api-a-ed15c1-56c5d6557c-g5df2\" (UID: \"5ac8f265-8646-430c-bca6-36655877def2\") " pod="openshift-ingress/payload-processing-e2e-api-a-ed15c1-56c5d6557c-g5df2"
Nov 02 03:20:00.991083 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:00.987164    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xtkw\" (UniqueName: \"kubernetes.io/projected/4cf48f92-69f5-47aa-8163-26f2d6df1921-kube-api-access-7xtkw\") pod \"payload-pre-processing-e2e-api-a-ed15c1-57bddc8787-l7kbv\" (UID: \"4cf48f92-69f5-47aa-8163-26f2d6df1921\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ed15c1-57bddc8787-l7kbv"
Nov 02 03:20:00.991083 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:00.987245    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cf48f92-69f5-47aa-8163-26f2d6df1921-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-ed15c1-57bddc8787-l7kbv\" (UID: \"4cf48f92-69f5-47aa-8163-26f2d6df1921\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ed15c1-57bddc8787-l7kbv"
Nov 02 03:20:00.991083 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:00.988259    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cf48f92-69f5-47aa-8163-26f2d6df1921-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-ed15c1-57bddc8787-l7kbv\" (UID: \"4cf48f92-69f5-47aa-8163-26f2d6df1921\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ed15c1-57bddc8787-l7kbv"
Nov 02 03:20:01.052326 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:01.050339    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xtkw\" (UniqueName: \"kubernetes.io/projected/4cf48f92-69f5-47aa-8163-26f2d6df1921-kube-api-access-7xtkw\") pod \"payload-pre-processing-e2e-api-a-ed15c1-57bddc8787-l7kbv\" (UID: \"4cf48f92-69f5-47aa-8163-26f2d6df1921\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ed15c1-57bddc8787-l7kbv"
Nov 02 03:20:01.054286 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:01.054252    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ed15c1-57bddc8787-l7kbv"
Nov 02 03:20:01.090671 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:01.090630    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ac8f265-8646-430c-bca6-36655877def2-plugins-config-volume\") pod \"payload-processing-e2e-api-a-ed15c1-56c5d6557c-g5df2\" (UID: \"5ac8f265-8646-430c-bca6-36655877def2\") " pod="openshift-ingress/payload-processing-e2e-api-a-ed15c1-56c5d6557c-g5df2"
Nov 02 03:20:01.090839 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:01.090686    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jq4wn\" (UniqueName: \"kubernetes.io/projected/5ac8f265-8646-430c-bca6-36655877def2-kube-api-access-jq4wn\") pod \"payload-processing-e2e-api-a-ed15c1-56c5d6557c-g5df2\" (UID: \"5ac8f265-8646-430c-bca6-36655877def2\") " pod="openshift-ingress/payload-processing-e2e-api-a-ed15c1-56c5d6557c-g5df2"
Nov 02 03:20:01.091533 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:01.091504    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ac8f265-8646-430c-bca6-36655877def2-plugins-config-volume\") pod \"payload-processing-e2e-api-a-ed15c1-56c5d6557c-g5df2\" (UID: \"5ac8f265-8646-430c-bca6-36655877def2\") " pod="openshift-ingress/payload-processing-e2e-api-a-ed15c1-56c5d6557c-g5df2"
Nov 02 03:20:01.149129 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:01.149064    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jq4wn\" (UniqueName: \"kubernetes.io/projected/5ac8f265-8646-430c-bca6-36655877def2-kube-api-access-jq4wn\") pod \"payload-processing-e2e-api-a-ed15c1-56c5d6557c-g5df2\" (UID: \"5ac8f265-8646-430c-bca6-36655877def2\") " pod="openshift-ingress/payload-processing-e2e-api-a-ed15c1-56c5d6557c-g5df2"
Nov 02 03:20:01.201522 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:01.201429    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-ed15c1-56c5d6557c-g5df2"
Nov 02 03:20:01.444495 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:20:01.444463    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cf48f92_69f5_47aa_8163_26f2d6df1921.slice/crio-44d1f3452f4129b602804b1b9ea25b8c677b356633a93af3d24d81c0d2d526f9 WatchSource:0}: Error finding container 44d1f3452f4129b602804b1b9ea25b8c677b356633a93af3d24d81c0d2d526f9: Status 404 returned error can't find the container with id 44d1f3452f4129b602804b1b9ea25b8c677b356633a93af3d24d81c0d2d526f9
Nov 02 03:20:01.448752 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:01.448724    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ed15c1-57bddc8787-l7kbv"]
Nov 02 03:20:01.586685 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:20:01.586656    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ac8f265_8646_430c_bca6_36655877def2.slice/crio-d44f6ea2f54188781b12c3e5ff3ae039014e44dcba6bbe53ccd1daa24e38aa2d WatchSource:0}: Error finding container d44f6ea2f54188781b12c3e5ff3ae039014e44dcba6bbe53ccd1daa24e38aa2d: Status 404 returned error can't find the container with id d44f6ea2f54188781b12c3e5ff3ae039014e44dcba6bbe53ccd1daa24e38aa2d
Nov 02 03:20:01.587324 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:01.587254    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ed15c1-56c5d6557c-g5df2"]
Nov 02 03:20:02.248571 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:02.248524    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ed15c1-57bddc8787-l7kbv" event={"ID":"4cf48f92-69f5-47aa-8163-26f2d6df1921","Type":"ContainerStarted","Data":"0d00e0c98d00b1689c7313b1a401b1a59eddbfacd71fd0746e46caae0cd8b578"}
Nov 02 03:20:02.249040 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:02.248628    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ed15c1-57bddc8787-l7kbv"
Nov 02 03:20:02.249040 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:02.248655    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ed15c1-57bddc8787-l7kbv" event={"ID":"4cf48f92-69f5-47aa-8163-26f2d6df1921","Type":"ContainerStarted","Data":"44d1f3452f4129b602804b1b9ea25b8c677b356633a93af3d24d81c0d2d526f9"}
Nov 02 03:20:02.250409 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:02.250384    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-ed15c1-56c5d6557c-g5df2" event={"ID":"5ac8f265-8646-430c-bca6-36655877def2","Type":"ContainerStarted","Data":"7240d587c48d2d15da42e1dc5d035be3e2e6eb681b15ea42c5a440183f088e14"}
Nov 02 03:20:02.250533 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:02.250423    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-ed15c1-56c5d6557c-g5df2" event={"ID":"5ac8f265-8646-430c-bca6-36655877def2","Type":"ContainerStarted","Data":"d44f6ea2f54188781b12c3e5ff3ae039014e44dcba6bbe53ccd1daa24e38aa2d"}
Nov 02 03:20:02.250533 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:02.250532    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-ed15c1-56c5d6557c-g5df2"
Nov 02 03:20:02.368177 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:02.368123    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ed15c1-57bddc8787-l7kbv" podStartSLOduration=2.368106234 podStartE2EDuration="2.368106234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:20:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:20:02.363537317 +0000 UTC m=+1942.942144695" watchObservedRunningTime="2026-11-02 03:20:02.368106234 +0000 UTC m=+1942.946713614"
Nov 02 03:20:02.482561 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:02.482504    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-ed15c1-56c5d6557c-g5df2" podStartSLOduration=2.482482577 podStartE2EDuration="2.482482577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:20:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:20:02.480431222 +0000 UTC m=+1943.059038600" watchObservedRunningTime="2026-11-02 03:20:02.482482577 +0000 UTC m=+1943.061089959"
Nov 02 03:20:03.526702 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:20:03.526664    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 03:20:05.578217 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:05.578179    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0d26af-64f5446f47-qnwqh"]
Nov 02 03:20:05.588795 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:05.588769    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0d26af-64f5446f47-qnwqh"
Nov 02 03:20:05.628186 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:05.628122    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0d26af-64f5446f47-qnwqh"]
Nov 02 03:20:05.660067 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:05.659995    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15d2020a-9cef-4fa1-b3f5-7d1f2f1a9bcf-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-0d26af-64f5446f47-qnwqh\" (UID: \"15d2020a-9cef-4fa1-b3f5-7d1f2f1a9bcf\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0d26af-64f5446f47-qnwqh"
Nov 02 03:20:05.672095 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:05.660458    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcv7f\" (UniqueName: \"kubernetes.io/projected/15d2020a-9cef-4fa1-b3f5-7d1f2f1a9bcf-kube-api-access-vcv7f\") pod \"payload-pre-processing-e2e-api-a-0d26af-64f5446f47-qnwqh\" (UID: \"15d2020a-9cef-4fa1-b3f5-7d1f2f1a9bcf\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0d26af-64f5446f47-qnwqh"
Nov 02 03:20:05.693037 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:05.672332    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-0d26af-dockercfg-j2npp\""
Nov 02 03:20:05.693037 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:05.672951    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-0d26af\""
Nov 02 03:20:05.743066 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:05.742178    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0d26af-59668b6f65-dz2f7"]
Nov 02 03:20:05.747918 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:05.747890    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-0d26af-59668b6f65-dz2f7"
Nov 02 03:20:05.761655 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:05.761627    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15d2020a-9cef-4fa1-b3f5-7d1f2f1a9bcf-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-0d26af-64f5446f47-qnwqh\" (UID: \"15d2020a-9cef-4fa1-b3f5-7d1f2f1a9bcf\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0d26af-64f5446f47-qnwqh"
Nov 02 03:20:05.761801 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:05.761760    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcv7f\" (UniqueName: \"kubernetes.io/projected/15d2020a-9cef-4fa1-b3f5-7d1f2f1a9bcf-kube-api-access-vcv7f\") pod \"payload-pre-processing-e2e-api-a-0d26af-64f5446f47-qnwqh\" (UID: \"15d2020a-9cef-4fa1-b3f5-7d1f2f1a9bcf\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0d26af-64f5446f47-qnwqh"
Nov 02 03:20:05.762501 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:05.762476    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15d2020a-9cef-4fa1-b3f5-7d1f2f1a9bcf-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-0d26af-64f5446f47-qnwqh\" (UID: \"15d2020a-9cef-4fa1-b3f5-7d1f2f1a9bcf\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0d26af-64f5446f47-qnwqh"
Nov 02 03:20:05.791167 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:05.791136    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0d26af-59668b6f65-dz2f7"]
Nov 02 03:20:05.820693 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:05.820664    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcv7f\" (UniqueName: \"kubernetes.io/projected/15d2020a-9cef-4fa1-b3f5-7d1f2f1a9bcf-kube-api-access-vcv7f\") pod \"payload-pre-processing-e2e-api-a-0d26af-64f5446f47-qnwqh\" (UID: \"15d2020a-9cef-4fa1-b3f5-7d1f2f1a9bcf\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0d26af-64f5446f47-qnwqh"
Nov 02 03:20:05.862888 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:05.862790    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpd9g\" (UniqueName: \"kubernetes.io/projected/d3a1ea4e-eb3e-462b-900c-ea2181bb88d9-kube-api-access-fpd9g\") pod \"payload-processing-e2e-api-a-0d26af-59668b6f65-dz2f7\" (UID: \"d3a1ea4e-eb3e-462b-900c-ea2181bb88d9\") " pod="openshift-ingress/payload-processing-e2e-api-a-0d26af-59668b6f65-dz2f7"
Nov 02 03:20:05.863069 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:05.862920    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3a1ea4e-eb3e-462b-900c-ea2181bb88d9-plugins-config-volume\") pod \"payload-processing-e2e-api-a-0d26af-59668b6f65-dz2f7\" (UID: \"d3a1ea4e-eb3e-462b-900c-ea2181bb88d9\") " pod="openshift-ingress/payload-processing-e2e-api-a-0d26af-59668b6f65-dz2f7"
Nov 02 03:20:05.904360 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:05.904329    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0d26af-64f5446f47-qnwqh"
Nov 02 03:20:05.965560 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:05.965114    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpd9g\" (UniqueName: \"kubernetes.io/projected/d3a1ea4e-eb3e-462b-900c-ea2181bb88d9-kube-api-access-fpd9g\") pod \"payload-processing-e2e-api-a-0d26af-59668b6f65-dz2f7\" (UID: \"d3a1ea4e-eb3e-462b-900c-ea2181bb88d9\") " pod="openshift-ingress/payload-processing-e2e-api-a-0d26af-59668b6f65-dz2f7"
Nov 02 03:20:05.965560 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:05.965202    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3a1ea4e-eb3e-462b-900c-ea2181bb88d9-plugins-config-volume\") pod \"payload-processing-e2e-api-a-0d26af-59668b6f65-dz2f7\" (UID: \"d3a1ea4e-eb3e-462b-900c-ea2181bb88d9\") " pod="openshift-ingress/payload-processing-e2e-api-a-0d26af-59668b6f65-dz2f7"
Nov 02 03:20:05.966931 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:05.966887    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3a1ea4e-eb3e-462b-900c-ea2181bb88d9-plugins-config-volume\") pod \"payload-processing-e2e-api-a-0d26af-59668b6f65-dz2f7\" (UID: \"d3a1ea4e-eb3e-462b-900c-ea2181bb88d9\") " pod="openshift-ingress/payload-processing-e2e-api-a-0d26af-59668b6f65-dz2f7"
Nov 02 03:20:06.026081 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:06.026048    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpd9g\" (UniqueName: \"kubernetes.io/projected/d3a1ea4e-eb3e-462b-900c-ea2181bb88d9-kube-api-access-fpd9g\") pod \"payload-processing-e2e-api-a-0d26af-59668b6f65-dz2f7\" (UID: \"d3a1ea4e-eb3e-462b-900c-ea2181bb88d9\") " pod="openshift-ingress/payload-processing-e2e-api-a-0d26af-59668b6f65-dz2f7"
Nov 02 03:20:06.063374 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:06.063346    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-0d26af-59668b6f65-dz2f7"
Nov 02 03:20:06.067575 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:06.067548    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-2ktj7"]
Nov 02 03:20:06.084080 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:06.078726    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-2ktj7"
Nov 02 03:20:06.130820 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:06.130691    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-2ktj7"]
Nov 02 03:20:06.168550 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:06.168031    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8d9e090-928d-489e-9eb4-82ad15bb6aa5-istiod-ca-cert\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-2ktj7\" (UID: \"a8d9e090-928d-489e-9eb4-82ad15bb6aa5\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-2ktj7"
Nov 02 03:20:06.168550 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:06.168100    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8d9e090-928d-489e-9eb4-82ad15bb6aa5-istio-envoy\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-2ktj7\" (UID: \"a8d9e090-928d-489e-9eb4-82ad15bb6aa5\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-2ktj7"
Nov 02 03:20:06.168550 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:06.168168    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8d9e090-928d-489e-9eb4-82ad15bb6aa5-istio-data\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-2ktj7\" (UID: \"a8d9e090-928d-489e-9eb4-82ad15bb6aa5\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-2ktj7"
Nov 02 03:20:06.168550 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:06.168211    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8d9e090-928d-489e-9eb4-82ad15bb6aa5-istio-token\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-2ktj7\" (UID: \"a8d9e090-928d-489e-9eb4-82ad15bb6aa5\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-2ktj7"
Nov 02 03:20:06.168550 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:06.168300    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8d9e090-928d-489e-9eb4-82ad15bb6aa5-istio-podinfo\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-2ktj7\" (UID: \"a8d9e090-928d-489e-9eb4-82ad15bb6aa5\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-2ktj7"
Nov 02 03:20:06.168550 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:06.168434    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8d9e090-928d-489e-9eb4-82ad15bb6aa5-workload-socket\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-2ktj7\" (UID: \"a8d9e090-928d-489e-9eb4-82ad15bb6aa5\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-2ktj7"
Nov 02 03:20:06.168550 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:06.168478    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8d9e090-928d-489e-9eb4-82ad15bb6aa5-credential-socket\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-2ktj7\" (UID: \"a8d9e090-928d-489e-9eb4-82ad15bb6aa5\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-2ktj7"
Nov 02 03:20:06.168550 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:06.168558    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8d9e090-928d-489e-9eb4-82ad15bb6aa5-workload-certs\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-2ktj7\" (UID: \"a8d9e090-928d-489e-9eb4-82ad15bb6aa5\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-2ktj7"
Nov 02 03:20:06.169099 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:06.168593    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-746t5\" (UniqueName: \"kubernetes.io/projected/a8d9e090-928d-489e-9eb4-82ad15bb6aa5-kube-api-access-746t5\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-2ktj7\" (UID: \"a8d9e090-928d-489e-9eb4-82ad15bb6aa5\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-2ktj7"
Nov 02 03:20:06.269956 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:06.269916    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8d9e090-928d-489e-9eb4-82ad15bb6aa5-workload-socket\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-2ktj7\" (UID: \"a8d9e090-928d-489e-9eb4-82ad15bb6aa5\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-2ktj7"
Nov 02 03:20:06.270156 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:06.269964    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8d9e090-928d-489e-9eb4-82ad15bb6aa5-credential-socket\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-2ktj7\" (UID: \"a8d9e090-928d-489e-9eb4-82ad15bb6aa5\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-2ktj7"
Nov 02 03:20:06.270156 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:06.270010    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8d9e090-928d-489e-9eb4-82ad15bb6aa5-workload-certs\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-2ktj7\" (UID: \"a8d9e090-928d-489e-9eb4-82ad15bb6aa5\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-2ktj7"
Nov 02 03:20:06.270156 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:06.270054    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-746t5\" (UniqueName: \"kubernetes.io/projected/a8d9e090-928d-489e-9eb4-82ad15bb6aa5-kube-api-access-746t5\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-2ktj7\" (UID: \"a8d9e090-928d-489e-9eb4-82ad15bb6aa5\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-2ktj7"
Nov 02 03:20:06.270156 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:06.270109    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8d9e090-928d-489e-9eb4-82ad15bb6aa5-istiod-ca-cert\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-2ktj7\" (UID: \"a8d9e090-928d-489e-9eb4-82ad15bb6aa5\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-2ktj7"
Nov 02 03:20:06.270156 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:06.270139    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8d9e090-928d-489e-9eb4-82ad15bb6aa5-istio-envoy\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-2ktj7\" (UID: \"a8d9e090-928d-489e-9eb4-82ad15bb6aa5\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-2ktj7"
Nov 02 03:20:06.270443 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:06.270420    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8d9e090-928d-489e-9eb4-82ad15bb6aa5-workload-socket\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-2ktj7\" (UID: \"a8d9e090-928d-489e-9eb4-82ad15bb6aa5\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-2ktj7"
Nov 02 03:20:06.270506 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:06.270472    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8d9e090-928d-489e-9eb4-82ad15bb6aa5-credential-socket\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-2ktj7\" (UID: \"a8d9e090-928d-489e-9eb4-82ad15bb6aa5\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-2ktj7"
Nov 02 03:20:06.270549 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:06.270495    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8d9e090-928d-489e-9eb4-82ad15bb6aa5-workload-certs\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-2ktj7\" (UID: \"a8d9e090-928d-489e-9eb4-82ad15bb6aa5\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-2ktj7"
Nov 02 03:20:06.270593 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:06.270581    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8d9e090-928d-489e-9eb4-82ad15bb6aa5-istio-data\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-2ktj7\" (UID: \"a8d9e090-928d-489e-9eb4-82ad15bb6aa5\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-2ktj7"
Nov 02 03:20:06.270633 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:06.270624    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8d9e090-928d-489e-9eb4-82ad15bb6aa5-istio-token\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-2ktj7\" (UID: \"a8d9e090-928d-489e-9eb4-82ad15bb6aa5\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-2ktj7"
Nov 02 03:20:06.270723 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:06.270706    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8d9e090-928d-489e-9eb4-82ad15bb6aa5-istio-podinfo\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-2ktj7\" (UID: \"a8d9e090-928d-489e-9eb4-82ad15bb6aa5\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-2ktj7"
Nov 02 03:20:06.270883 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:06.270859    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8d9e090-928d-489e-9eb4-82ad15bb6aa5-istiod-ca-cert\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-2ktj7\" (UID: \"a8d9e090-928d-489e-9eb4-82ad15bb6aa5\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-2ktj7"
Nov 02 03:20:06.270949 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:06.270870    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8d9e090-928d-489e-9eb4-82ad15bb6aa5-istio-data\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-2ktj7\" (UID: \"a8d9e090-928d-489e-9eb4-82ad15bb6aa5\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-2ktj7"
Nov 02 03:20:06.272555 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:06.272537    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8d9e090-928d-489e-9eb4-82ad15bb6aa5-istio-envoy\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-2ktj7\" (UID: \"a8d9e090-928d-489e-9eb4-82ad15bb6aa5\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-2ktj7"
Nov 02 03:20:06.273042 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:06.273007    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8d9e090-928d-489e-9eb4-82ad15bb6aa5-istio-podinfo\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-2ktj7\" (UID: \"a8d9e090-928d-489e-9eb4-82ad15bb6aa5\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-2ktj7"
Nov 02 03:20:06.295956 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:20:06.295935    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15d2020a_9cef_4fa1_b3f5_7d1f2f1a9bcf.slice/crio-2d64d3df7d2c5f4608ccc032f276615247cad841848ef2e4775de003c295fcdc WatchSource:0}: Error finding container 2d64d3df7d2c5f4608ccc032f276615247cad841848ef2e4775de003c295fcdc: Status 404 returned error can't find the container with id 2d64d3df7d2c5f4608ccc032f276615247cad841848ef2e4775de003c295fcdc
Nov 02 03:20:06.297921 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:06.297902    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0d26af-64f5446f47-qnwqh"]
Nov 02 03:20:06.327561 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:06.327539    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8d9e090-928d-489e-9eb4-82ad15bb6aa5-istio-token\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-2ktj7\" (UID: \"a8d9e090-928d-489e-9eb4-82ad15bb6aa5\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-2ktj7"
Nov 02 03:20:06.327653 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:06.327640    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-746t5\" (UniqueName: \"kubernetes.io/projected/a8d9e090-928d-489e-9eb4-82ad15bb6aa5-kube-api-access-746t5\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-2ktj7\" (UID: \"a8d9e090-928d-489e-9eb4-82ad15bb6aa5\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-2ktj7"
Nov 02 03:20:06.414886 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:06.414818    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-2ktj7"
Nov 02 03:20:06.478371 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:06.477688    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0d26af-59668b6f65-dz2f7"]
Nov 02 03:20:06.480184 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:20:06.480152    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3a1ea4e_eb3e_462b_900c_ea2181bb88d9.slice/crio-34bc285a738bdbb8cd11ce1fce15f8ce10c34c9ce899b41f2f45ab21c2021b1e WatchSource:0}: Error finding container 34bc285a738bdbb8cd11ce1fce15f8ce10c34c9ce899b41f2f45ab21c2021b1e: Status 404 returned error can't find the container with id 34bc285a738bdbb8cd11ce1fce15f8ce10c34c9ce899b41f2f45ab21c2021b1e
Nov 02 03:20:06.802858 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:20:06.802824    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8d9e090_928d_489e_9eb4_82ad15bb6aa5.slice/crio-b9cdedd775b1a44c925047fe2e40810d3a8b8e5ce75195c7f59b8e48357593d1 WatchSource:0}: Error finding container b9cdedd775b1a44c925047fe2e40810d3a8b8e5ce75195c7f59b8e48357593d1: Status 404 returned error can't find the container with id b9cdedd775b1a44c925047fe2e40810d3a8b8e5ce75195c7f59b8e48357593d1
Nov 02 03:20:06.803721 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:06.803696    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-2ktj7"]
Nov 02 03:20:06.805513 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:06.805472    4496 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 03:20:06.805587 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:06.805550    4496 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 03:20:06.805587 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:06.805578    4496 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 03:20:07.301877 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:07.301842    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-2ktj7" event={"ID":"a8d9e090-928d-489e-9eb4-82ad15bb6aa5","Type":"ContainerStarted","Data":"e1e157fedc734b8e972890434e5dbf002a597948de2ffecde03d9b19eb081831"}
Nov 02 03:20:07.302110 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:07.301907    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-2ktj7" event={"ID":"a8d9e090-928d-489e-9eb4-82ad15bb6aa5","Type":"ContainerStarted","Data":"b9cdedd775b1a44c925047fe2e40810d3a8b8e5ce75195c7f59b8e48357593d1"}
Nov 02 03:20:07.307071 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:07.303944    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0d26af-64f5446f47-qnwqh" event={"ID":"15d2020a-9cef-4fa1-b3f5-7d1f2f1a9bcf","Type":"ContainerStarted","Data":"2455831f6d1cd11a7a2862b497e58693a935bbb0e8755e7f9ffd9f77659b4988"}
Nov 02 03:20:07.307071 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:07.303998    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0d26af-64f5446f47-qnwqh" event={"ID":"15d2020a-9cef-4fa1-b3f5-7d1f2f1a9bcf","Type":"ContainerStarted","Data":"2d64d3df7d2c5f4608ccc032f276615247cad841848ef2e4775de003c295fcdc"}
Nov 02 03:20:07.307071 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:07.304086    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0d26af-64f5446f47-qnwqh"
Nov 02 03:20:07.307071 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:07.306047    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-0d26af-59668b6f65-dz2f7" event={"ID":"d3a1ea4e-eb3e-462b-900c-ea2181bb88d9","Type":"ContainerStarted","Data":"54673afff06f24dbb38b20f242cc2106831690ffc0c35691cf3c26f382abfefa"}
Nov 02 03:20:07.307071 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:07.306110    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-0d26af-59668b6f65-dz2f7" event={"ID":"d3a1ea4e-eb3e-462b-900c-ea2181bb88d9","Type":"ContainerStarted","Data":"34bc285a738bdbb8cd11ce1fce15f8ce10c34c9ce899b41f2f45ab21c2021b1e"}
Nov 02 03:20:07.307071 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:07.306135    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-0d26af-59668b6f65-dz2f7"
Nov 02 03:20:07.416044 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:07.415979    4496 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-2ktj7"
Nov 02 03:20:07.417840 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:07.417807    4496 patch_prober.go:29] interesting pod/e2e-worker-w3-858236-openshift-default-74fcd64bf5-2ktj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body=
Nov 02 03:20:07.417963 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:07.417876    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-2ktj7" podUID="a8d9e090-928d-489e-9eb4-82ad15bb6aa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused"
Nov 02 03:20:07.425354 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:07.425303    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-2ktj7" podStartSLOduration=1.425292449 podStartE2EDuration="1.425292449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:20:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:20:07.422762027 +0000 UTC m=+1948.001369420" watchObservedRunningTime="2026-11-02 03:20:07.425292449 +0000 UTC m=+1948.003899828"
Nov 02 03:20:07.540883 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:07.540833    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-0d26af-59668b6f65-dz2f7" podStartSLOduration=2.540820383 podStartE2EDuration="2.540820383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:20:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:20:07.539131467 +0000 UTC m=+1948.117738847" watchObservedRunningTime="2026-11-02 03:20:07.540820383 +0000 UTC m=+1948.119427774"
Nov 02 03:20:07.668791 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:07.667730    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0d26af-64f5446f47-qnwqh" podStartSLOduration=2.667709351 podStartE2EDuration="2.667709351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:20:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:20:07.656587112 +0000 UTC m=+1948.235194493" watchObservedRunningTime="2026-11-02 03:20:07.667709351 +0000 UTC m=+1948.246316748"
Nov 02 03:20:08.415189 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:08.415152    4496 patch_prober.go:29] interesting pod/e2e-worker-w3-858236-openshift-default-74fcd64bf5-2ktj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body=
Nov 02 03:20:08.415563 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:08.415216    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-2ktj7" podUID="a8d9e090-928d-489e-9eb4-82ad15bb6aa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused"
Nov 02 03:20:09.415879 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:09.415831    4496 patch_prober.go:29] interesting pod/e2e-worker-w3-858236-openshift-default-74fcd64bf5-2ktj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body=
Nov 02 03:20:09.416367 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:09.415897    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-2ktj7" podUID="a8d9e090-928d-489e-9eb4-82ad15bb6aa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused"
Nov 02 03:20:10.416120 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:10.416087    4496 patch_prober.go:29] interesting pod/e2e-worker-w3-858236-openshift-default-74fcd64bf5-2ktj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body=
Nov 02 03:20:10.416677 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:10.416132    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-2ktj7" podUID="a8d9e090-928d-489e-9eb4-82ad15bb6aa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused"
Nov 02 03:20:11.415206 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:11.415168    4496 patch_prober.go:29] interesting pod/e2e-worker-w3-858236-openshift-default-74fcd64bf5-2ktj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body=
Nov 02 03:20:11.415395 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:11.415227    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-2ktj7" podUID="a8d9e090-928d-489e-9eb4-82ad15bb6aa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused"
Nov 02 03:20:12.415162 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:12.415128    4496 patch_prober.go:29] interesting pod/e2e-worker-w3-858236-openshift-default-74fcd64bf5-2ktj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body=
Nov 02 03:20:12.415645 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:12.415187    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-2ktj7" podUID="a8d9e090-928d-489e-9eb4-82ad15bb6aa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused"
Nov 02 03:20:13.291042 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:13.285219    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-ed15c1-56c5d6557c-g5df2"
Nov 02 03:20:13.291503 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:13.291471    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ed15c1-57bddc8787-l7kbv"
Nov 02 03:20:13.417075 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:13.416174    4496 patch_prober.go:29] interesting pod/e2e-worker-w3-858236-openshift-default-74fcd64bf5-2ktj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body=
Nov 02 03:20:13.417075 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:13.416247    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-2ktj7" podUID="a8d9e090-928d-489e-9eb4-82ad15bb6aa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused"
Nov 02 03:20:14.418037 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:14.417930    4496 patch_prober.go:29] interesting pod/e2e-worker-w3-858236-openshift-default-74fcd64bf5-2ktj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body=
Nov 02 03:20:14.431041 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:14.418011    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-2ktj7" podUID="a8d9e090-928d-489e-9eb4-82ad15bb6aa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused"
Nov 02 03:20:15.564350 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:15.564317    4496 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-2ktj7"
Nov 02 03:20:15.704050 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:15.685548    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-2ktj7"
Nov 02 03:20:15.704556 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:15.704529    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-2ktj7"
Nov 02 03:20:16.126272 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:16.126226    4496 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-398828/distinct-2-w4-398828-kserve-545449d49b-6mg4b" podUID="36c1912c-2ab1-40c8-a09d-2425ab08ec8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.125:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 03:20:16.334794 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:16.334756    4496 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-277pb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 03:20:16.335001 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:16.334814    4496 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-277pb" podUID="3051f935-d87a-422e-9073-909b2c2768ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 03:20:18.319281 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:18.319251    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-0d26af-59668b6f65-dz2f7"
Nov 02 03:20:18.319720 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:18.319329    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0d26af-64f5446f47-qnwqh"
Nov 02 03:20:33.527333 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:20:33.527291    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 03:20:45.443299 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:45.443263    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b4a230-5666cf5b6c-vv8cx"]
Nov 02 03:20:45.449729 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:45.449704    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b4a230-5666cf5b6c-vv8cx"
Nov 02 03:20:45.508664 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:45.508624    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b4a230-5666cf5b6c-vv8cx"]
Nov 02 03:20:45.549779 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:45.549746    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-b4a230-dockercfg-hg54v\""
Nov 02 03:20:45.550112 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:45.550091    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-b4a230\""
Nov 02 03:20:45.558964 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:45.556979    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0dfde42a-42cb-482d-a667-6ee53e804104-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-b4a230-5666cf5b6c-vv8cx\" (UID: \"0dfde42a-42cb-482d-a667-6ee53e804104\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b4a230-5666cf5b6c-vv8cx"
Nov 02 03:20:45.558964 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:45.557063    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqn5r\" (UniqueName: \"kubernetes.io/projected/0dfde42a-42cb-482d-a667-6ee53e804104-kube-api-access-gqn5r\") pod \"payload-pre-processing-e2e-api-a-b4a230-5666cf5b6c-vv8cx\" (UID: \"0dfde42a-42cb-482d-a667-6ee53e804104\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b4a230-5666cf5b6c-vv8cx"
Nov 02 03:20:45.628984 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:45.628946    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b4a230-7f49b69f9f-jfcml"]
Nov 02 03:20:45.635519 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:45.635490    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-b4a230-7f49b69f9f-jfcml"
Nov 02 03:20:45.677638 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:45.677602    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b4a230-7f49b69f9f-jfcml"]
Nov 02 03:20:45.697729 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:45.697631    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54bac357-122d-43a3-909e-87f1cb110ffa-plugins-config-volume\") pod \"payload-processing-e2e-api-a-b4a230-7f49b69f9f-jfcml\" (UID: \"54bac357-122d-43a3-909e-87f1cb110ffa\") " pod="openshift-ingress/payload-processing-e2e-api-a-b4a230-7f49b69f9f-jfcml"
Nov 02 03:20:45.697963 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:45.697940    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqn5r\" (UniqueName: \"kubernetes.io/projected/0dfde42a-42cb-482d-a667-6ee53e804104-kube-api-access-gqn5r\") pod \"payload-pre-processing-e2e-api-a-b4a230-5666cf5b6c-vv8cx\" (UID: \"0dfde42a-42cb-482d-a667-6ee53e804104\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b4a230-5666cf5b6c-vv8cx"
Nov 02 03:20:45.698266 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:45.698246    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0dfde42a-42cb-482d-a667-6ee53e804104-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-b4a230-5666cf5b6c-vv8cx\" (UID: \"0dfde42a-42cb-482d-a667-6ee53e804104\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b4a230-5666cf5b6c-vv8cx"
Nov 02 03:20:45.698443 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:45.698426    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bx5qn\" (UniqueName: \"kubernetes.io/projected/54bac357-122d-43a3-909e-87f1cb110ffa-kube-api-access-bx5qn\") pod \"payload-processing-e2e-api-a-b4a230-7f49b69f9f-jfcml\" (UID: \"54bac357-122d-43a3-909e-87f1cb110ffa\") " pod="openshift-ingress/payload-processing-e2e-api-a-b4a230-7f49b69f9f-jfcml"
Nov 02 03:20:45.700373 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:45.700349    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0dfde42a-42cb-482d-a667-6ee53e804104-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-b4a230-5666cf5b6c-vv8cx\" (UID: \"0dfde42a-42cb-482d-a667-6ee53e804104\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b4a230-5666cf5b6c-vv8cx"
Nov 02 03:20:45.760124 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:45.760095    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqn5r\" (UniqueName: \"kubernetes.io/projected/0dfde42a-42cb-482d-a667-6ee53e804104-kube-api-access-gqn5r\") pod \"payload-pre-processing-e2e-api-a-b4a230-5666cf5b6c-vv8cx\" (UID: \"0dfde42a-42cb-482d-a667-6ee53e804104\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b4a230-5666cf5b6c-vv8cx"
Nov 02 03:20:45.767615 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:45.767588    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b4a230-5666cf5b6c-vv8cx"
Nov 02 03:20:45.800962 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:45.800917    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54bac357-122d-43a3-909e-87f1cb110ffa-plugins-config-volume\") pod \"payload-processing-e2e-api-a-b4a230-7f49b69f9f-jfcml\" (UID: \"54bac357-122d-43a3-909e-87f1cb110ffa\") " pod="openshift-ingress/payload-processing-e2e-api-a-b4a230-7f49b69f9f-jfcml"
Nov 02 03:20:45.801214 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:45.801192    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx5qn\" (UniqueName: \"kubernetes.io/projected/54bac357-122d-43a3-909e-87f1cb110ffa-kube-api-access-bx5qn\") pod \"payload-processing-e2e-api-a-b4a230-7f49b69f9f-jfcml\" (UID: \"54bac357-122d-43a3-909e-87f1cb110ffa\") " pod="openshift-ingress/payload-processing-e2e-api-a-b4a230-7f49b69f9f-jfcml"
Nov 02 03:20:45.801763 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:45.801741    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54bac357-122d-43a3-909e-87f1cb110ffa-plugins-config-volume\") pod \"payload-processing-e2e-api-a-b4a230-7f49b69f9f-jfcml\" (UID: \"54bac357-122d-43a3-909e-87f1cb110ffa\") " pod="openshift-ingress/payload-processing-e2e-api-a-b4a230-7f49b69f9f-jfcml"
Nov 02 03:20:45.861543 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:45.861504    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx5qn\" (UniqueName: \"kubernetes.io/projected/54bac357-122d-43a3-909e-87f1cb110ffa-kube-api-access-bx5qn\") pod \"payload-processing-e2e-api-a-b4a230-7f49b69f9f-jfcml\" (UID: \"54bac357-122d-43a3-909e-87f1cb110ffa\") " pod="openshift-ingress/payload-processing-e2e-api-a-b4a230-7f49b69f9f-jfcml"
Nov 02 03:20:45.969882 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:45.969791    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-b4a230-7f49b69f9f-jfcml"
Nov 02 03:20:46.146506 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:20:46.146476    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dfde42a_42cb_482d_a667_6ee53e804104.slice/crio-c6d04b6db58517e36537698b0fc6105b417bd13d0ca587de0f6e809ad422331e WatchSource:0}: Error finding container c6d04b6db58517e36537698b0fc6105b417bd13d0ca587de0f6e809ad422331e: Status 404 returned error can't find the container with id c6d04b6db58517e36537698b0fc6105b417bd13d0ca587de0f6e809ad422331e
Nov 02 03:20:46.148719 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:46.148693    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b4a230-5666cf5b6c-vv8cx"]
Nov 02 03:20:46.351900 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:20:46.351870    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54bac357_122d_43a3_909e_87f1cb110ffa.slice/crio-70c387a354e304b2b5fb8fd1a968dc47639fa28493df1140b0b413d8bfdb2864 WatchSource:0}: Error finding container 70c387a354e304b2b5fb8fd1a968dc47639fa28493df1140b0b413d8bfdb2864: Status 404 returned error can't find the container with id 70c387a354e304b2b5fb8fd1a968dc47639fa28493df1140b0b413d8bfdb2864
Nov 02 03:20:46.352962 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:46.352885    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b4a230-7f49b69f9f-jfcml"]
Nov 02 03:20:46.787271 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:46.787236    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-b4a230-7f49b69f9f-jfcml" event={"ID":"54bac357-122d-43a3-909e-87f1cb110ffa","Type":"ContainerStarted","Data":"8385a1531b982dfb9d00e307092cbae8bd514870b425a685ed91b288b1c062df"}
Nov 02 03:20:46.787733 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:46.787301    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-b4a230-7f49b69f9f-jfcml" event={"ID":"54bac357-122d-43a3-909e-87f1cb110ffa","Type":"ContainerStarted","Data":"70c387a354e304b2b5fb8fd1a968dc47639fa28493df1140b0b413d8bfdb2864"}
Nov 02 03:20:46.787733 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:46.787328    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-b4a230-7f49b69f9f-jfcml"
Nov 02 03:20:46.789209 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:46.789175    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b4a230-5666cf5b6c-vv8cx" event={"ID":"0dfde42a-42cb-482d-a667-6ee53e804104","Type":"ContainerStarted","Data":"471c15032821a5d41c47d97941e8bf0758308737148ba1e44c307b9d3bce5868"}
Nov 02 03:20:46.789320 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:46.789222    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b4a230-5666cf5b6c-vv8cx" event={"ID":"0dfde42a-42cb-482d-a667-6ee53e804104","Type":"ContainerStarted","Data":"c6d04b6db58517e36537698b0fc6105b417bd13d0ca587de0f6e809ad422331e"}
Nov 02 03:20:46.789320 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:46.789294    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b4a230-5666cf5b6c-vv8cx"
Nov 02 03:20:46.906051 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:46.905983    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-b4a230-7f49b69f9f-jfcml" podStartSLOduration=1.905969244 podStartE2EDuration="1.905969244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:20:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:20:46.902772575 +0000 UTC m=+1987.481379956" watchObservedRunningTime="2026-11-02 03:20:46.905969244 +0000 UTC m=+1987.484576617"
Nov 02 03:20:47.024331 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:47.023092    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b4a230-5666cf5b6c-vv8cx" podStartSLOduration=2.023071246 podStartE2EDuration="2.023071246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:20:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:20:47.018276761 +0000 UTC m=+1987.596884144" watchObservedRunningTime="2026-11-02 03:20:47.023071246 +0000 UTC m=+1987.601678658"
Nov 02 03:20:53.049086 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:53.049050    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5130b62f-5lzt2"]
Nov 02 03:20:53.100486 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:53.100450    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5130b62f-5lzt2"]
Nov 02 03:20:53.576485 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:53.576450    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0b5742c-f6a7-4fa5-8fa6-e7aaa32175bb" path="/var/lib/kubelet/pods/f0b5742c-f6a7-4fa5-8fa6-e7aaa32175bb/volumes"
Nov 02 03:20:57.801773 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:57.801739    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-b4a230-7f49b69f9f-jfcml"
Nov 02 03:20:57.802301 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:20:57.801847    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b4a230-5666cf5b6c-vv8cx"
Nov 02 03:21:03.526819 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:21:03.526781    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 03:21:07.051207 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:07.051171    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a20dfb7-7s952"]
Nov 02 03:21:07.108116 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:07.108075    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a40aafd9-8rzqh"]
Nov 02 03:21:07.113093 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:07.113062    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a20dfb7-7s952"]
Nov 02 03:21:07.119962 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:07.119934    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a40aafd9-8rzqh"]
Nov 02 03:21:07.575169 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:07.575132    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a16f45f-6782-4818-b31d-a341e55d8017" path="/var/lib/kubelet/pods/4a16f45f-6782-4818-b31d-a341e55d8017/volumes"
Nov 02 03:21:07.575723 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:07.575705    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b51ad1e-c78a-44b8-bad0-870b9e5ec1bf" path="/var/lib/kubelet/pods/6b51ad1e-c78a-44b8-bad0-870b9e5ec1bf/volumes"
Nov 02 03:21:23.050525 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:23.050449    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4209a816-mtqkz"]
Nov 02 03:21:23.100836 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:23.100795    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a22e3d1e-vfjwn"]
Nov 02 03:21:23.106198 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:23.106175    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4209a816-mtqkz"]
Nov 02 03:21:23.110629 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:23.110608    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a22e3d1e-vfjwn"]
Nov 02 03:21:23.577803 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:23.577766    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a19741d-1644-42f2-a560-b14b5de2dcf9" path="/var/lib/kubelet/pods/2a19741d-1644-42f2-a560-b14b5de2dcf9/volumes"
Nov 02 03:21:23.578442 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:23.578425    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bb06cf0-f055-46d1-81f2-81aea3dc5cdd" path="/var/lib/kubelet/pods/4bb06cf0-f055-46d1-81f2-81aea3dc5cdd/volumes"
Nov 02 03:21:31.952230 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:31.952192    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11256-3318f5"]
Nov 02 03:21:31.972347 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:31.972301    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11256-3318f5"
Nov 02 03:21:32.025598 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:32.025572    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-t4r8t\""
Nov 02 03:21:32.067200 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:32.067150    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86lzh\" (UniqueName: \"kubernetes.io/projected/d37d94d2-76c3-4420-a697-70131ba1706a-kube-api-access-86lzh\") pod \"test-curl-11256-3318f5\" (UID: \"d37d94d2-76c3-4420-a697-70131ba1706a\") " pod="opendatahub/test-curl-11256-3318f5"
Nov 02 03:21:32.083188 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:32.083155    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11256-3318f5"]
Nov 02 03:21:32.177039 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:32.175487    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86lzh\" (UniqueName: \"kubernetes.io/projected/d37d94d2-76c3-4420-a697-70131ba1706a-kube-api-access-86lzh\") pod \"test-curl-11256-3318f5\" (UID: \"d37d94d2-76c3-4420-a697-70131ba1706a\") " pod="opendatahub/test-curl-11256-3318f5"
Nov 02 03:21:32.233155 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:32.233122    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86lzh\" (UniqueName: \"kubernetes.io/projected/d37d94d2-76c3-4420-a697-70131ba1706a-kube-api-access-86lzh\") pod \"test-curl-11256-3318f5\" (UID: \"d37d94d2-76c3-4420-a697-70131ba1706a\") " pod="opendatahub/test-curl-11256-3318f5"
Nov 02 03:21:32.288438 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:32.288408    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11256-3318f5"
Nov 02 03:21:32.676906 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:21:32.676873    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd37d94d2_76c3_4420_a697_70131ba1706a.slice/crio-188317eb35370a200f62f51e9ac057c47af35cdda6b46036422aecbab882078a WatchSource:0}: Error finding container 188317eb35370a200f62f51e9ac057c47af35cdda6b46036422aecbab882078a: Status 404 returned error can't find the container with id 188317eb35370a200f62f51e9ac057c47af35cdda6b46036422aecbab882078a
Nov 02 03:21:32.678369 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:32.678337    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11256-3318f5"]
Nov 02 03:21:33.078046 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:33.074160    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d71d8e5-rscj2"]
Nov 02 03:21:33.087048 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:33.082347    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d71d8e5-rscj2"]
Nov 02 03:21:33.284233 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:33.284193    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11256-3318f5" event={"ID":"d37d94d2-76c3-4420-a697-70131ba1706a","Type":"ContainerStarted","Data":"188317eb35370a200f62f51e9ac057c47af35cdda6b46036422aecbab882078a"}
Nov 02 03:21:33.528942 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:21:33.528905    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 03:21:33.576126 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:33.576098    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71f8d674-60a8-4b1f-85c7-80f97b47623e" path="/var/lib/kubelet/pods/71f8d674-60a8-4b1f-85c7-80f97b47623e/volumes"
Nov 02 03:21:36.323704 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:36.323647    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11256-3318f5" event={"ID":"d37d94d2-76c3-4420-a697-70131ba1706a","Type":"ContainerStarted","Data":"160898e9290952bef792b393765cf146d533802536a2a77a9c3d0bed640228ab"}
Nov 02 03:21:36.442391 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:36.442337    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11256-3318f5" podStartSLOduration=2.350220901 podStartE2EDuration="5.442322344s" totalImagesPullingTime="3.092101443s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:21:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:21:36.439185991 +0000 UTC m=+2037.017793372" watchObservedRunningTime="2026-11-02 03:21:36.442322344 +0000 UTC m=+2037.020929723"
Nov 02 03:21:37.949135 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:37.949098    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11256-3318f5"]
Nov 02 03:21:37.965186 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:37.965140    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11256-3318f5"]
Nov 02 03:21:38.340400 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:38.340357    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11256-3318f5" podUID="d37d94d2-76c3-4420-a697-70131ba1706a" containerName="test-curl-11256-3318f5" containerID="cri-o://160898e9290952bef792b393765cf146d533802536a2a77a9c3d0bed640228ab" gracePeriod=2
Nov 02 03:21:38.392301 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:21:38.392262    4496 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11256-3318f5\" is forbidden: User \"system:node:ip-10-0-133-163.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-163.us-west-1.compute.internal' and this object" podUID="d37d94d2-76c3-4420-a697-70131ba1706a" pod="opendatahub/test-curl-11256-3318f5"
Nov 02 03:21:39.055197 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:39.055161    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94a98249-75mhd"]
Nov 02 03:21:39.101112 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:39.101070    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94a98249-75mhd"]
Nov 02 03:21:39.515409 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:39.515366    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d6544b-75f96f97f-6m8n2"]
Nov 02 03:21:39.516446 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:39.516418    4496 cpu_manager.go:414] "containerMap: removing container" podUID="d37d94d2-76c3-4420-a697-70131ba1706a" containerName="test-curl-11256-3318f5"
Nov 02 03:21:39.516588 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:39.516451    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d37d94d2-76c3-4420-a697-70131ba1706a" containerName="test-curl-11256-3318f5"
Nov 02 03:21:39.516702 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:39.516687    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="d37d94d2-76c3-4420-a697-70131ba1706a" containerName="test-curl-11256-3318f5"
Nov 02 03:21:39.520858 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:39.520838    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d6544b-75f96f97f-6m8n2"
Nov 02 03:21:39.579278 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:39.577132    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-d6544b-dockercfg-nvblh\""
Nov 02 03:21:39.579278 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:21:39.577127    4496 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11256-3318f5\" is forbidden: User \"system:node:ip-10-0-133-163.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-163.us-west-1.compute.internal' and this object" podUID="d37d94d2-76c3-4420-a697-70131ba1706a" pod="opendatahub/test-curl-11256-3318f5"
Nov 02 03:21:39.579278 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:39.577254    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-d6544b\""
Nov 02 03:21:39.594047 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:39.593944    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3261337-656c-4a60-8aa9-46d16545bb2e" path="/var/lib/kubelet/pods/f3261337-656c-4a60-8aa9-46d16545bb2e/volumes"
Nov 02 03:21:39.594755 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:39.594728    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d6544b-75f96f97f-6m8n2"]
Nov 02 03:21:39.629917 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:39.629858    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed8d5844-0ec4-4050-959c-f8864624ebd5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-d6544b-75f96f97f-6m8n2\" (UID: \"ed8d5844-0ec4-4050-959c-f8864624ebd5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d6544b-75f96f97f-6m8n2"
Nov 02 03:21:39.630298 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:39.630273    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czmvp\" (UniqueName: \"kubernetes.io/projected/ed8d5844-0ec4-4050-959c-f8864624ebd5-kube-api-access-czmvp\") pod \"payload-pre-processing-e2e-api-b-d6544b-75f96f97f-6m8n2\" (UID: \"ed8d5844-0ec4-4050-959c-f8864624ebd5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d6544b-75f96f97f-6m8n2"
Nov 02 03:21:39.682287 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:39.682252    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d6544b-5fc969d5f9-7k5sq"]
Nov 02 03:21:39.752076 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:39.744130    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czmvp\" (UniqueName: \"kubernetes.io/projected/ed8d5844-0ec4-4050-959c-f8864624ebd5-kube-api-access-czmvp\") pod \"payload-pre-processing-e2e-api-b-d6544b-75f96f97f-6m8n2\" (UID: \"ed8d5844-0ec4-4050-959c-f8864624ebd5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d6544b-75f96f97f-6m8n2"
Nov 02 03:21:39.752076 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:39.744200    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed8d5844-0ec4-4050-959c-f8864624ebd5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-d6544b-75f96f97f-6m8n2\" (UID: \"ed8d5844-0ec4-4050-959c-f8864624ebd5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d6544b-75f96f97f-6m8n2"
Nov 02 03:21:39.752076 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:39.745251    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed8d5844-0ec4-4050-959c-f8864624ebd5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-d6544b-75f96f97f-6m8n2\" (UID: \"ed8d5844-0ec4-4050-959c-f8864624ebd5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d6544b-75f96f97f-6m8n2"
Nov 02 03:21:39.794530 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:39.792097    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d6544b-5fc969d5f9-7k5sq"
Nov 02 03:21:39.800123 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:39.799477    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d6544b-5fc969d5f9-7k5sq"]
Nov 02 03:21:39.815996 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:39.813886    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czmvp\" (UniqueName: \"kubernetes.io/projected/ed8d5844-0ec4-4050-959c-f8864624ebd5-kube-api-access-czmvp\") pod \"payload-pre-processing-e2e-api-b-d6544b-75f96f97f-6m8n2\" (UID: \"ed8d5844-0ec4-4050-959c-f8864624ebd5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d6544b-75f96f97f-6m8n2"
Nov 02 03:21:39.839699 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:39.839032    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d6544b-75f96f97f-6m8n2"
Nov 02 03:21:39.979744 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:39.979702    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25ab21e1-7f9b-4017-86bd-b3e45bb4d823-plugins-config-volume\") pod \"payload-processing-e2e-api-b-d6544b-5fc969d5f9-7k5sq\" (UID: \"25ab21e1-7f9b-4017-86bd-b3e45bb4d823\") " pod="openshift-ingress/payload-processing-e2e-api-b-d6544b-5fc969d5f9-7k5sq"
Nov 02 03:21:39.979916 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:39.979768    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brh4g\" (UniqueName: \"kubernetes.io/projected/25ab21e1-7f9b-4017-86bd-b3e45bb4d823-kube-api-access-brh4g\") pod \"payload-processing-e2e-api-b-d6544b-5fc969d5f9-7k5sq\" (UID: \"25ab21e1-7f9b-4017-86bd-b3e45bb4d823\") " pod="openshift-ingress/payload-processing-e2e-api-b-d6544b-5fc969d5f9-7k5sq"
Nov 02 03:21:40.082246 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:40.082125    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25ab21e1-7f9b-4017-86bd-b3e45bb4d823-plugins-config-volume\") pod \"payload-processing-e2e-api-b-d6544b-5fc969d5f9-7k5sq\" (UID: \"25ab21e1-7f9b-4017-86bd-b3e45bb4d823\") " pod="openshift-ingress/payload-processing-e2e-api-b-d6544b-5fc969d5f9-7k5sq"
Nov 02 03:21:40.082246 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:40.082183    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brh4g\" (UniqueName: \"kubernetes.io/projected/25ab21e1-7f9b-4017-86bd-b3e45bb4d823-kube-api-access-brh4g\") pod \"payload-processing-e2e-api-b-d6544b-5fc969d5f9-7k5sq\" (UID: \"25ab21e1-7f9b-4017-86bd-b3e45bb4d823\") " pod="openshift-ingress/payload-processing-e2e-api-b-d6544b-5fc969d5f9-7k5sq"
Nov 02 03:21:40.083336 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:40.083311    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25ab21e1-7f9b-4017-86bd-b3e45bb4d823-plugins-config-volume\") pod \"payload-processing-e2e-api-b-d6544b-5fc969d5f9-7k5sq\" (UID: \"25ab21e1-7f9b-4017-86bd-b3e45bb4d823\") " pod="openshift-ingress/payload-processing-e2e-api-b-d6544b-5fc969d5f9-7k5sq"
Nov 02 03:21:40.132837 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:40.132804    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d5dcca72-4pgb9"]
Nov 02 03:21:40.139737 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:40.139712    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d5dcca72-4pgb9"
Nov 02 03:21:40.140138 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:40.140107    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brh4g\" (UniqueName: \"kubernetes.io/projected/25ab21e1-7f9b-4017-86bd-b3e45bb4d823-kube-api-access-brh4g\") pod \"payload-processing-e2e-api-b-d6544b-5fc969d5f9-7k5sq\" (UID: \"25ab21e1-7f9b-4017-86bd-b3e45bb4d823\") " pod="openshift-ingress/payload-processing-e2e-api-b-d6544b-5fc969d5f9-7k5sq"
Nov 02 03:21:40.192821 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:40.192798    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 03:21:40.192821 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:40.192805    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 03:21:40.193895 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:40.193868    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d5dcca72-4pgb9"]
Nov 02 03:21:40.259408 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:40.259380    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 03:21:40.259536 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:40.259423    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9z5bk\""
Nov 02 03:21:40.285169 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:40.285136    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kjrz\" (UniqueName: \"kubernetes.io/projected/3fa39fec-bd1c-400e-a2b4-3b072c5bac77-kube-api-access-8kjrz\") pod \"maas-api-revoke-keys-e2e-mt-d5dcca72-4pgb9\" (UID: \"3fa39fec-bd1c-400e-a2b4-3b072c5bac77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d5dcca72-4pgb9"
Nov 02 03:21:40.285299 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:40.285263    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fa39fec-bd1c-400e-a2b4-3b072c5bac77-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-d5dcca72-4pgb9\" (UID: \"3fa39fec-bd1c-400e-a2b4-3b072c5bac77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d5dcca72-4pgb9"
Nov 02 03:21:40.285359 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:40.285344    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fa39fec-bd1c-400e-a2b4-3b072c5bac77-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-d5dcca72-4pgb9\" (UID: \"3fa39fec-bd1c-400e-a2b4-3b072c5bac77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d5dcca72-4pgb9"
Nov 02 03:21:40.301559 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:40.301538    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d6544b-75f96f97f-6m8n2"]
Nov 02 03:21:40.362612 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:40.362586    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d6544b-75f96f97f-6m8n2" event={"ID":"ed8d5844-0ec4-4050-959c-f8864624ebd5","Type":"ContainerStarted","Data":"a565b0ad63ce42b490f8c6811ebc5e50342c640db9549dfcd190d9c69dd8f7fd"}
Nov 02 03:21:40.386965 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:40.386935    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kjrz\" (UniqueName: \"kubernetes.io/projected/3fa39fec-bd1c-400e-a2b4-3b072c5bac77-kube-api-access-8kjrz\") pod \"maas-api-revoke-keys-e2e-mt-d5dcca72-4pgb9\" (UID: \"3fa39fec-bd1c-400e-a2b4-3b072c5bac77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d5dcca72-4pgb9"
Nov 02 03:21:40.387102 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:40.387052    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fa39fec-bd1c-400e-a2b4-3b072c5bac77-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-d5dcca72-4pgb9\" (UID: \"3fa39fec-bd1c-400e-a2b4-3b072c5bac77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d5dcca72-4pgb9"
Nov 02 03:21:40.387186 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:40.387169    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fa39fec-bd1c-400e-a2b4-3b072c5bac77-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-d5dcca72-4pgb9\" (UID: \"3fa39fec-bd1c-400e-a2b4-3b072c5bac77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d5dcca72-4pgb9"
Nov 02 03:21:40.388212 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:40.387946    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fa39fec-bd1c-400e-a2b4-3b072c5bac77-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-d5dcca72-4pgb9\" (UID: \"3fa39fec-bd1c-400e-a2b4-3b072c5bac77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d5dcca72-4pgb9"
Nov 02 03:21:40.389491 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:40.389458    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fa39fec-bd1c-400e-a2b4-3b072c5bac77-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-d5dcca72-4pgb9\" (UID: \"3fa39fec-bd1c-400e-a2b4-3b072c5bac77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d5dcca72-4pgb9"
Nov 02 03:21:40.423847 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:40.423819    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d6544b-5fc969d5f9-7k5sq"
Nov 02 03:21:40.446234 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:40.444952    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kjrz\" (UniqueName: \"kubernetes.io/projected/3fa39fec-bd1c-400e-a2b4-3b072c5bac77-kube-api-access-8kjrz\") pod \"maas-api-revoke-keys-e2e-mt-d5dcca72-4pgb9\" (UID: \"3fa39fec-bd1c-400e-a2b4-3b072c5bac77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d5dcca72-4pgb9"
Nov 02 03:21:40.477043 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:40.475874    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d5dcca72-4pgb9"
Nov 02 03:21:40.659063 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:40.659001    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11256-3318f5"
Nov 02 03:21:40.710913 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:21:40.710877    4496 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11256-3318f5\" is forbidden: User \"system:node:ip-10-0-133-163.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-163.us-west-1.compute.internal' and this object" podUID="d37d94d2-76c3-4420-a697-70131ba1706a" pod="opendatahub/test-curl-11256-3318f5"
Nov 02 03:21:40.795155 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:40.795112    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d37d94d2-76c3-4420-a697-70131ba1706a-kube-api-access-86lzh\" (UniqueName: \"kubernetes.io/projected/d37d94d2-76c3-4420-a697-70131ba1706a-kube-api-access-86lzh\") pod \"d37d94d2-76c3-4420-a697-70131ba1706a\" (UID: \"d37d94d2-76c3-4420-a697-70131ba1706a\") "
Nov 02 03:21:40.797188 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:40.797162    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d37d94d2-76c3-4420-a697-70131ba1706a-kube-api-access-86lzh" pod "d37d94d2-76c3-4420-a697-70131ba1706a" (UID: "d37d94d2-76c3-4420-a697-70131ba1706a"). InnerVolumeSpecName "kube-api-access-86lzh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:21:40.820713 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:21:40.820683    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25ab21e1_7f9b_4017_86bd_b3e45bb4d823.slice/crio-d7bb8d60bd9ed8be60488fcd8f0d660ea53bbe320e1dae547d5342eafe836f00 WatchSource:0}: Error finding container d7bb8d60bd9ed8be60488fcd8f0d660ea53bbe320e1dae547d5342eafe836f00: Status 404 returned error can't find the container with id d7bb8d60bd9ed8be60488fcd8f0d660ea53bbe320e1dae547d5342eafe836f00
Nov 02 03:21:40.822279 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:40.822250    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d6544b-5fc969d5f9-7k5sq"]
Nov 02 03:21:40.897367 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:40.897295    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86lzh\" (UniqueName: \"kubernetes.io/projected/d37d94d2-76c3-4420-a697-70131ba1706a-kube-api-access-86lzh\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:21:40.928548 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:21:40.928518    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fa39fec_bd1c_400e_a2b4_3b072c5bac77.slice/crio-5f2b008c50e8b25971e28ecab7a0385600a6802ec7505266bf1918e9b750a00a WatchSource:0}: Error finding container 5f2b008c50e8b25971e28ecab7a0385600a6802ec7505266bf1918e9b750a00a: Status 404 returned error can't find the container with id 5f2b008c50e8b25971e28ecab7a0385600a6802ec7505266bf1918e9b750a00a
Nov 02 03:21:40.932327 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:40.932295    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d5dcca72-4pgb9"]
Nov 02 03:21:41.374477 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:41.373778    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d6544b-75f96f97f-6m8n2" event={"ID":"ed8d5844-0ec4-4050-959c-f8864624ebd5","Type":"ContainerStarted","Data":"78ce13fdaaf013b61afcd17ecad524a5dcb9e83ed635961327ee0e48bc138abe"}
Nov 02 03:21:41.374477 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:41.373872    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d6544b-75f96f97f-6m8n2"
Nov 02 03:21:41.377203 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:41.377174    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d6544b-5fc969d5f9-7k5sq" event={"ID":"25ab21e1-7f9b-4017-86bd-b3e45bb4d823","Type":"ContainerStarted","Data":"691b36370c3cb7c848f814bd4e66be5e6018279c746d0ce079e5140ba7caee7c"}
Nov 02 03:21:41.377334 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:41.377230    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d6544b-5fc969d5f9-7k5sq" event={"ID":"25ab21e1-7f9b-4017-86bd-b3e45bb4d823","Type":"ContainerStarted","Data":"d7bb8d60bd9ed8be60488fcd8f0d660ea53bbe320e1dae547d5342eafe836f00"}
Nov 02 03:21:41.377334 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:41.377278    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-d6544b-5fc969d5f9-7k5sq"
Nov 02 03:21:41.378940 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:41.378912    4496 generic.go:356] "Generic (PLEG): container finished" podID="d37d94d2-76c3-4420-a697-70131ba1706a" containerID="160898e9290952bef792b393765cf146d533802536a2a77a9c3d0bed640228ab" exitCode=137
Nov 02 03:21:41.379103 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:41.379086    4496 scope.go:122] "RemoveContainer" containerID="160898e9290952bef792b393765cf146d533802536a2a77a9c3d0bed640228ab"
Nov 02 03:21:41.379182 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:41.379113    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11256-3318f5"
Nov 02 03:21:41.382061 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:41.381768    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d5dcca72-4pgb9" event={"ID":"3fa39fec-bd1c-400e-a2b4-3b072c5bac77","Type":"ContainerStarted","Data":"5f2b008c50e8b25971e28ecab7a0385600a6802ec7505266bf1918e9b750a00a"}
Nov 02 03:21:41.408461 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:41.408434    4496 scope.go:122] "RemoveContainer" containerID="160898e9290952bef792b393765cf146d533802536a2a77a9c3d0bed640228ab"
Nov 02 03:21:41.408820 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:21:41.408797    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"160898e9290952bef792b393765cf146d533802536a2a77a9c3d0bed640228ab\": container with ID starting with 160898e9290952bef792b393765cf146d533802536a2a77a9c3d0bed640228ab not found: ID does not exist" containerID="160898e9290952bef792b393765cf146d533802536a2a77a9c3d0bed640228ab"
Nov 02 03:21:41.408900 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:41.408827    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"160898e9290952bef792b393765cf146d533802536a2a77a9c3d0bed640228ab"} err="failed to get container status \"160898e9290952bef792b393765cf146d533802536a2a77a9c3d0bed640228ab\": rpc error: code = NotFound desc = could not find container \"160898e9290952bef792b393765cf146d533802536a2a77a9c3d0bed640228ab\": container with ID starting with 160898e9290952bef792b393765cf146d533802536a2a77a9c3d0bed640228ab not found: ID does not exist"
Nov 02 03:21:41.425992 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:21:41.425958    4496 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11256-3318f5\" is forbidden: User \"system:node:ip-10-0-133-163.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-163.us-west-1.compute.internal' and this object" podUID="d37d94d2-76c3-4420-a697-70131ba1706a" pod="opendatahub/test-curl-11256-3318f5"
Nov 02 03:21:41.543773 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:41.543715    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d6544b-75f96f97f-6m8n2" podStartSLOduration=2.543700491 podStartE2EDuration="2.543700491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:21:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:21:41.541633701 +0000 UTC m=+2042.120241082" watchObservedRunningTime="2026-11-02 03:21:41.543700491 +0000 UTC m=+2042.122307871"
Nov 02 03:21:41.578174 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:41.578145    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d37d94d2-76c3-4420-a697-70131ba1706a" path="/var/lib/kubelet/pods/d37d94d2-76c3-4420-a697-70131ba1706a/volumes"
Nov 02 03:21:41.724488 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:21:41.724397    4496 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11256-3318f5\" is forbidden: User \"system:node:ip-10-0-133-163.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-163.us-west-1.compute.internal' and this object" podUID="d37d94d2-76c3-4420-a697-70131ba1706a" pod="opendatahub/test-curl-11256-3318f5"
Nov 02 03:21:42.394229 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:42.394133    4496 generic.go:356] "Generic (PLEG): container finished" podID="3fa39fec-bd1c-400e-a2b4-3b072c5bac77" containerID="f6efcf46d08e8aa21bfd3e6a3aa4e7e581de9bc152aac5619021018109eed5a4" exitCode=0
Nov 02 03:21:42.394620 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:42.394222    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d5dcca72-4pgb9" event={"ID":"3fa39fec-bd1c-400e-a2b4-3b072c5bac77","Type":"ContainerDied","Data":"f6efcf46d08e8aa21bfd3e6a3aa4e7e581de9bc152aac5619021018109eed5a4"}
Nov 02 03:21:42.509236 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:42.509189    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-d6544b-5fc969d5f9-7k5sq" podStartSLOduration=3.509174951 podStartE2EDuration="3.509174951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:21:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:21:41.672526658 +0000 UTC m=+2042.251134038" watchObservedRunningTime="2026-11-02 03:21:42.509174951 +0000 UTC m=+2043.087782335"
Nov 02 03:21:43.607168 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:43.607141    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d5dcca72-4pgb9"
Nov 02 03:21:43.737198 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:43.737165    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fa39fec-bd1c-400e-a2b4-3b072c5bac77-kube-api-access-8kjrz\" (UniqueName: \"kubernetes.io/projected/3fa39fec-bd1c-400e-a2b4-3b072c5bac77-kube-api-access-8kjrz\") pod \"3fa39fec-bd1c-400e-a2b4-3b072c5bac77\" (UID: \"3fa39fec-bd1c-400e-a2b4-3b072c5bac77\") "
Nov 02 03:21:43.737397 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:43.737292    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fa39fec-bd1c-400e-a2b4-3b072c5bac77-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fa39fec-bd1c-400e-a2b4-3b072c5bac77-maas-api-openshift-ca\") pod \"3fa39fec-bd1c-400e-a2b4-3b072c5bac77\" (UID: \"3fa39fec-bd1c-400e-a2b4-3b072c5bac77\") "
Nov 02 03:21:43.737397 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:43.737350    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3fa39fec-bd1c-400e-a2b4-3b072c5bac77-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fa39fec-bd1c-400e-a2b4-3b072c5bac77-maas-api-xks-ca\") pod \"3fa39fec-bd1c-400e-a2b4-3b072c5bac77\" (UID: \"3fa39fec-bd1c-400e-a2b4-3b072c5bac77\") "
Nov 02 03:21:43.737716 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:43.737690    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fa39fec-bd1c-400e-a2b4-3b072c5bac77-maas-api-openshift-ca" pod "3fa39fec-bd1c-400e-a2b4-3b072c5bac77" (UID: "3fa39fec-bd1c-400e-a2b4-3b072c5bac77"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:21:43.737947 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:43.737928    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fa39fec-bd1c-400e-a2b4-3b072c5bac77-maas-api-openshift-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:21:43.739477 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:43.739447    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fa39fec-bd1c-400e-a2b4-3b072c5bac77-kube-api-access-8kjrz" pod "3fa39fec-bd1c-400e-a2b4-3b072c5bac77" (UID: "3fa39fec-bd1c-400e-a2b4-3b072c5bac77"). InnerVolumeSpecName "kube-api-access-8kjrz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:21:43.739648 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:43.739629    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fa39fec-bd1c-400e-a2b4-3b072c5bac77-maas-api-xks-ca" pod "3fa39fec-bd1c-400e-a2b4-3b072c5bac77" (UID: "3fa39fec-bd1c-400e-a2b4-3b072c5bac77"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 03:21:43.839234 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:43.839200    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kjrz\" (UniqueName: \"kubernetes.io/projected/3fa39fec-bd1c-400e-a2b4-3b072c5bac77-kube-api-access-8kjrz\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:21:43.839234 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:43.839228    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fa39fec-bd1c-400e-a2b4-3b072c5bac77-maas-api-xks-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:21:44.413854 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:44.413823    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d5dcca72-4pgb9"
Nov 02 03:21:44.413854 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:44.413844    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d5dcca72-4pgb9" event={"ID":"3fa39fec-bd1c-400e-a2b4-3b072c5bac77","Type":"ContainerDied","Data":"5f2b008c50e8b25971e28ecab7a0385600a6802ec7505266bf1918e9b750a00a"}
Nov 02 03:21:44.414104 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:44.413895    4496 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f2b008c50e8b25971e28ecab7a0385600a6802ec7505266bf1918e9b750a00a"
Nov 02 03:21:45.550827 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:45.550790    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c9d1fc-5b9c6db48c-lkdss"]
Nov 02 03:21:45.554270 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:45.554243    4496 cpu_manager.go:414] "containerMap: removing container" podUID="3fa39fec-bd1c-400e-a2b4-3b072c5bac77" containerName="revoke-keys"
Nov 02 03:21:45.554459 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:45.554438    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fa39fec-bd1c-400e-a2b4-3b072c5bac77" containerName="revoke-keys"
Nov 02 03:21:45.554677 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:45.554663    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fa39fec-bd1c-400e-a2b4-3b072c5bac77" containerName="revoke-keys"
Nov 02 03:21:45.573350 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:45.573321    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c9d1fc-5b9c6db48c-lkdss"
Nov 02 03:21:45.576476 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:45.576411    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rw8k\" (UniqueName: \"kubernetes.io/projected/2d399530-92b8-40ea-b7d0-05e30081f5f2-kube-api-access-9rw8k\") pod \"payload-pre-processing-e2e-api-a-c9d1fc-5b9c6db48c-lkdss\" (UID: \"2d399530-92b8-40ea-b7d0-05e30081f5f2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c9d1fc-5b9c6db48c-lkdss"
Nov 02 03:21:45.579133 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:45.579103    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d399530-92b8-40ea-b7d0-05e30081f5f2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-c9d1fc-5b9c6db48c-lkdss\" (UID: \"2d399530-92b8-40ea-b7d0-05e30081f5f2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c9d1fc-5b9c6db48c-lkdss"
Nov 02 03:21:45.615661 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:45.615050    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c9d1fc-5b9c6db48c-lkdss"]
Nov 02 03:21:45.653037 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:45.652317    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-c9d1fc\""
Nov 02 03:21:45.653037 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:45.652458    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-c9d1fc-dockercfg-ljkls\""
Nov 02 03:21:45.685974 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:45.685701    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rw8k\" (UniqueName: \"kubernetes.io/projected/2d399530-92b8-40ea-b7d0-05e30081f5f2-kube-api-access-9rw8k\") pod \"payload-pre-processing-e2e-api-a-c9d1fc-5b9c6db48c-lkdss\" (UID: \"2d399530-92b8-40ea-b7d0-05e30081f5f2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c9d1fc-5b9c6db48c-lkdss"
Nov 02 03:21:45.685974 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:45.685829    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d399530-92b8-40ea-b7d0-05e30081f5f2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-c9d1fc-5b9c6db48c-lkdss\" (UID: \"2d399530-92b8-40ea-b7d0-05e30081f5f2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c9d1fc-5b9c6db48c-lkdss"
Nov 02 03:21:45.687961 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:45.687914    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d399530-92b8-40ea-b7d0-05e30081f5f2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-c9d1fc-5b9c6db48c-lkdss\" (UID: \"2d399530-92b8-40ea-b7d0-05e30081f5f2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c9d1fc-5b9c6db48c-lkdss"
Nov 02 03:21:45.708830 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:45.708794    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c9d1fc-ffdc6cb67-6nwgq"]
Nov 02 03:21:45.714273 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:45.714249    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c9d1fc-ffdc6cb67-6nwgq"
Nov 02 03:21:45.745259 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:45.745228    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rw8k\" (UniqueName: \"kubernetes.io/projected/2d399530-92b8-40ea-b7d0-05e30081f5f2-kube-api-access-9rw8k\") pod \"payload-pre-processing-e2e-api-a-c9d1fc-5b9c6db48c-lkdss\" (UID: \"2d399530-92b8-40ea-b7d0-05e30081f5f2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c9d1fc-5b9c6db48c-lkdss"
Nov 02 03:21:45.801136 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:45.801050    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6spbm\" (UniqueName: \"kubernetes.io/projected/2371a226-dcd2-49f3-a966-41d7da7cafa8-kube-api-access-6spbm\") pod \"payload-processing-e2e-api-a-c9d1fc-ffdc6cb67-6nwgq\" (UID: \"2371a226-dcd2-49f3-a966-41d7da7cafa8\") " pod="openshift-ingress/payload-processing-e2e-api-a-c9d1fc-ffdc6cb67-6nwgq"
Nov 02 03:21:45.801312 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:45.801225    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2371a226-dcd2-49f3-a966-41d7da7cafa8-plugins-config-volume\") pod \"payload-processing-e2e-api-a-c9d1fc-ffdc6cb67-6nwgq\" (UID: \"2371a226-dcd2-49f3-a966-41d7da7cafa8\") " pod="openshift-ingress/payload-processing-e2e-api-a-c9d1fc-ffdc6cb67-6nwgq"
Nov 02 03:21:45.801734 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:45.801708    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c9d1fc-ffdc6cb67-6nwgq"]
Nov 02 03:21:45.904085 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:45.902529    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6spbm\" (UniqueName: \"kubernetes.io/projected/2371a226-dcd2-49f3-a966-41d7da7cafa8-kube-api-access-6spbm\") pod \"payload-processing-e2e-api-a-c9d1fc-ffdc6cb67-6nwgq\" (UID: \"2371a226-dcd2-49f3-a966-41d7da7cafa8\") " pod="openshift-ingress/payload-processing-e2e-api-a-c9d1fc-ffdc6cb67-6nwgq"
Nov 02 03:21:45.904085 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:45.902724    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2371a226-dcd2-49f3-a966-41d7da7cafa8-plugins-config-volume\") pod \"payload-processing-e2e-api-a-c9d1fc-ffdc6cb67-6nwgq\" (UID: \"2371a226-dcd2-49f3-a966-41d7da7cafa8\") " pod="openshift-ingress/payload-processing-e2e-api-a-c9d1fc-ffdc6cb67-6nwgq"
Nov 02 03:21:45.904085 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:45.903698    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2371a226-dcd2-49f3-a966-41d7da7cafa8-plugins-config-volume\") pod \"payload-processing-e2e-api-a-c9d1fc-ffdc6cb67-6nwgq\" (UID: \"2371a226-dcd2-49f3-a966-41d7da7cafa8\") " pod="openshift-ingress/payload-processing-e2e-api-a-c9d1fc-ffdc6cb67-6nwgq"
Nov 02 03:21:45.924104 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:45.924072    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c9d1fc-5b9c6db48c-lkdss"
Nov 02 03:21:45.960744 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:45.960713    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6spbm\" (UniqueName: \"kubernetes.io/projected/2371a226-dcd2-49f3-a966-41d7da7cafa8-kube-api-access-6spbm\") pod \"payload-processing-e2e-api-a-c9d1fc-ffdc6cb67-6nwgq\" (UID: \"2371a226-dcd2-49f3-a966-41d7da7cafa8\") " pod="openshift-ingress/payload-processing-e2e-api-a-c9d1fc-ffdc6cb67-6nwgq"
Nov 02 03:21:46.034062 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:46.030398    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c9d1fc-ffdc6cb67-6nwgq"
Nov 02 03:21:46.317372 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:46.314385    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c9d1fc-5b9c6db48c-lkdss"]
Nov 02 03:21:46.419066 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:21:46.419029    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2371a226_dcd2_49f3_a966_41d7da7cafa8.slice/crio-a6173ee9223410d462d1e031fd11ce3b543bf94658eaa06d9b54aaa945bdf6d4 WatchSource:0}: Error finding container a6173ee9223410d462d1e031fd11ce3b543bf94658eaa06d9b54aaa945bdf6d4: Status 404 returned error can't find the container with id a6173ee9223410d462d1e031fd11ce3b543bf94658eaa06d9b54aaa945bdf6d4
Nov 02 03:21:46.421067 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:46.421038    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c9d1fc-ffdc6cb67-6nwgq"]
Nov 02 03:21:46.436374 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:46.436342    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c9d1fc-ffdc6cb67-6nwgq" event={"ID":"2371a226-dcd2-49f3-a966-41d7da7cafa8","Type":"ContainerStarted","Data":"a6173ee9223410d462d1e031fd11ce3b543bf94658eaa06d9b54aaa945bdf6d4"}
Nov 02 03:21:46.438385 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:46.438355    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c9d1fc-5b9c6db48c-lkdss" event={"ID":"2d399530-92b8-40ea-b7d0-05e30081f5f2","Type":"ContainerStarted","Data":"fbd0fe60d51e45bdaac91bbfe255f5da59b7917e4b971291493616ce74527777"}
Nov 02 03:21:46.438516 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:46.438412    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c9d1fc-5b9c6db48c-lkdss" event={"ID":"2d399530-92b8-40ea-b7d0-05e30081f5f2","Type":"ContainerStarted","Data":"9a185dfab07ec381347648b0326f1d6e1496185abe0f681243f9f61abc383ac8"}
Nov 02 03:21:46.438516 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:46.438445    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c9d1fc-5b9c6db48c-lkdss"
Nov 02 03:21:46.557127 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:46.556992    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c9d1fc-5b9c6db48c-lkdss" podStartSLOduration=1.556976221 podStartE2EDuration="1.556976221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:21:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:21:46.552785876 +0000 UTC m=+2047.131393256" watchObservedRunningTime="2026-11-02 03:21:46.556976221 +0000 UTC m=+2047.135583601"
Nov 02 03:21:47.449515 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:47.449477    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c9d1fc-ffdc6cb67-6nwgq" event={"ID":"2371a226-dcd2-49f3-a966-41d7da7cafa8","Type":"ContainerStarted","Data":"e34ac0a1cacb1d594227b40f5c1dcf4a8d9a5d7a98e5ed186306c3d0b68c8505"}
Nov 02 03:21:47.449705 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:47.449545    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-c9d1fc-ffdc6cb67-6nwgq"
Nov 02 03:21:47.568660 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:47.568597    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-c9d1fc-ffdc6cb67-6nwgq" podStartSLOduration=2.568581731 podStartE2EDuration="2.568581731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:21:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:21:47.56368238 +0000 UTC m=+2048.142289762" watchObservedRunningTime="2026-11-02 03:21:47.568581731 +0000 UTC m=+2048.147189111"
Nov 02 03:21:48.055570 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:48.055531    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-018cfa-7979545787-tsnwm"]
Nov 02 03:21:48.062072 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:48.061318    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-018cfa-7979545787-tsnwm"
Nov 02 03:21:48.121263 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:48.121223    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-018cfa\""
Nov 02 03:21:48.124268 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:48.124217    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-018cfa-dockercfg-zhfxg\""
Nov 02 03:21:48.136037 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:48.135923    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzf54\" (UniqueName: \"kubernetes.io/projected/8db02a90-7174-4696-90b4-5fc024addbc1-kube-api-access-hzf54\") pod \"payload-pre-processing-e2e-api-b-018cfa-7979545787-tsnwm\" (UID: \"8db02a90-7174-4696-90b4-5fc024addbc1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-018cfa-7979545787-tsnwm"
Nov 02 03:21:48.138316 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:48.136215    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8db02a90-7174-4696-90b4-5fc024addbc1-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-018cfa-7979545787-tsnwm\" (UID: \"8db02a90-7174-4696-90b4-5fc024addbc1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-018cfa-7979545787-tsnwm"
Nov 02 03:21:48.150717 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:48.150683    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-018cfa-7979545787-tsnwm"]
Nov 02 03:21:48.211539 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:48.206777    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-018cfa-74666c99c9-pv5nv"]
Nov 02 03:21:48.215080 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:48.215049    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-018cfa-74666c99c9-pv5nv"
Nov 02 03:21:48.237383 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:48.237346    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8db02a90-7174-4696-90b4-5fc024addbc1-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-018cfa-7979545787-tsnwm\" (UID: \"8db02a90-7174-4696-90b4-5fc024addbc1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-018cfa-7979545787-tsnwm"
Nov 02 03:21:48.237536 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:48.237459    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wskzv\" (UniqueName: \"kubernetes.io/projected/839528ca-8c68-42e2-9a79-a926450a021d-kube-api-access-wskzv\") pod \"payload-processing-e2e-api-b-018cfa-74666c99c9-pv5nv\" (UID: \"839528ca-8c68-42e2-9a79-a926450a021d\") " pod="openshift-ingress/payload-processing-e2e-api-b-018cfa-74666c99c9-pv5nv"
Nov 02 03:21:48.237594 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:48.237565    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzf54\" (UniqueName: \"kubernetes.io/projected/8db02a90-7174-4696-90b4-5fc024addbc1-kube-api-access-hzf54\") pod \"payload-pre-processing-e2e-api-b-018cfa-7979545787-tsnwm\" (UID: \"8db02a90-7174-4696-90b4-5fc024addbc1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-018cfa-7979545787-tsnwm"
Nov 02 03:21:48.237662 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:48.237617    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/839528ca-8c68-42e2-9a79-a926450a021d-plugins-config-volume\") pod \"payload-processing-e2e-api-b-018cfa-74666c99c9-pv5nv\" (UID: \"839528ca-8c68-42e2-9a79-a926450a021d\") " pod="openshift-ingress/payload-processing-e2e-api-b-018cfa-74666c99c9-pv5nv"
Nov 02 03:21:48.240094 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:48.240062    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8db02a90-7174-4696-90b4-5fc024addbc1-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-018cfa-7979545787-tsnwm\" (UID: \"8db02a90-7174-4696-90b4-5fc024addbc1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-018cfa-7979545787-tsnwm"
Nov 02 03:21:48.263408 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:48.262045    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-018cfa-74666c99c9-pv5nv"]
Nov 02 03:21:48.296743 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:48.296699    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzf54\" (UniqueName: \"kubernetes.io/projected/8db02a90-7174-4696-90b4-5fc024addbc1-kube-api-access-hzf54\") pod \"payload-pre-processing-e2e-api-b-018cfa-7979545787-tsnwm\" (UID: \"8db02a90-7174-4696-90b4-5fc024addbc1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-018cfa-7979545787-tsnwm"
Nov 02 03:21:48.371147 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:48.370945    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wskzv\" (UniqueName: \"kubernetes.io/projected/839528ca-8c68-42e2-9a79-a926450a021d-kube-api-access-wskzv\") pod \"payload-processing-e2e-api-b-018cfa-74666c99c9-pv5nv\" (UID: \"839528ca-8c68-42e2-9a79-a926450a021d\") " pod="openshift-ingress/payload-processing-e2e-api-b-018cfa-74666c99c9-pv5nv"
Nov 02 03:21:48.379613 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:48.371202    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/839528ca-8c68-42e2-9a79-a926450a021d-plugins-config-volume\") pod \"payload-processing-e2e-api-b-018cfa-74666c99c9-pv5nv\" (UID: \"839528ca-8c68-42e2-9a79-a926450a021d\") " pod="openshift-ingress/payload-processing-e2e-api-b-018cfa-74666c99c9-pv5nv"
Nov 02 03:21:48.379613 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:48.372405    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/839528ca-8c68-42e2-9a79-a926450a021d-plugins-config-volume\") pod \"payload-processing-e2e-api-b-018cfa-74666c99c9-pv5nv\" (UID: \"839528ca-8c68-42e2-9a79-a926450a021d\") " pod="openshift-ingress/payload-processing-e2e-api-b-018cfa-74666c99c9-pv5nv"
Nov 02 03:21:48.388101 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:48.379827    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-018cfa-7979545787-tsnwm"
Nov 02 03:21:48.460079 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:48.454491    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wskzv\" (UniqueName: \"kubernetes.io/projected/839528ca-8c68-42e2-9a79-a926450a021d-kube-api-access-wskzv\") pod \"payload-processing-e2e-api-b-018cfa-74666c99c9-pv5nv\" (UID: \"839528ca-8c68-42e2-9a79-a926450a021d\") " pod="openshift-ingress/payload-processing-e2e-api-b-018cfa-74666c99c9-pv5nv"
Nov 02 03:21:48.557571 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:48.557535    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-018cfa-74666c99c9-pv5nv"
Nov 02 03:21:48.799151 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:21:48.799113    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8db02a90_7174_4696_90b4_5fc024addbc1.slice/crio-09ddec3aa75db10608584a694976ec746e3d92fbb968d3213ff0c3e15704e12a WatchSource:0}: Error finding container 09ddec3aa75db10608584a694976ec746e3d92fbb968d3213ff0c3e15704e12a: Status 404 returned error can't find the container with id 09ddec3aa75db10608584a694976ec746e3d92fbb968d3213ff0c3e15704e12a
Nov 02 03:21:48.800732 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:48.800712    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-018cfa-7979545787-tsnwm"]
Nov 02 03:21:48.936246 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:21:48.936215    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod839528ca_8c68_42e2_9a79_a926450a021d.slice/crio-84df445189e616bbc98b6dfd8093f88a820f1c5b47fc4e2e1dbf8c19a558e980 WatchSource:0}: Error finding container 84df445189e616bbc98b6dfd8093f88a820f1c5b47fc4e2e1dbf8c19a558e980: Status 404 returned error can't find the container with id 84df445189e616bbc98b6dfd8093f88a820f1c5b47fc4e2e1dbf8c19a558e980
Nov 02 03:21:48.937867 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:48.937837    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-018cfa-74666c99c9-pv5nv"]
Nov 02 03:21:49.480248 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:49.480217    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-018cfa-7979545787-tsnwm" event={"ID":"8db02a90-7174-4696-90b4-5fc024addbc1","Type":"ContainerStarted","Data":"fb9003858c3d7b0464cc6312f1937d366f1bcc2ee76e807047e4db1b834a08ed"}
Nov 02 03:21:49.480455 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:49.480269    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-018cfa-7979545787-tsnwm" event={"ID":"8db02a90-7174-4696-90b4-5fc024addbc1","Type":"ContainerStarted","Data":"09ddec3aa75db10608584a694976ec746e3d92fbb968d3213ff0c3e15704e12a"}
Nov 02 03:21:49.480455 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:49.480326    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-018cfa-7979545787-tsnwm"
Nov 02 03:21:49.482344 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:49.482315    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-018cfa-74666c99c9-pv5nv" event={"ID":"839528ca-8c68-42e2-9a79-a926450a021d","Type":"ContainerStarted","Data":"a79a4369ff678a6a1d51ea993a1ea7df2d6b694190012e23e5250944a35dac80"}
Nov 02 03:21:49.482492 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:49.482364    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-018cfa-74666c99c9-pv5nv" event={"ID":"839528ca-8c68-42e2-9a79-a926450a021d","Type":"ContainerStarted","Data":"84df445189e616bbc98b6dfd8093f88a820f1c5b47fc4e2e1dbf8c19a558e980"}
Nov 02 03:21:49.482492 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:49.482396    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-018cfa-74666c99c9-pv5nv"
Nov 02 03:21:49.596646 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:49.596596    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-018cfa-7979545787-tsnwm" podStartSLOduration=1.596584454 podStartE2EDuration="1.596584454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:21:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:21:49.596193526 +0000 UTC m=+2050.174800905" watchObservedRunningTime="2026-11-02 03:21:49.596584454 +0000 UTC m=+2050.175191834"
Nov 02 03:21:49.722886 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:49.722833    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-018cfa-74666c99c9-pv5nv" podStartSLOduration=1.722814288 podStartE2EDuration="1.722814288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:21:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:21:49.71181525 +0000 UTC m=+2050.290422632" watchObservedRunningTime="2026-11-02 03:21:49.722814288 +0000 UTC m=+2050.301421668"
Nov 02 03:21:50.055673 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:50.055622    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c249c442-t79d4"]
Nov 02 03:21:50.061536 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:50.061502    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c249c442-t79d4"]
Nov 02 03:21:51.068193 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:51.068159    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d5dcca72-846bb99569-wldb8"]
Nov 02 03:21:51.180621 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:51.180583    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d5dcca72-7746cbb699-h8cx9"]
Nov 02 03:21:51.578868 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:51.578822    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95533a1b-befb-4b8c-89cb-ee3d53329c12" path="/var/lib/kubelet/pods/95533a1b-befb-4b8c-89cb-ee3d53329c12/volumes"
Nov 02 03:21:52.398096 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:52.397987    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d6544b-75f96f97f-6m8n2"
Nov 02 03:21:52.398096 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:52.398087    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-d6544b-5fc969d5f9-7k5sq"
Nov 02 03:21:54.263787 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:54.263754    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef76bd9a-jkjjp"]
Nov 02 03:21:54.269607 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:54.269586    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef76bd9a-jkjjp"
Nov 02 03:21:54.323994 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:54.323960    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef76bd9a-jkjjp"]
Nov 02 03:21:54.361743 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:54.361707    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4451fd8f-ab45-4c9b-8e91-c9b6f0a780fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef76bd9a-jkjjp\" (UID: \"4451fd8f-ab45-4c9b-8e91-c9b6f0a780fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef76bd9a-jkjjp"
Nov 02 03:21:54.361904 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:54.361774    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4451fd8f-ab45-4c9b-8e91-c9b6f0a780fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef76bd9a-jkjjp\" (UID: \"4451fd8f-ab45-4c9b-8e91-c9b6f0a780fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef76bd9a-jkjjp"
Nov 02 03:21:54.361971 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:54.361940    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h498x\" (UniqueName: \"kubernetes.io/projected/4451fd8f-ab45-4c9b-8e91-c9b6f0a780fe-kube-api-access-h498x\") pod \"maas-api-revoke-sub-ef76bd9a-jkjjp\" (UID: \"4451fd8f-ab45-4c9b-8e91-c9b6f0a780fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef76bd9a-jkjjp"
Nov 02 03:21:54.368540 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:54.368517    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 03:21:54.368540 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:54.368524    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 03:21:54.368705 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:54.368586    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9z5bk\""
Nov 02 03:21:54.368705 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:54.368643    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 03:21:54.463772 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:54.463741    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4451fd8f-ab45-4c9b-8e91-c9b6f0a780fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef76bd9a-jkjjp\" (UID: \"4451fd8f-ab45-4c9b-8e91-c9b6f0a780fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef76bd9a-jkjjp"
Nov 02 03:21:54.463951 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:54.463790    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4451fd8f-ab45-4c9b-8e91-c9b6f0a780fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef76bd9a-jkjjp\" (UID: \"4451fd8f-ab45-4c9b-8e91-c9b6f0a780fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef76bd9a-jkjjp"
Nov 02 03:21:54.463951 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:54.463852    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h498x\" (UniqueName: \"kubernetes.io/projected/4451fd8f-ab45-4c9b-8e91-c9b6f0a780fe-kube-api-access-h498x\") pod \"maas-api-revoke-sub-ef76bd9a-jkjjp\" (UID: \"4451fd8f-ab45-4c9b-8e91-c9b6f0a780fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef76bd9a-jkjjp"
Nov 02 03:21:54.464668 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:54.464638    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4451fd8f-ab45-4c9b-8e91-c9b6f0a780fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef76bd9a-jkjjp\" (UID: \"4451fd8f-ab45-4c9b-8e91-c9b6f0a780fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef76bd9a-jkjjp"
Nov 02 03:21:54.466219 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:54.466194    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4451fd8f-ab45-4c9b-8e91-c9b6f0a780fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef76bd9a-jkjjp\" (UID: \"4451fd8f-ab45-4c9b-8e91-c9b6f0a780fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef76bd9a-jkjjp"
Nov 02 03:21:54.524065 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:54.523995    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h498x\" (UniqueName: \"kubernetes.io/projected/4451fd8f-ab45-4c9b-8e91-c9b6f0a780fe-kube-api-access-h498x\") pod \"maas-api-revoke-sub-ef76bd9a-jkjjp\" (UID: \"4451fd8f-ab45-4c9b-8e91-c9b6f0a780fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef76bd9a-jkjjp"
Nov 02 03:21:54.585550 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:54.585516    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef76bd9a-jkjjp"
Nov 02 03:21:54.966167 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:21:54.966125    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4451fd8f_ab45_4c9b_8e91_c9b6f0a780fe.slice/crio-0d073221244b2a5ce4ef1920fba54774b03cb965cd1c38ceadba9d7265f71ce8 WatchSource:0}: Error finding container 0d073221244b2a5ce4ef1920fba54774b03cb965cd1c38ceadba9d7265f71ce8: Status 404 returned error can't find the container with id 0d073221244b2a5ce4ef1920fba54774b03cb965cd1c38ceadba9d7265f71ce8
Nov 02 03:21:54.968395 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:54.968358    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef76bd9a-jkjjp"]
Nov 02 03:21:55.588745 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:55.588642    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef76bd9a-jkjjp" event={"ID":"4451fd8f-ab45-4c9b-8e91-c9b6f0a780fe","Type":"ContainerStarted","Data":"d5afad9ae36dd33ee75c1a364c7cc5f01dc7bca87bea0e0047bac619b1c6bc40"}
Nov 02 03:21:55.588745 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:55.588702    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef76bd9a-jkjjp" event={"ID":"4451fd8f-ab45-4c9b-8e91-c9b6f0a780fe","Type":"ContainerStarted","Data":"0d073221244b2a5ce4ef1920fba54774b03cb965cd1c38ceadba9d7265f71ce8"}
Nov 02 03:21:55.705719 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:55.705664    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef76bd9a-jkjjp" podStartSLOduration=1.705649947 podStartE2EDuration="1.705649947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:21:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:21:55.702919936 +0000 UTC m=+2056.281527329" watchObservedRunningTime="2026-11-02 03:21:55.705649947 +0000 UTC m=+2056.284257356"
Nov 02 03:21:56.058046 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:56.057976    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84a7e2e3-bpr4q"]
Nov 02 03:21:56.063005 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:56.062973    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84a7e2e3-bpr4q"]
Nov 02 03:21:56.068489 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:56.068451    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-d5dcca72-846bb99569-wldb8" podUID="473be601-bbd9-4dcb-89ec-e43b66619766" containerName="payload-processing" containerID="cri-o://67c8426e350846de1fd54ad4b63f2156d324c9a6a10a6d69677671639670881b" gracePeriod=25
Nov 02 03:21:56.181803 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:56.181699    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-d5dcca72-7746cbb699-h8cx9" podUID="e54dff57-76c5-4a6f-bd9f-b59e6444b307" containerName="payload-pre-processing" containerID="cri-o://3d08984050e3663631649ec4e390035e89aed6fad6574495fe8d5b42d47bf05a" gracePeriod=25
Nov 02 03:21:56.598796 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:56.598766    4496 generic.go:356] "Generic (PLEG): container finished" podID="4451fd8f-ab45-4c9b-8e91-c9b6f0a780fe" containerID="d5afad9ae36dd33ee75c1a364c7cc5f01dc7bca87bea0e0047bac619b1c6bc40" exitCode=0
Nov 02 03:21:56.599210 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:56.598807    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef76bd9a-jkjjp" event={"ID":"4451fd8f-ab45-4c9b-8e91-c9b6f0a780fe","Type":"ContainerDied","Data":"d5afad9ae36dd33ee75c1a364c7cc5f01dc7bca87bea0e0047bac619b1c6bc40"}
Nov 02 03:21:57.452641 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:57.452611    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c9d1fc-5b9c6db48c-lkdss"
Nov 02 03:21:57.579843 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:57.579808    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9a2f2b6-c3a0-414f-89fa-50ce0279fb82" path="/var/lib/kubelet/pods/a9a2f2b6-c3a0-414f-89fa-50ce0279fb82/volumes"
Nov 02 03:21:57.813271 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:57.813246    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef76bd9a-jkjjp"
Nov 02 03:21:57.912336 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:57.912307    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4451fd8f-ab45-4c9b-8e91-c9b6f0a780fe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4451fd8f-ab45-4c9b-8e91-c9b6f0a780fe-maas-api-xks-ca\") pod \"4451fd8f-ab45-4c9b-8e91-c9b6f0a780fe\" (UID: \"4451fd8f-ab45-4c9b-8e91-c9b6f0a780fe\") "
Nov 02 03:21:57.912491 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:57.912356    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4451fd8f-ab45-4c9b-8e91-c9b6f0a780fe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4451fd8f-ab45-4c9b-8e91-c9b6f0a780fe-maas-api-openshift-ca\") pod \"4451fd8f-ab45-4c9b-8e91-c9b6f0a780fe\" (UID: \"4451fd8f-ab45-4c9b-8e91-c9b6f0a780fe\") "
Nov 02 03:21:57.912491 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:57.912400    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4451fd8f-ab45-4c9b-8e91-c9b6f0a780fe-kube-api-access-h498x\" (UniqueName: \"kubernetes.io/projected/4451fd8f-ab45-4c9b-8e91-c9b6f0a780fe-kube-api-access-h498x\") pod \"4451fd8f-ab45-4c9b-8e91-c9b6f0a780fe\" (UID: \"4451fd8f-ab45-4c9b-8e91-c9b6f0a780fe\") "
Nov 02 03:21:57.912800 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:57.912771    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4451fd8f-ab45-4c9b-8e91-c9b6f0a780fe-maas-api-openshift-ca" pod "4451fd8f-ab45-4c9b-8e91-c9b6f0a780fe" (UID: "4451fd8f-ab45-4c9b-8e91-c9b6f0a780fe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:21:57.913097 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:57.913080    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4451fd8f-ab45-4c9b-8e91-c9b6f0a780fe-maas-api-openshift-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:21:57.914537 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:57.914503    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4451fd8f-ab45-4c9b-8e91-c9b6f0a780fe-maas-api-xks-ca" pod "4451fd8f-ab45-4c9b-8e91-c9b6f0a780fe" (UID: "4451fd8f-ab45-4c9b-8e91-c9b6f0a780fe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 03:21:57.914537 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:57.914526    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4451fd8f-ab45-4c9b-8e91-c9b6f0a780fe-kube-api-access-h498x" pod "4451fd8f-ab45-4c9b-8e91-c9b6f0a780fe" (UID: "4451fd8f-ab45-4c9b-8e91-c9b6f0a780fe"). InnerVolumeSpecName "kube-api-access-h498x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:21:58.014399 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:58.014364    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4451fd8f-ab45-4c9b-8e91-c9b6f0a780fe-maas-api-xks-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:21:58.014567 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:58.014396    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h498x\" (UniqueName: \"kubernetes.io/projected/4451fd8f-ab45-4c9b-8e91-c9b6f0a780fe-kube-api-access-h498x\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:21:58.473056 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:58.473004    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-c9d1fc-ffdc6cb67-6nwgq"
Nov 02 03:21:58.619875 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:58.619846    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef76bd9a-jkjjp"
Nov 02 03:21:58.620079 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:58.619845    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef76bd9a-jkjjp" event={"ID":"4451fd8f-ab45-4c9b-8e91-c9b6f0a780fe","Type":"ContainerDied","Data":"0d073221244b2a5ce4ef1920fba54774b03cb965cd1c38ceadba9d7265f71ce8"}
Nov 02 03:21:58.620079 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:21:58.619970    4496 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0d073221244b2a5ce4ef1920fba54774b03cb965cd1c38ceadba9d7265f71ce8"
Nov 02 03:22:00.495794 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:22:00.495763    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-018cfa-7979545787-tsnwm"
Nov 02 03:22:00.496201 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:22:00.495836    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-018cfa-74666c99c9-pv5nv"
Nov 02 03:22:01.732969 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:22:01.732925    4496 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-d5dcca72-846bb99569-wldb8" podUID="473be601-bbd9-4dcb-89ec-e43b66619766" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 03:22:01.733397 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:22:01.732982    4496 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-d5dcca72-7746cbb699-h8cx9" podUID="e54dff57-76c5-4a6f-bd9f-b59e6444b307" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 03:22:03.528313 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:22:03.528276    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 03:22:06.047624 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:22:06.047589    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cea9a103-k9xxp"]
Nov 02 03:22:06.101460 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:22:06.101426    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cea9a103-k9xxp"]
Nov 02 03:22:07.050244 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:22:07.050205    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4b46d8f-m97rk"]
Nov 02 03:22:07.097607 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:22:07.097577    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4b46d8f-m97rk"]
Nov 02 03:22:07.575479 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:22:07.575445    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e29ac61-8a7e-41b7-ac1f-2067cd40dec9" path="/var/lib/kubelet/pods/7e29ac61-8a7e-41b7-ac1f-2067cd40dec9/volumes"
Nov 02 03:22:07.575946 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:22:07.575931    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d847479f-5794-498f-81b8-4834ee8d6b6b" path="/var/lib/kubelet/pods/d847479f-5794-498f-81b8-4834ee8d6b6b/volumes"
Nov 02 03:22:11.733295 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:22:11.733258    4496 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-d5dcca72-7746cbb699-h8cx9" podUID="e54dff57-76c5-4a6f-bd9f-b59e6444b307" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 03:22:11.733888 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:22:11.733273    4496 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-d5dcca72-846bb99569-wldb8" podUID="473be601-bbd9-4dcb-89ec-e43b66619766" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 03:22:16.391960 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:22:16.391908    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d5dcca72-846bb99569-wldb8"
Nov 02 03:22:16.486838 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:22:16.486804    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/473be601-bbd9-4dcb-89ec-e43b66619766-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/473be601-bbd9-4dcb-89ec-e43b66619766-plugins-config-volume\") pod \"473be601-bbd9-4dcb-89ec-e43b66619766\" (UID: \"473be601-bbd9-4dcb-89ec-e43b66619766\") "
Nov 02 03:22:16.486981 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:22:16.486861    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/473be601-bbd9-4dcb-89ec-e43b66619766-kube-api-access-wgwwj\" (UniqueName: \"kubernetes.io/projected/473be601-bbd9-4dcb-89ec-e43b66619766-kube-api-access-wgwwj\") pod \"473be601-bbd9-4dcb-89ec-e43b66619766\" (UID: \"473be601-bbd9-4dcb-89ec-e43b66619766\") "
Nov 02 03:22:16.487756 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:22:16.487726    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/473be601-bbd9-4dcb-89ec-e43b66619766-plugins-config-volume" pod "473be601-bbd9-4dcb-89ec-e43b66619766" (UID: "473be601-bbd9-4dcb-89ec-e43b66619766"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:22:16.489575 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:22:16.489552    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/473be601-bbd9-4dcb-89ec-e43b66619766-kube-api-access-wgwwj" pod "473be601-bbd9-4dcb-89ec-e43b66619766" (UID: "473be601-bbd9-4dcb-89ec-e43b66619766"). InnerVolumeSpecName "kube-api-access-wgwwj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:22:16.494562 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:22:16.494542    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d5dcca72-7746cbb699-h8cx9"
Nov 02 03:22:16.588241 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:22:16.588208    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e54dff57-76c5-4a6f-bd9f-b59e6444b307-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e54dff57-76c5-4a6f-bd9f-b59e6444b307-plugins-config-volume\") pod \"e54dff57-76c5-4a6f-bd9f-b59e6444b307\" (UID: \"e54dff57-76c5-4a6f-bd9f-b59e6444b307\") "
Nov 02 03:22:16.588416 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:22:16.588271    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e54dff57-76c5-4a6f-bd9f-b59e6444b307-kube-api-access-jvsvd\" (UniqueName: \"kubernetes.io/projected/e54dff57-76c5-4a6f-bd9f-b59e6444b307-kube-api-access-jvsvd\") pod \"e54dff57-76c5-4a6f-bd9f-b59e6444b307\" (UID: \"e54dff57-76c5-4a6f-bd9f-b59e6444b307\") "
Nov 02 03:22:16.588866 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:22:16.588566    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e54dff57-76c5-4a6f-bd9f-b59e6444b307-plugins-config-volume" pod "e54dff57-76c5-4a6f-bd9f-b59e6444b307" (UID: "e54dff57-76c5-4a6f-bd9f-b59e6444b307"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:22:16.588866 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:22:16.588855    4496 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e54dff57-76c5-4a6f-bd9f-b59e6444b307-plugins-config-volume\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:22:16.589066 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:22:16.588879    4496 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/473be601-bbd9-4dcb-89ec-e43b66619766-plugins-config-volume\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:22:16.589066 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:22:16.588896    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgwwj\" (UniqueName: \"kubernetes.io/projected/473be601-bbd9-4dcb-89ec-e43b66619766-kube-api-access-wgwwj\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:22:16.590546 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:22:16.590519    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e54dff57-76c5-4a6f-bd9f-b59e6444b307-kube-api-access-jvsvd" pod "e54dff57-76c5-4a6f-bd9f-b59e6444b307" (UID: "e54dff57-76c5-4a6f-bd9f-b59e6444b307"). InnerVolumeSpecName "kube-api-access-jvsvd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:22:16.690800 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:22:16.690722    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvsvd\" (UniqueName: \"kubernetes.io/projected/e54dff57-76c5-4a6f-bd9f-b59e6444b307-kube-api-access-jvsvd\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:22:16.794990 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:22:16.794947    4496 generic.go:356] "Generic (PLEG): container finished" podID="e54dff57-76c5-4a6f-bd9f-b59e6444b307" containerID="3d08984050e3663631649ec4e390035e89aed6fad6574495fe8d5b42d47bf05a" exitCode=0
Nov 02 03:22:16.795200 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:22:16.795000    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d5dcca72-7746cbb699-h8cx9"
Nov 02 03:22:16.795200 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:22:16.795062    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d5dcca72-7746cbb699-h8cx9" event={"ID":"e54dff57-76c5-4a6f-bd9f-b59e6444b307","Type":"ContainerDied","Data":"3d08984050e3663631649ec4e390035e89aed6fad6574495fe8d5b42d47bf05a"}
Nov 02 03:22:16.795200 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:22:16.795132    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d5dcca72-7746cbb699-h8cx9" event={"ID":"e54dff57-76c5-4a6f-bd9f-b59e6444b307","Type":"ContainerDied","Data":"cd703a792c2b8ab031b4fbf3bd8a57b5ded0c374ee55dbeb7974664eb4f49e96"}
Nov 02 03:22:16.795200 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:22:16.795160    4496 scope.go:122] "RemoveContainer" containerID="3d08984050e3663631649ec4e390035e89aed6fad6574495fe8d5b42d47bf05a"
Nov 02 03:22:16.796989 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:22:16.796966    4496 generic.go:356] "Generic (PLEG): container finished" podID="473be601-bbd9-4dcb-89ec-e43b66619766" containerID="67c8426e350846de1fd54ad4b63f2156d324c9a6a10a6d69677671639670881b" exitCode=0
Nov 02 03:22:16.797142 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:22:16.797034    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d5dcca72-846bb99569-wldb8" event={"ID":"473be601-bbd9-4dcb-89ec-e43b66619766","Type":"ContainerDied","Data":"67c8426e350846de1fd54ad4b63f2156d324c9a6a10a6d69677671639670881b"}
Nov 02 03:22:16.797142 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:22:16.797074    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d5dcca72-846bb99569-wldb8" event={"ID":"473be601-bbd9-4dcb-89ec-e43b66619766","Type":"ContainerDied","Data":"647e0619114b94b09ef98e908c59d13b89c4c0b632cc0b342ea5a626c1227c7c"}
Nov 02 03:22:16.797142 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:22:16.797114    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d5dcca72-846bb99569-wldb8"
Nov 02 03:22:16.807690 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:22:16.807668    4496 scope.go:122] "RemoveContainer" containerID="3d08984050e3663631649ec4e390035e89aed6fad6574495fe8d5b42d47bf05a"
Nov 02 03:22:16.807986 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:22:16.807966    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d08984050e3663631649ec4e390035e89aed6fad6574495fe8d5b42d47bf05a\": container with ID starting with 3d08984050e3663631649ec4e390035e89aed6fad6574495fe8d5b42d47bf05a not found: ID does not exist" containerID="3d08984050e3663631649ec4e390035e89aed6fad6574495fe8d5b42d47bf05a"
Nov 02 03:22:16.808096 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:22:16.807993    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d08984050e3663631649ec4e390035e89aed6fad6574495fe8d5b42d47bf05a"} err="failed to get container status \"3d08984050e3663631649ec4e390035e89aed6fad6574495fe8d5b42d47bf05a\": rpc error: code = NotFound desc = could not find container \"3d08984050e3663631649ec4e390035e89aed6fad6574495fe8d5b42d47bf05a\": container with ID starting with 3d08984050e3663631649ec4e390035e89aed6fad6574495fe8d5b42d47bf05a not found: ID does not exist"
Nov 02 03:22:16.808096 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:22:16.808011    4496 scope.go:122] "RemoveContainer" containerID="67c8426e350846de1fd54ad4b63f2156d324c9a6a10a6d69677671639670881b"
Nov 02 03:22:16.818923 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:22:16.818904    4496 scope.go:122] "RemoveContainer" containerID="67c8426e350846de1fd54ad4b63f2156d324c9a6a10a6d69677671639670881b"
Nov 02 03:22:16.819226 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:22:16.819203    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67c8426e350846de1fd54ad4b63f2156d324c9a6a10a6d69677671639670881b\": container with ID starting with 67c8426e350846de1fd54ad4b63f2156d324c9a6a10a6d69677671639670881b not found: ID does not exist" containerID="67c8426e350846de1fd54ad4b63f2156d324c9a6a10a6d69677671639670881b"
Nov 02 03:22:16.819299 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:22:16.819236    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67c8426e350846de1fd54ad4b63f2156d324c9a6a10a6d69677671639670881b"} err="failed to get container status \"67c8426e350846de1fd54ad4b63f2156d324c9a6a10a6d69677671639670881b\": rpc error: code = NotFound desc = could not find container \"67c8426e350846de1fd54ad4b63f2156d324c9a6a10a6d69677671639670881b\": container with ID starting with 67c8426e350846de1fd54ad4b63f2156d324c9a6a10a6d69677671639670881b not found: ID does not exist"
Nov 02 03:22:16.965331 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:22:16.965249    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d5dcca72-846bb99569-wldb8"]
Nov 02 03:22:16.970040 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:22:16.969990    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d5dcca72-846bb99569-wldb8"]
Nov 02 03:22:17.103768 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:22:17.102855    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d5dcca72-7746cbb699-h8cx9"]
Nov 02 03:22:17.106758 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:22:17.106725    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d5dcca72-7746cbb699-h8cx9"]
Nov 02 03:22:17.582136 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:22:17.582097    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="473be601-bbd9-4dcb-89ec-e43b66619766" path="/var/lib/kubelet/pods/473be601-bbd9-4dcb-89ec-e43b66619766/volumes"
Nov 02 03:22:17.582771 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:22:17.582751    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e54dff57-76c5-4a6f-bd9f-b59e6444b307" path="/var/lib/kubelet/pods/e54dff57-76c5-4a6f-bd9f-b59e6444b307/volumes"
Nov 02 03:22:18.052582 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:22:18.052545    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b77760e-5fwc5"]
Nov 02 03:22:18.107040 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:22:18.105708    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b77760e-5fwc5"]
Nov 02 03:22:19.577077 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:22:19.577039    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86fbde4b-8928-459a-bcc4-764f91c15ac4" path="/var/lib/kubelet/pods/86fbde4b-8928-459a-bcc4-764f91c15ac4/volumes"
Nov 02 03:22:23.048499 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:22:23.048464    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69fd8800-z6r62"]
Nov 02 03:22:23.099329 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:22:23.099294    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69fd8800-z6r62"]
Nov 02 03:22:23.605582 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:22:23.605540    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b852e4a-9db3-499e-8cc5-acd426d9247d" path="/var/lib/kubelet/pods/8b852e4a-9db3-499e-8cc5-acd426d9247d/volumes"
Nov 02 03:22:26.038348 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:22:26.038306    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a27da5-85ccf9bfcf-mhsrh"]
Nov 02 03:22:26.039206 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:22:26.039187    4496 cpu_manager.go:414] "containerMap: removing container" podUID="e54dff57-76c5-4a6f-bd9f-b59e6444b307" containerName="payload-pre-processing"
Nov 02 03:22:26.039206 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:22:26.039205    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e54dff57-76c5-4a6f-bd9f-b59e6444b307" containerName="payload-pre-processing"
Nov 02 03:22:26.039377 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:22:26.039219    4496 cpu_manager.go:414] "containerMap: removing container" podUID="4451fd8f-ab45-4c9b-8e91-c9b6f0a780fe" containerName="revoke-keys"
Nov 02 03:22:26.039377 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:22:26.039227    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4451fd8f-ab45-4c9b-8e91-c9b6f0a780fe" containerName="revoke-keys"
Nov 02 03:22:26.039377 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:22:26.039245    4496 cpu_manager.go:414] "containerMap: removing container" podUID="473be601-bbd9-4dcb-89ec-e43b66619766" containerName="payload-processing"
Nov 02 03:22:26.039377 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:22:26.039250    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="473be601-bbd9-4dcb-89ec-e43b66619766" containerName="payload-processing"
Nov 02 03:22:26.039546 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:22:26.039389    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="473be601-bbd9-4dcb-89ec-e43b66619766" containerName="payload-processing"
Nov 02 03:22:26.039546 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:22:26.039410    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="e54dff57-76c5-4a6f-bd9f-b59e6444b307" containerName="payload-pre-processing"
Nov 02 03:22:26.039546 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:22:26.039417    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="4451fd8f-ab45-4c9b-8e91-c9b6f0a780fe" containerName="revoke-keys"
Nov 02 03:22:26.044934 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:22:26.044914    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a27da5-85ccf9bfcf-mhsrh"
Nov 02 03:22:26.102359 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:22:26.102321    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a27da5-85ccf9bfcf-mhsrh"]
Nov 02 03:22:26.178627 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:22:26.178594    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-a27da5-dockercfg-2lmjl\""
Nov 02 03:22:26.179571 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:22:26.178906    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-a27da5\""
Nov 02 03:22:26.230335 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:22:26.218451    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsrpd\" (UniqueName: \"kubernetes.io/projected/b8658350-dded-408d-9904-40b3403119c9-kube-api-access-rsrpd\") pod \"payload-pre-processing-e2e-api-b-a27da5-85ccf9bfcf-mhsrh\" (UID: \"b8658350-dded-408d-9904-40b3403119c9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a27da5-85ccf9bfcf-mhsrh"
Nov 02 03:22:26.230684 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:22:26.230653    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8658350-dded-408d-9904-40b3403119c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-a27da5-85ccf9bfcf-mhsrh\" (UID: \"b8658350-dded-408d-9904-40b3403119c9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a27da5-85ccf9bfcf-mhsrh"
Nov 02 03:22:26.231255 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:22:26.225005    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a27da5-74bf9f44b5-xdcnr"]
Nov 02 03:22:26.241982 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:22:26.241956    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a27da5-74bf9f44b5-xdcnr"
Nov 02 03:22:26.277653 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:22:26.277621    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a27da5-74bf9f44b5-xdcnr"]
Nov 02 03:22:26.332817 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:22:26.332723    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsrpd\" (UniqueName: \"kubernetes.io/projected/b8658350-dded-408d-9904-40b3403119c9-kube-api-access-rsrpd\") pod \"payload-pre-processing-e2e-api-b-a27da5-85ccf9bfcf-mhsrh\" (UID: \"b8658350-dded-408d-9904-40b3403119c9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a27da5-85ccf9bfcf-mhsrh"
Nov 02 03:22:26.332999 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:22:26.332832    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8658350-dded-408d-9904-40b3403119c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-a27da5-85ccf9bfcf-mhsrh\" (UID: \"b8658350-dded-408d-9904-40b3403119c9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a27da5-85ccf9bfcf-mhsrh"
Nov 02 03:22:26.332999 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:22:26.332887    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42ccr\" (UniqueName: \"kubernetes.io/projected/020a5526-3ffd-46fb-8015-63d0e7657d37-kube-api-access-42ccr\") pod \"payload-processing-e2e-api-b-a27da5-74bf9f44b5-xdcnr\" (UID: \"020a5526-3ffd-46fb-8015-63d0e7657d37\") " pod="openshift-ingress/payload-processing-e2e-api-b-a27da5-74bf9f44b5-xdcnr"
Nov 02 03:22:26.333208 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:22:26.333115    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/020a5526-3ffd-46fb-8015-63d0e7657d37-plugins-config-volume\") pod \"payload-processing-e2e-api-b-a27da5-74bf9f44b5-xdcnr\" (UID: \"020a5526-3ffd-46fb-8015-63d0e7657d37\") " pod="openshift-ingress/payload-processing-e2e-api-b-a27da5-74bf9f44b5-xdcnr"
Nov 02 03:22:26.333890 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:22:26.333863    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8658350-dded-408d-9904-40b3403119c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-a27da5-85ccf9bfcf-mhsrh\" (UID: \"b8658350-dded-408d-9904-40b3403119c9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a27da5-85ccf9bfcf-mhsrh"
Nov 02 03:22:26.390964 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:22:26.390932    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsrpd\" (UniqueName: \"kubernetes.io/projected/b8658350-dded-408d-9904-40b3403119c9-kube-api-access-rsrpd\") pod \"payload-pre-processing-e2e-api-b-a27da5-85ccf9bfcf-mhsrh\" (UID: \"b8658350-dded-408d-9904-40b3403119c9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a27da5-85ccf9bfcf-mhsrh"
Nov 02 03:22:26.434958 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:22:26.434916    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/020a5526-3ffd-46fb-8015-63d0e7657d37-plugins-config-volume\") pod \"payload-processing-e2e-api-b-a27da5-74bf9f44b5-xdcnr\" (UID: \"020a5526-3ffd-46fb-8015-63d0e7657d37\") " pod="openshift-ingress/payload-processing-e2e-api-b-a27da5-74bf9f44b5-xdcnr"
Nov 02 03:22:26.435198 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:22:26.435178    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42ccr\" (UniqueName: \"kubernetes.io/projected/020a5526-3ffd-46fb-8015-63d0e7657d37-kube-api-access-42ccr\") pod \"payload-processing-e2e-api-b-a27da5-74bf9f44b5-xdcnr\" (UID: \"020a5526-3ffd-46fb-8015-63d0e7657d37\") " pod="openshift-ingress/payload-processing-e2e-api-b-a27da5-74bf9f44b5-xdcnr"
Nov 02 03:22:26.435707 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:22:26.435680    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/020a5526-3ffd-46fb-8015-63d0e7657d37-plugins-config-volume\") pod \"payload-processing-e2e-api-b-a27da5-74bf9f44b5-xdcnr\" (UID: \"020a5526-3ffd-46fb-8015-63d0e7657d37\") " pod="openshift-ingress/payload-processing-e2e-api-b-a27da5-74bf9f44b5-xdcnr"
Nov 02 03:22:26.492580 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:22:26.492546    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42ccr\" (UniqueName: \"kubernetes.io/projected/020a5526-3ffd-46fb-8015-63d0e7657d37-kube-api-access-42ccr\") pod \"payload-processing-e2e-api-b-a27da5-74bf9f44b5-xdcnr\" (UID: \"020a5526-3ffd-46fb-8015-63d0e7657d37\") " pod="openshift-ingress/payload-processing-e2e-api-b-a27da5-74bf9f44b5-xdcnr"
Nov 02 03:22:26.593259 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:22:26.593177    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a27da5-74bf9f44b5-xdcnr"
Nov 02 03:22:26.660127 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:22:26.660099    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a27da5-85ccf9bfcf-mhsrh"
Nov 02 03:22:26.980118 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:22:26.980086    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod020a5526_3ffd_46fb_8015_63d0e7657d37.slice/crio-99da226b3c60bac02f92822f16fd4e042292defbc0a482683915b3203dce6eff WatchSource:0}: Error finding container 99da226b3c60bac02f92822f16fd4e042292defbc0a482683915b3203dce6eff: Status 404 returned error can't find the container with id 99da226b3c60bac02f92822f16fd4e042292defbc0a482683915b3203dce6eff
Nov 02 03:22:26.981271 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:22:26.981246    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a27da5-74bf9f44b5-xdcnr"]
Nov 02 03:22:27.085241 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:22:27.085215    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8658350_dded_408d_9904_40b3403119c9.slice/crio-4e7395ca8e88191a5dcbb288305be219701a363d357083184727227051f12734 WatchSource:0}: Error finding container 4e7395ca8e88191a5dcbb288305be219701a363d357083184727227051f12734: Status 404 returned error can't find the container with id 4e7395ca8e88191a5dcbb288305be219701a363d357083184727227051f12734
Nov 02 03:22:27.087232 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:22:27.087211    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a27da5-85ccf9bfcf-mhsrh"]
Nov 02 03:22:27.911973 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:22:27.911932    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a27da5-85ccf9bfcf-mhsrh" event={"ID":"b8658350-dded-408d-9904-40b3403119c9","Type":"ContainerStarted","Data":"75b08842f147c8834a2c225eea9932ce212eb052a8594129606df98086aee582"}
Nov 02 03:22:27.912180 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:22:27.912000    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a27da5-85ccf9bfcf-mhsrh"
Nov 02 03:22:27.912180 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:22:27.912050    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a27da5-85ccf9bfcf-mhsrh" event={"ID":"b8658350-dded-408d-9904-40b3403119c9","Type":"ContainerStarted","Data":"4e7395ca8e88191a5dcbb288305be219701a363d357083184727227051f12734"}
Nov 02 03:22:27.913801 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:22:27.913773    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a27da5-74bf9f44b5-xdcnr" event={"ID":"020a5526-3ffd-46fb-8015-63d0e7657d37","Type":"ContainerStarted","Data":"9a3285a39f385b7c7bc1a1c821480f143057d25bf2945a9b4c61893030924917"}
Nov 02 03:22:27.913897 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:22:27.913818    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a27da5-74bf9f44b5-xdcnr" event={"ID":"020a5526-3ffd-46fb-8015-63d0e7657d37","Type":"ContainerStarted","Data":"99da226b3c60bac02f92822f16fd4e042292defbc0a482683915b3203dce6eff"}
Nov 02 03:22:27.913934 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:22:27.913894    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-a27da5-74bf9f44b5-xdcnr"
Nov 02 03:22:28.032043 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:22:28.031971    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a27da5-85ccf9bfcf-mhsrh" podStartSLOduration=3.03195105 podStartE2EDuration="3.03195105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:22:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:22:28.025931592 +0000 UTC m=+2088.604538973" watchObservedRunningTime="2026-11-02 03:22:28.03195105 +0000 UTC m=+2088.610558433"
Nov 02 03:22:28.149597 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:22:28.147741    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-a27da5-74bf9f44b5-xdcnr" podStartSLOduration=2.147721515 podStartE2EDuration="2.147721515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:22:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:22:28.146475397 +0000 UTC m=+2088.725082793" watchObservedRunningTime="2026-11-02 03:22:28.147721515 +0000 UTC m=+2088.726328897"
Nov 02 03:22:33.533639 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:22:33.533600    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 03:22:35.049074 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:22:35.049030    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce9e7480-qhxqv"]
Nov 02 03:22:35.098589 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:22:35.098555    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce9e7480-qhxqv"]
Nov 02 03:22:35.575164 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:22:35.575124    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f61f56ac-c17c-4227-8d05-fadb7732cf42" path="/var/lib/kubelet/pods/f61f56ac-c17c-4227-8d05-fadb7732cf42/volumes"
Nov 02 03:22:36.047639 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:22:36.047597    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a94dd74-6lwkz"]
Nov 02 03:22:36.097667 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:22:36.097634    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a94dd74-6lwkz"]
Nov 02 03:22:37.576868 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:22:37.576820    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dda7659a-4dd9-42bd-b072-902b9077062a" path="/var/lib/kubelet/pods/dda7659a-4dd9-42bd-b072-902b9077062a/volumes"
Nov 02 03:22:38.925938 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:22:38.925896    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a27da5-85ccf9bfcf-mhsrh"
Nov 02 03:22:38.926440 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:22:38.926387    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-a27da5-74bf9f44b5-xdcnr"
Nov 02 03:22:39.993991 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:22:39.993962    4496 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hww8q_5c797b08-a314-44ed-8026-8a0ce1f6c36f/ovn-acl-logging/0.log"
Nov 02 03:22:39.997122 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:22:39.997089    4496 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hww8q_5c797b08-a314-44ed-8026-8a0ce1f6c36f/ovn-acl-logging/0.log"
Nov 02 03:22:40.335372 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:22:40.335277    4496 scope.go:122] "RemoveContainer" containerID="dbb18c9e28037021d0458cdd0e34d2cfda0714f564ccadea9cd81f2172435be3"
Nov 02 03:22:40.347827 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:22:40.347807    4496 scope.go:122] "RemoveContainer" containerID="f7128300b8d451451498b3dfe1fca4de99070c3e2371d412ba2bbee2c21530cf"
Nov 02 03:22:40.359219 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:22:40.359200    4496 scope.go:122] "RemoveContainer" containerID="eff6ee9c20c069f3e7640ae158d7c32918e21df9be2403347c0a69d525ae905f"
Nov 02 03:22:40.368493 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:22:40.368472    4496 scope.go:122] "RemoveContainer" containerID="d1430193ebb75e2a8ff159897334a9554c1bc8c5392d7c48c0bbbce5d1ee27a4"
Nov 02 03:22:40.377896 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:22:40.377876    4496 scope.go:122] "RemoveContainer" containerID="b3951b9189d98c23c099a2324723e090841adf39327a9530d56f7e197e18e556"
Nov 02 03:22:40.388307 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:22:40.388287    4496 scope.go:122] "RemoveContainer" containerID="eabdf3ad41ba806ce5e37ec69d6535351452c26a36e80f59301011a2ebb6c210"
Nov 02 03:22:40.397537 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:22:40.397516    4496 scope.go:122] "RemoveContainer" containerID="877ad8569a3b7a138dce60e9c8e75867599f15bf6a97d261920987830fe9d751"
Nov 02 03:22:48.051584 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:22:48.051547    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-813da41b-mx857"]
Nov 02 03:22:48.098959 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:22:48.098925    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-813da41b-mx857"]
Nov 02 03:22:49.575995 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:22:49.575960    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f095bbfb-6214-411c-a1e1-6f59ddf05837" path="/var/lib/kubelet/pods/f095bbfb-6214-411c-a1e1-6f59ddf05837/volumes"
Nov 02 03:22:54.049874 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:22:54.049787    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7d14541-sfbvq"]
Nov 02 03:22:54.096657 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:22:54.096625    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7d14541-sfbvq"]
Nov 02 03:22:55.051548 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:22:55.051494    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a96d142e-8ck27"]
Nov 02 03:22:55.099439 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:22:55.099401    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a96d142e-8ck27"]
Nov 02 03:22:55.576661 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:22:55.576623    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fc3e00d-f565-40cc-8c0d-2d409da9bffc" path="/var/lib/kubelet/pods/4fc3e00d-f565-40cc-8c0d-2d409da9bffc/volumes"
Nov 02 03:22:55.577122 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:22:55.577106    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdaadbc3-63c0-4353-b3b6-576f63617326" path="/var/lib/kubelet/pods/fdaadbc3-63c0-4353-b3b6-576f63617326/volumes"
Nov 02 03:23:03.050424 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:03.050388    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d5dcca72-4pgb9"]
Nov 02 03:23:03.101053 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:03.100979    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d5dcca72-4pgb9"]
Nov 02 03:23:03.527375 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:23:03.527343    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 03:23:03.575412 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:03.575384    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fa39fec-bd1c-400e-a2b4-3b072c5bac77" path="/var/lib/kubelet/pods/3fa39fec-bd1c-400e-a2b4-3b072c5bac77/volumes"
Nov 02 03:23:14.049576 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:14.049535    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1606467d-4g8s2"]
Nov 02 03:23:14.098438 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:14.098391    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1606467d-4g8s2"]
Nov 02 03:23:15.579489 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:15.579456    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="789eba8d-594a-4f33-bc35-1d9c93281593" path="/var/lib/kubelet/pods/789eba8d-594a-4f33-bc35-1d9c93281593/volumes"
Nov 02 03:23:16.175779 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:16.175740    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d5dcca72-846bb99569-sjdx4"]
Nov 02 03:23:16.181996 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:16.181972    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d5dcca72-846bb99569-sjdx4"
Nov 02 03:23:16.211310 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:16.211280    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba6ad9c9-ab6b-43cd-bd4b-c7042792d7f3-plugins-config-volume\") pod \"payload-processing-e2e-mt-d5dcca72-846bb99569-sjdx4\" (UID: \"ba6ad9c9-ab6b-43cd-bd4b-c7042792d7f3\") " pod="openshift-ingress/payload-processing-e2e-mt-d5dcca72-846bb99569-sjdx4"
Nov 02 03:23:16.211459 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:16.211420    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxvgp\" (UniqueName: \"kubernetes.io/projected/ba6ad9c9-ab6b-43cd-bd4b-c7042792d7f3-kube-api-access-gxvgp\") pod \"payload-processing-e2e-mt-d5dcca72-846bb99569-sjdx4\" (UID: \"ba6ad9c9-ab6b-43cd-bd4b-c7042792d7f3\") " pod="openshift-ingress/payload-processing-e2e-mt-d5dcca72-846bb99569-sjdx4"
Nov 02 03:23:16.264657 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:16.264616    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d5dcca72-846bb99569-sjdx4"]
Nov 02 03:23:16.318402 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:16.315822    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba6ad9c9-ab6b-43cd-bd4b-c7042792d7f3-plugins-config-volume\") pod \"payload-processing-e2e-mt-d5dcca72-846bb99569-sjdx4\" (UID: \"ba6ad9c9-ab6b-43cd-bd4b-c7042792d7f3\") " pod="openshift-ingress/payload-processing-e2e-mt-d5dcca72-846bb99569-sjdx4"
Nov 02 03:23:16.321762 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:16.320365    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-d5dcca72\""
Nov 02 03:23:16.321762 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:16.320503    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-d5dcca72-dockercfg-qg2gb\""
Nov 02 03:23:16.348038 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:16.323442    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxvgp\" (UniqueName: \"kubernetes.io/projected/ba6ad9c9-ab6b-43cd-bd4b-c7042792d7f3-kube-api-access-gxvgp\") pod \"payload-processing-e2e-mt-d5dcca72-846bb99569-sjdx4\" (UID: \"ba6ad9c9-ab6b-43cd-bd4b-c7042792d7f3\") " pod="openshift-ingress/payload-processing-e2e-mt-d5dcca72-846bb99569-sjdx4"
Nov 02 03:23:16.348552 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:16.336969    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba6ad9c9-ab6b-43cd-bd4b-c7042792d7f3-plugins-config-volume\") pod \"payload-processing-e2e-mt-d5dcca72-846bb99569-sjdx4\" (UID: \"ba6ad9c9-ab6b-43cd-bd4b-c7042792d7f3\") " pod="openshift-ingress/payload-processing-e2e-mt-d5dcca72-846bb99569-sjdx4"
Nov 02 03:23:16.348720 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:16.345031    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d5dcca72-7746cbb699-tmsrl"]
Nov 02 03:23:16.365365 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:16.365295    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d5dcca72-7746cbb699-tmsrl"
Nov 02 03:23:16.400521 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:16.400485    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d5dcca72-7746cbb699-tmsrl"]
Nov 02 03:23:16.408500 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:16.408459    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxvgp\" (UniqueName: \"kubernetes.io/projected/ba6ad9c9-ab6b-43cd-bd4b-c7042792d7f3-kube-api-access-gxvgp\") pod \"payload-processing-e2e-mt-d5dcca72-846bb99569-sjdx4\" (UID: \"ba6ad9c9-ab6b-43cd-bd4b-c7042792d7f3\") " pod="openshift-ingress/payload-processing-e2e-mt-d5dcca72-846bb99569-sjdx4"
Nov 02 03:23:16.449909 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:16.449786    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfmck\" (UniqueName: \"kubernetes.io/projected/010d8313-308d-4bce-a4fa-277ead1e6d63-kube-api-access-gfmck\") pod \"payload-pre-processing-e2e-mt-d5dcca72-7746cbb699-tmsrl\" (UID: \"010d8313-308d-4bce-a4fa-277ead1e6d63\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d5dcca72-7746cbb699-tmsrl"
Nov 02 03:23:16.450096 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:16.449935    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/010d8313-308d-4bce-a4fa-277ead1e6d63-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d5dcca72-7746cbb699-tmsrl\" (UID: \"010d8313-308d-4bce-a4fa-277ead1e6d63\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d5dcca72-7746cbb699-tmsrl"
Nov 02 03:23:16.497335 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:16.497305    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d5dcca72-846bb99569-sjdx4"
Nov 02 03:23:16.551767 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:16.551729    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfmck\" (UniqueName: \"kubernetes.io/projected/010d8313-308d-4bce-a4fa-277ead1e6d63-kube-api-access-gfmck\") pod \"payload-pre-processing-e2e-mt-d5dcca72-7746cbb699-tmsrl\" (UID: \"010d8313-308d-4bce-a4fa-277ead1e6d63\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d5dcca72-7746cbb699-tmsrl"
Nov 02 03:23:16.551947 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:16.551867    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/010d8313-308d-4bce-a4fa-277ead1e6d63-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d5dcca72-7746cbb699-tmsrl\" (UID: \"010d8313-308d-4bce-a4fa-277ead1e6d63\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d5dcca72-7746cbb699-tmsrl"
Nov 02 03:23:16.557177 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:16.557116    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/010d8313-308d-4bce-a4fa-277ead1e6d63-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d5dcca72-7746cbb699-tmsrl\" (UID: \"010d8313-308d-4bce-a4fa-277ead1e6d63\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d5dcca72-7746cbb699-tmsrl"
Nov 02 03:23:16.610589 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:16.610555    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfmck\" (UniqueName: \"kubernetes.io/projected/010d8313-308d-4bce-a4fa-277ead1e6d63-kube-api-access-gfmck\") pod \"payload-pre-processing-e2e-mt-d5dcca72-7746cbb699-tmsrl\" (UID: \"010d8313-308d-4bce-a4fa-277ead1e6d63\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d5dcca72-7746cbb699-tmsrl"
Nov 02 03:23:16.688602 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:16.688554    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d5dcca72-7746cbb699-tmsrl"
Nov 02 03:23:16.889164 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:23:16.889130    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba6ad9c9_ab6b_43cd_bd4b_c7042792d7f3.slice/crio-a93dfd5db852294bb852807a610c65fff9b94d0ff81c4d44db868d8df8a87a3a WatchSource:0}: Error finding container a93dfd5db852294bb852807a610c65fff9b94d0ff81c4d44db868d8df8a87a3a: Status 404 returned error can't find the container with id a93dfd5db852294bb852807a610c65fff9b94d0ff81c4d44db868d8df8a87a3a
Nov 02 03:23:16.892586 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:16.892555    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d5dcca72-846bb99569-sjdx4"]
Nov 02 03:23:17.074918 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:23:17.074889    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod010d8313_308d_4bce_a4fa_277ead1e6d63.slice/crio-b849487e2a1a06e97d4e664ff270aceb367d22656da3441a853fbf8c8eccabf8 WatchSource:0}: Error finding container b849487e2a1a06e97d4e664ff270aceb367d22656da3441a853fbf8c8eccabf8: Status 404 returned error can't find the container with id b849487e2a1a06e97d4e664ff270aceb367d22656da3441a853fbf8c8eccabf8
Nov 02 03:23:17.076512 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:17.076484    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d5dcca72-7746cbb699-tmsrl"]
Nov 02 03:23:17.434121 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:17.431298    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d5dcca72-7746cbb699-tmsrl" event={"ID":"010d8313-308d-4bce-a4fa-277ead1e6d63","Type":"ContainerStarted","Data":"33b5621da04e1a4b0cc030c775712482ccadb8c8ab960b24c346489f24907235"}
Nov 02 03:23:17.434121 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:17.431406    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d5dcca72-7746cbb699-tmsrl" event={"ID":"010d8313-308d-4bce-a4fa-277ead1e6d63","Type":"ContainerStarted","Data":"b849487e2a1a06e97d4e664ff270aceb367d22656da3441a853fbf8c8eccabf8"}
Nov 02 03:23:17.434121 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:17.432088    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-d5dcca72-7746cbb699-tmsrl"
Nov 02 03:23:17.434404 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:17.434218    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d5dcca72-846bb99569-sjdx4" event={"ID":"ba6ad9c9-ab6b-43cd-bd4b-c7042792d7f3","Type":"ContainerStarted","Data":"842acaea42c2fa883c1e06099997759df180b89272c00304f20a2adebc314e5d"}
Nov 02 03:23:17.434404 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:17.434268    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d5dcca72-846bb99569-sjdx4" event={"ID":"ba6ad9c9-ab6b-43cd-bd4b-c7042792d7f3","Type":"ContainerStarted","Data":"a93dfd5db852294bb852807a610c65fff9b94d0ff81c4d44db868d8df8a87a3a"}
Nov 02 03:23:17.437682 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:17.434643    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-d5dcca72-846bb99569-sjdx4"
Nov 02 03:23:17.552455 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:17.552352    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-d5dcca72-7746cbb699-tmsrl" podStartSLOduration=1.552336135 podStartE2EDuration="1.552336135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:23:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:23:17.5484631 +0000 UTC m=+2138.127070495" watchObservedRunningTime="2026-11-02 03:23:17.552336135 +0000 UTC m=+2138.130943515"
Nov 02 03:23:28.447371 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:28.447341    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-d5dcca72-7746cbb699-tmsrl"
Nov 02 03:23:28.447956 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:28.447412    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-d5dcca72-846bb99569-sjdx4"
Nov 02 03:23:28.570782 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:28.570729    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-d5dcca72-846bb99569-sjdx4" podStartSLOduration=12.570714045999999 podStartE2EDuration="12.570714046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:23:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:23:17.667385106 +0000 UTC m=+2138.245992487" watchObservedRunningTime="2026-11-02 03:23:28.570714046 +0000 UTC m=+2149.149321446"
Nov 02 03:23:33.622195 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:23:33.622159    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 03:23:38.868479 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:38.868435    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ed15c1-cfl44"]
Nov 02 03:23:38.873599 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:38.873575    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ed15c1-cfl44"
Nov 02 03:23:38.901720 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:38.901690    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31334cad-3472-45dc-b713-de51a828f9b3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ed15c1-cfl44\" (UID: \"31334cad-3472-45dc-b713-de51a828f9b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ed15c1-cfl44"
Nov 02 03:23:38.901873 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:38.901831    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31334cad-3472-45dc-b713-de51a828f9b3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ed15c1-cfl44\" (UID: \"31334cad-3472-45dc-b713-de51a828f9b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ed15c1-cfl44"
Nov 02 03:23:38.901935 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:38.901919    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzwnd\" (UniqueName: \"kubernetes.io/projected/31334cad-3472-45dc-b713-de51a828f9b3-kube-api-access-mzwnd\") pod \"maas-api-revoke-keys-e2e-api-a-ed15c1-cfl44\" (UID: \"31334cad-3472-45dc-b713-de51a828f9b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ed15c1-cfl44"
Nov 02 03:23:38.929322 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:38.929286    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ed15c1-cfl44"]
Nov 02 03:23:38.973350 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:38.973316    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 03:23:38.973603 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:38.973325    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 03:23:38.973603 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:38.973384    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 03:23:38.977385 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:38.977359    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9z5bk\""
Nov 02 03:23:39.003023 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:39.002995    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31334cad-3472-45dc-b713-de51a828f9b3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ed15c1-cfl44\" (UID: \"31334cad-3472-45dc-b713-de51a828f9b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ed15c1-cfl44"
Nov 02 03:23:39.003162 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:39.003147    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31334cad-3472-45dc-b713-de51a828f9b3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ed15c1-cfl44\" (UID: \"31334cad-3472-45dc-b713-de51a828f9b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ed15c1-cfl44"
Nov 02 03:23:39.003240 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:39.003226    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzwnd\" (UniqueName: \"kubernetes.io/projected/31334cad-3472-45dc-b713-de51a828f9b3-kube-api-access-mzwnd\") pod \"maas-api-revoke-keys-e2e-api-a-ed15c1-cfl44\" (UID: \"31334cad-3472-45dc-b713-de51a828f9b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ed15c1-cfl44"
Nov 02 03:23:39.003955 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:39.003929    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31334cad-3472-45dc-b713-de51a828f9b3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ed15c1-cfl44\" (UID: \"31334cad-3472-45dc-b713-de51a828f9b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ed15c1-cfl44"
Nov 02 03:23:39.005558 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:39.005532    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31334cad-3472-45dc-b713-de51a828f9b3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ed15c1-cfl44\" (UID: \"31334cad-3472-45dc-b713-de51a828f9b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ed15c1-cfl44"
Nov 02 03:23:39.060546 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:39.060511    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzwnd\" (UniqueName: \"kubernetes.io/projected/31334cad-3472-45dc-b713-de51a828f9b3-kube-api-access-mzwnd\") pod \"maas-api-revoke-keys-e2e-api-a-ed15c1-cfl44\" (UID: \"31334cad-3472-45dc-b713-de51a828f9b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ed15c1-cfl44"
Nov 02 03:23:39.191942 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:39.191867    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ed15c1-cfl44"
Nov 02 03:23:39.571815 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:23:39.571781    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31334cad_3472_45dc_b713_de51a828f9b3.slice/crio-8bd90c81b2d17fde75ea7d1d49d1b29400005c2196cda7763c636ea6ef005f9c WatchSource:0}: Error finding container 8bd90c81b2d17fde75ea7d1d49d1b29400005c2196cda7763c636ea6ef005f9c: Status 404 returned error can't find the container with id 8bd90c81b2d17fde75ea7d1d49d1b29400005c2196cda7763c636ea6ef005f9c
Nov 02 03:23:39.577657 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:39.577633    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ed15c1-cfl44"]
Nov 02 03:23:39.676964 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:39.676929    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ed15c1-cfl44" event={"ID":"31334cad-3472-45dc-b713-de51a828f9b3","Type":"ContainerStarted","Data":"8bd90c81b2d17fde75ea7d1d49d1b29400005c2196cda7763c636ea6ef005f9c"}
Nov 02 03:23:40.232046 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:40.232001    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hdjcq"]
Nov 02 03:23:40.232435 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:40.232205    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hdjcq" podUID="37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f" containerName="istio-proxy" containerID="cri-o://3c99703636204b5e27d2c604df31d3018e4377683b9d37883fa2f3b33a659b2d" gracePeriod=30
Nov 02 03:23:40.494813 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:40.494739    4496 scope.go:122] "RemoveContainer" containerID="c3da87de80b80e9357eda1f208e5baf3b419fce4ea1622766d27caee29ad4710"
Nov 02 03:23:40.506098 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:40.506075    4496 scope.go:122] "RemoveContainer" containerID="7a3938a92e4156c9ff8203a871f406e24ee6e45d44fd1e8afee5d5610c40376a"
Nov 02 03:23:40.516301 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:40.516276    4496 scope.go:122] "RemoveContainer" containerID="bb960bcc3372d1a20c682cac680245835e5d6c5bedabcecd4c36d5d7695ee8f2"
Nov 02 03:23:40.526174 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:40.526150    4496 scope.go:122] "RemoveContainer" containerID="096d2cff54f511a2a2a8be39b0eadbedb16ff4024ff6b00e54cde9bc3c1d2ec2"
Nov 02 03:23:40.536335 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:40.536318    4496 scope.go:122] "RemoveContainer" containerID="6bb217f3bcc2a4b6f4811595c68285f342fe405464a1d17312a994f99f4dcb90"
Nov 02 03:23:40.631954 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:40.631918    4496 scope.go:122] "RemoveContainer" containerID="9eb93870a5f91da823f9159681d7487d6058e537f2de8d4818e4a86ce2ce80b1"
Nov 02 03:23:40.642411 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:40.642383    4496 scope.go:122] "RemoveContainer" containerID="e9e99f3ff4c8dc35a537cbb0487547198d8e7a55401dd90f435807b6acb76903"
Nov 02 03:23:40.655007 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:40.654985    4496 scope.go:122] "RemoveContainer" containerID="40e9b46633730fe9758c269f0d869e2fb8335fbd7b402e3731cc369ab268b69b"
Nov 02 03:23:40.698643 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:40.698617    4496 generic.go:356] "Generic (PLEG): container finished" podID="31334cad-3472-45dc-b713-de51a828f9b3" containerID="74b4a9412faabcb34bcab873f489ed3563572c463ce0042cc498624c6caecde8" exitCode=0
Nov 02 03:23:40.698766 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:40.698701    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ed15c1-cfl44" event={"ID":"31334cad-3472-45dc-b713-de51a828f9b3","Type":"ContainerDied","Data":"74b4a9412faabcb34bcab873f489ed3563572c463ce0042cc498624c6caecde8"}
Nov 02 03:23:41.921780 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:41.921745    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ed15c1-cfl44"
Nov 02 03:23:42.048643 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:42.048607    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31334cad-3472-45dc-b713-de51a828f9b3-kube-api-access-mzwnd\" (UniqueName: \"kubernetes.io/projected/31334cad-3472-45dc-b713-de51a828f9b3-kube-api-access-mzwnd\") pod \"31334cad-3472-45dc-b713-de51a828f9b3\" (UID: \"31334cad-3472-45dc-b713-de51a828f9b3\") "
Nov 02 03:23:42.048845 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:42.048816    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31334cad-3472-45dc-b713-de51a828f9b3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31334cad-3472-45dc-b713-de51a828f9b3-maas-api-openshift-ca\") pod \"31334cad-3472-45dc-b713-de51a828f9b3\" (UID: \"31334cad-3472-45dc-b713-de51a828f9b3\") "
Nov 02 03:23:42.048918 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:42.048855    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31334cad-3472-45dc-b713-de51a828f9b3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31334cad-3472-45dc-b713-de51a828f9b3-maas-api-xks-ca\") pod \"31334cad-3472-45dc-b713-de51a828f9b3\" (UID: \"31334cad-3472-45dc-b713-de51a828f9b3\") "
Nov 02 03:23:42.049255 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:42.049215    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31334cad-3472-45dc-b713-de51a828f9b3-maas-api-openshift-ca" pod "31334cad-3472-45dc-b713-de51a828f9b3" (UID: "31334cad-3472-45dc-b713-de51a828f9b3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:23:42.049712 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:42.049665    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31334cad-3472-45dc-b713-de51a828f9b3-maas-api-openshift-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:23:42.054385 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:42.054311    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31334cad-3472-45dc-b713-de51a828f9b3-kube-api-access-mzwnd" pod "31334cad-3472-45dc-b713-de51a828f9b3" (UID: "31334cad-3472-45dc-b713-de51a828f9b3"). InnerVolumeSpecName "kube-api-access-mzwnd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:23:42.054385 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:42.054327    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31334cad-3472-45dc-b713-de51a828f9b3-maas-api-xks-ca" pod "31334cad-3472-45dc-b713-de51a828f9b3" (UID: "31334cad-3472-45dc-b713-de51a828f9b3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 03:23:42.151101 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:42.151067    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31334cad-3472-45dc-b713-de51a828f9b3-maas-api-xks-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:23:42.151101 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:42.151095    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzwnd\" (UniqueName: \"kubernetes.io/projected/31334cad-3472-45dc-b713-de51a828f9b3-kube-api-access-mzwnd\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:23:42.718215 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:42.718187    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ed15c1-cfl44"
Nov 02 03:23:42.718392 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:42.718225    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ed15c1-cfl44" event={"ID":"31334cad-3472-45dc-b713-de51a828f9b3","Type":"ContainerDied","Data":"8bd90c81b2d17fde75ea7d1d49d1b29400005c2196cda7763c636ea6ef005f9c"}
Nov 02 03:23:42.718392 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:42.718271    4496 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8bd90c81b2d17fde75ea7d1d49d1b29400005c2196cda7763c636ea6ef005f9c"
Nov 02 03:23:45.522996 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:45.522934    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hdjcq"
Nov 02 03:23:45.600040 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:45.600001    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f-kube-api-access-9flpz\" (UniqueName: \"kubernetes.io/projected/37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f-kube-api-access-9flpz\") pod \"37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f\" (UID: \"37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f\") "
Nov 02 03:23:45.600192 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:45.600115    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f-istio-envoy\") pod \"37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f\" (UID: \"37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f\") "
Nov 02 03:23:45.600192 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:45.600145    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f-workload-socket\") pod \"37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f\" (UID: \"37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f\") "
Nov 02 03:23:45.600192 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:45.600165    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f-workload-certs\") pod \"37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f\" (UID: \"37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f\") "
Nov 02 03:23:45.600385 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:45.600276    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f-credential-socket\") pod \"37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f\" (UID: \"37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f\") "
Nov 02 03:23:45.600385 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:45.600315    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f-istio-token\" (UniqueName: \"kubernetes.io/projected/37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f-istio-token\") pod \"37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f\" (UID: \"37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f\") "
Nov 02 03:23:45.600385 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:45.600355    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f-istio-podinfo\") pod \"37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f\" (UID: \"37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f\") "
Nov 02 03:23:45.600548 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:45.600430    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f-istio-data\") pod \"37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f\" (UID: \"37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f\") "
Nov 02 03:23:45.600548 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:45.600468    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f-istiod-ca-cert\") pod \"37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f\" (UID: \"37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f\") "
Nov 02 03:23:45.600548 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:45.600450    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f-workload-socket" pod "37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f" (UID: "37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:23:45.600548 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:45.600524    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f-workload-certs" pod "37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f" (UID: "37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:23:45.600773 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:45.600544    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f-credential-socket" pod "37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f" (UID: "37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:23:45.600773 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:45.600740    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f-istio-data" pod "37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f" (UID: "37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:23:45.600950 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:45.600927    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f-istiod-ca-cert" pod "37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f" (UID: "37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:23:45.601252 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:45.601230    4496 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f-workload-socket\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:23:45.601357 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:45.601256    4496 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f-workload-certs\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:23:45.601357 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:45.601271    4496 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f-credential-socket\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:23:45.601357 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:45.601286    4496 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f-istio-data\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:23:45.601357 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:45.601300    4496 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f-istiod-ca-cert\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:23:45.602486 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:45.602463    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f-istio-envoy" pod "37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f" (UID: "37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:23:45.602646 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:45.602629    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f-istio-podinfo" pod "37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f" (UID: "37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 03:23:45.602725 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:45.602700    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f-kube-api-access-9flpz" pod "37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f" (UID: "37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f"). InnerVolumeSpecName "kube-api-access-9flpz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:23:45.602790 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:45.602748    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f-istio-token" pod "37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f" (UID: "37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:23:45.702536 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:45.702506    4496 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f-istio-token\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:23:45.702536 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:45.702531    4496 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f-istio-podinfo\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:23:45.702536 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:45.702541    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9flpz\" (UniqueName: \"kubernetes.io/projected/37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f-kube-api-access-9flpz\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:23:45.702536 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:45.702551    4496 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f-istio-envoy\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:23:45.770070 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:45.766077    4496 generic.go:356] "Generic (PLEG): container finished" podID="37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f" containerID="3c99703636204b5e27d2c604df31d3018e4377683b9d37883fa2f3b33a659b2d" exitCode=0
Nov 02 03:23:45.770070 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:45.766347    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hdjcq" event={"ID":"37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f","Type":"ContainerDied","Data":"3c99703636204b5e27d2c604df31d3018e4377683b9d37883fa2f3b33a659b2d"}
Nov 02 03:23:45.770070 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:45.766396    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hdjcq" event={"ID":"37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f","Type":"ContainerDied","Data":"b898b0cc29ae06258212b8ad9b2a98cc991ed84aa7d88b6e144823f40cf386e6"}
Nov 02 03:23:45.770070 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:45.766425    4496 scope.go:122] "RemoveContainer" containerID="3c99703636204b5e27d2c604df31d3018e4377683b9d37883fa2f3b33a659b2d"
Nov 02 03:23:45.770070 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:45.766607    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hdjcq"
Nov 02 03:23:45.798218 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:45.798196    4496 scope.go:122] "RemoveContainer" containerID="3c99703636204b5e27d2c604df31d3018e4377683b9d37883fa2f3b33a659b2d"
Nov 02 03:23:45.798600 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:23:45.798558    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c99703636204b5e27d2c604df31d3018e4377683b9d37883fa2f3b33a659b2d\": container with ID starting with 3c99703636204b5e27d2c604df31d3018e4377683b9d37883fa2f3b33a659b2d not found: ID does not exist" containerID="3c99703636204b5e27d2c604df31d3018e4377683b9d37883fa2f3b33a659b2d"
Nov 02 03:23:45.798667 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:45.798617    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c99703636204b5e27d2c604df31d3018e4377683b9d37883fa2f3b33a659b2d"} err="failed to get container status \"3c99703636204b5e27d2c604df31d3018e4377683b9d37883fa2f3b33a659b2d\": rpc error: code = NotFound desc = could not find container \"3c99703636204b5e27d2c604df31d3018e4377683b9d37883fa2f3b33a659b2d\": container with ID starting with 3c99703636204b5e27d2c604df31d3018e4377683b9d37883fa2f3b33a659b2d not found: ID does not exist"
Nov 02 03:23:45.955801 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:45.955766    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hdjcq"]
Nov 02 03:23:45.960206 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:45.960178    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hdjcq"]
Nov 02 03:23:46.503180 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:46.503142    4496 patch_prober.go:29] interesting pod/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hdjcq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 03:23:46.503357 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:46.503223    4496 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hdjcq" podUID="37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 03:23:47.579134 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:47.579100    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f" path="/var/lib/kubelet/pods/37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f/volumes"
Nov 02 03:23:52.719834 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:52.719804    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ed15c1-56c5d6557c-g5df2"]
Nov 02 03:23:52.821425 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:52.821389    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ed15c1-57bddc8787-l7kbv"]
Nov 02 03:23:54.429076 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:54.423514    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0d26af-7m7t5"]
Nov 02 03:23:54.429076 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:54.424736    4496 cpu_manager.go:414] "containerMap: removing container" podUID="37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f" containerName="istio-proxy"
Nov 02 03:23:54.429076 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:54.424758    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f" containerName="istio-proxy"
Nov 02 03:23:54.429076 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:54.424782    4496 cpu_manager.go:414] "containerMap: removing container" podUID="31334cad-3472-45dc-b713-de51a828f9b3" containerName="revoke-keys"
Nov 02 03:23:54.429076 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:54.424790    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31334cad-3472-45dc-b713-de51a828f9b3" containerName="revoke-keys"
Nov 02 03:23:54.429076 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:54.424938    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="37c23d26-aa7d-41dd-b3d9-7a3408ec2f1f" containerName="istio-proxy"
Nov 02 03:23:54.429076 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:54.424950    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="31334cad-3472-45dc-b713-de51a828f9b3" containerName="revoke-keys"
Nov 02 03:23:54.430790 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:54.430768    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0d26af-7m7t5"
Nov 02 03:23:54.540891 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:54.537097    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9b6d8fd-c404-4c32-a6ae-8ba7b48314e3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-0d26af-7m7t5\" (UID: \"f9b6d8fd-c404-4c32-a6ae-8ba7b48314e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0d26af-7m7t5"
Nov 02 03:23:54.540891 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:54.537226    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9b6d8fd-c404-4c32-a6ae-8ba7b48314e3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-0d26af-7m7t5\" (UID: \"f9b6d8fd-c404-4c32-a6ae-8ba7b48314e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0d26af-7m7t5"
Nov 02 03:23:54.540891 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:54.537526    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mm2s\" (UniqueName: \"kubernetes.io/projected/f9b6d8fd-c404-4c32-a6ae-8ba7b48314e3-kube-api-access-9mm2s\") pod \"maas-api-revoke-keys-e2e-api-a-0d26af-7m7t5\" (UID: \"f9b6d8fd-c404-4c32-a6ae-8ba7b48314e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0d26af-7m7t5"
Nov 02 03:23:54.540891 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:54.540736    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0d26af-7m7t5"]
Nov 02 03:23:54.556123 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:54.554913    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 03:23:54.556123 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:54.555151    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9z5bk\""
Nov 02 03:23:54.556123 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:54.555239    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 03:23:54.556123 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:54.555341    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 03:23:54.638502 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:54.638459    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9b6d8fd-c404-4c32-a6ae-8ba7b48314e3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-0d26af-7m7t5\" (UID: \"f9b6d8fd-c404-4c32-a6ae-8ba7b48314e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0d26af-7m7t5"
Nov 02 03:23:54.638722 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:54.638608    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mm2s\" (UniqueName: \"kubernetes.io/projected/f9b6d8fd-c404-4c32-a6ae-8ba7b48314e3-kube-api-access-9mm2s\") pod \"maas-api-revoke-keys-e2e-api-a-0d26af-7m7t5\" (UID: \"f9b6d8fd-c404-4c32-a6ae-8ba7b48314e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0d26af-7m7t5"
Nov 02 03:23:54.638722 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:54.638716    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9b6d8fd-c404-4c32-a6ae-8ba7b48314e3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-0d26af-7m7t5\" (UID: \"f9b6d8fd-c404-4c32-a6ae-8ba7b48314e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0d26af-7m7t5"
Nov 02 03:23:54.639376 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:54.639344    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9b6d8fd-c404-4c32-a6ae-8ba7b48314e3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-0d26af-7m7t5\" (UID: \"f9b6d8fd-c404-4c32-a6ae-8ba7b48314e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0d26af-7m7t5"
Nov 02 03:23:54.641062 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:54.641041    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9b6d8fd-c404-4c32-a6ae-8ba7b48314e3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-0d26af-7m7t5\" (UID: \"f9b6d8fd-c404-4c32-a6ae-8ba7b48314e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0d26af-7m7t5"
Nov 02 03:23:54.696498 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:54.696428    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mm2s\" (UniqueName: \"kubernetes.io/projected/f9b6d8fd-c404-4c32-a6ae-8ba7b48314e3-kube-api-access-9mm2s\") pod \"maas-api-revoke-keys-e2e-api-a-0d26af-7m7t5\" (UID: \"f9b6d8fd-c404-4c32-a6ae-8ba7b48314e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0d26af-7m7t5"
Nov 02 03:23:54.757739 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:54.757702    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0d26af-7m7t5"
Nov 02 03:23:55.136368 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:23:55.136224    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9b6d8fd_c404_4c32_a6ae_8ba7b48314e3.slice/crio-d538cafff563c89cdaa08217fad458e71aae3f09f7b7c174745f1fbaed99df9e WatchSource:0}: Error finding container d538cafff563c89cdaa08217fad458e71aae3f09f7b7c174745f1fbaed99df9e: Status 404 returned error can't find the container with id d538cafff563c89cdaa08217fad458e71aae3f09f7b7c174745f1fbaed99df9e
Nov 02 03:23:55.142771 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:55.142742    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0d26af-7m7t5"]
Nov 02 03:23:55.873981 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:55.873941    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0d26af-7m7t5" event={"ID":"f9b6d8fd-c404-4c32-a6ae-8ba7b48314e3","Type":"ContainerStarted","Data":"d335a6aa085bc63caa501612721dce29696ef8ab2d99e8114fa393ab340afa23"}
Nov 02 03:23:55.874595 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:55.874003    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0d26af-7m7t5" event={"ID":"f9b6d8fd-c404-4c32-a6ae-8ba7b48314e3","Type":"ContainerStarted","Data":"d538cafff563c89cdaa08217fad458e71aae3f09f7b7c174745f1fbaed99df9e"}
Nov 02 03:23:55.992665 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:55.992610    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0d26af-7m7t5" podStartSLOduration=1.992595044 podStartE2EDuration="1.992595044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:23:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:23:55.987490549 +0000 UTC m=+2176.566097930" watchObservedRunningTime="2026-11-02 03:23:55.992595044 +0000 UTC m=+2176.571202424"
Nov 02 03:23:56.883575 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:56.883539    4496 generic.go:356] "Generic (PLEG): container finished" podID="f9b6d8fd-c404-4c32-a6ae-8ba7b48314e3" containerID="d335a6aa085bc63caa501612721dce29696ef8ab2d99e8114fa393ab340afa23" exitCode=0
Nov 02 03:23:56.883968 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:56.883631    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0d26af-7m7t5" event={"ID":"f9b6d8fd-c404-4c32-a6ae-8ba7b48314e3","Type":"ContainerDied","Data":"d335a6aa085bc63caa501612721dce29696ef8ab2d99e8114fa393ab340afa23"}
Nov 02 03:23:57.720261 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:57.720221    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-ed15c1-56c5d6557c-g5df2" podUID="5ac8f265-8646-430c-bca6-36655877def2" containerName="payload-processing" containerID="cri-o://7240d587c48d2d15da42e1dc5d035be3e2e6eb681b15ea42c5a440183f088e14" gracePeriod=25
Nov 02 03:23:57.822725 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:57.822662    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ed15c1-57bddc8787-l7kbv" podUID="4cf48f92-69f5-47aa-8163-26f2d6df1921" containerName="payload-pre-processing" containerID="cri-o://0d00e0c98d00b1689c7313b1a401b1a59eddbfacd71fd0746e46caae0cd8b578" gracePeriod=25
Nov 02 03:23:57.894802 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:57.894771    4496 generic.go:356] "Generic (PLEG): container finished" podID="5ac8f265-8646-430c-bca6-36655877def2" containerID="7240d587c48d2d15da42e1dc5d035be3e2e6eb681b15ea42c5a440183f088e14" exitCode=0
Nov 02 03:23:57.895243 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:57.894841    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-ed15c1-56c5d6557c-g5df2" event={"ID":"5ac8f265-8646-430c-bca6-36655877def2","Type":"ContainerDied","Data":"7240d587c48d2d15da42e1dc5d035be3e2e6eb681b15ea42c5a440183f088e14"}
Nov 02 03:23:58.042180 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:58.041736    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-ed15c1-56c5d6557c-g5df2"
Nov 02 03:23:58.154239 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:58.154207    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0d26af-7m7t5"
Nov 02 03:23:58.191218 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:58.191183    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ac8f265-8646-430c-bca6-36655877def2-kube-api-access-jq4wn\" (UniqueName: \"kubernetes.io/projected/5ac8f265-8646-430c-bca6-36655877def2-kube-api-access-jq4wn\") pod \"5ac8f265-8646-430c-bca6-36655877def2\" (UID: \"5ac8f265-8646-430c-bca6-36655877def2\") "
Nov 02 03:23:58.191358 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:58.191289    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ac8f265-8646-430c-bca6-36655877def2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ac8f265-8646-430c-bca6-36655877def2-plugins-config-volume\") pod \"5ac8f265-8646-430c-bca6-36655877def2\" (UID: \"5ac8f265-8646-430c-bca6-36655877def2\") "
Nov 02 03:23:58.191728 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:58.191693    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ac8f265-8646-430c-bca6-36655877def2-plugins-config-volume" pod "5ac8f265-8646-430c-bca6-36655877def2" (UID: "5ac8f265-8646-430c-bca6-36655877def2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:23:58.191982 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:58.191955    4496 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ac8f265-8646-430c-bca6-36655877def2-plugins-config-volume\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:23:58.193962 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:58.193934    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ac8f265-8646-430c-bca6-36655877def2-kube-api-access-jq4wn" pod "5ac8f265-8646-430c-bca6-36655877def2" (UID: "5ac8f265-8646-430c-bca6-36655877def2"). InnerVolumeSpecName "kube-api-access-jq4wn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:23:58.198854 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:58.198834    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ed15c1-57bddc8787-l7kbv"
Nov 02 03:23:58.293122 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:58.293030    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9b6d8fd-c404-4c32-a6ae-8ba7b48314e3-kube-api-access-9mm2s\" (UniqueName: \"kubernetes.io/projected/f9b6d8fd-c404-4c32-a6ae-8ba7b48314e3-kube-api-access-9mm2s\") pod \"f9b6d8fd-c404-4c32-a6ae-8ba7b48314e3\" (UID: \"f9b6d8fd-c404-4c32-a6ae-8ba7b48314e3\") "
Nov 02 03:23:58.293122 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:58.293075    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cf48f92-69f5-47aa-8163-26f2d6df1921-kube-api-access-7xtkw\" (UniqueName: \"kubernetes.io/projected/4cf48f92-69f5-47aa-8163-26f2d6df1921-kube-api-access-7xtkw\") pod \"4cf48f92-69f5-47aa-8163-26f2d6df1921\" (UID: \"4cf48f92-69f5-47aa-8163-26f2d6df1921\") "
Nov 02 03:23:58.293122 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:58.293106    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4cf48f92-69f5-47aa-8163-26f2d6df1921-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cf48f92-69f5-47aa-8163-26f2d6df1921-plugins-config-volume\") pod \"4cf48f92-69f5-47aa-8163-26f2d6df1921\" (UID: \"4cf48f92-69f5-47aa-8163-26f2d6df1921\") "
Nov 02 03:23:58.293471 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:58.293276    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9b6d8fd-c404-4c32-a6ae-8ba7b48314e3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9b6d8fd-c404-4c32-a6ae-8ba7b48314e3-maas-api-openshift-ca\") pod \"f9b6d8fd-c404-4c32-a6ae-8ba7b48314e3\" (UID: \"f9b6d8fd-c404-4c32-a6ae-8ba7b48314e3\") "
Nov 02 03:23:58.293599 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:58.293573    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cf48f92-69f5-47aa-8163-26f2d6df1921-plugins-config-volume" pod "4cf48f92-69f5-47aa-8163-26f2d6df1921" (UID: "4cf48f92-69f5-47aa-8163-26f2d6df1921"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:23:58.293665 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:58.293602    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9b6d8fd-c404-4c32-a6ae-8ba7b48314e3-maas-api-openshift-ca" pod "f9b6d8fd-c404-4c32-a6ae-8ba7b48314e3" (UID: "f9b6d8fd-c404-4c32-a6ae-8ba7b48314e3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:23:58.293748 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:58.293734    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9b6d8fd-c404-4c32-a6ae-8ba7b48314e3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9b6d8fd-c404-4c32-a6ae-8ba7b48314e3-maas-api-xks-ca\") pod \"f9b6d8fd-c404-4c32-a6ae-8ba7b48314e3\" (UID: \"f9b6d8fd-c404-4c32-a6ae-8ba7b48314e3\") "
Nov 02 03:23:58.294346 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:58.294322    4496 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cf48f92-69f5-47aa-8163-26f2d6df1921-plugins-config-volume\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:23:58.294346 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:58.294347    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9b6d8fd-c404-4c32-a6ae-8ba7b48314e3-maas-api-openshift-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:23:58.294499 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:58.294361    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jq4wn\" (UniqueName: \"kubernetes.io/projected/5ac8f265-8646-430c-bca6-36655877def2-kube-api-access-jq4wn\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:23:58.295863 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:58.295841    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cf48f92-69f5-47aa-8163-26f2d6df1921-kube-api-access-7xtkw" pod "4cf48f92-69f5-47aa-8163-26f2d6df1921" (UID: "4cf48f92-69f5-47aa-8163-26f2d6df1921"). InnerVolumeSpecName "kube-api-access-7xtkw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:23:58.296148 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:58.296127    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9b6d8fd-c404-4c32-a6ae-8ba7b48314e3-maas-api-xks-ca" pod "f9b6d8fd-c404-4c32-a6ae-8ba7b48314e3" (UID: "f9b6d8fd-c404-4c32-a6ae-8ba7b48314e3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 03:23:58.296354 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:58.296328    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9b6d8fd-c404-4c32-a6ae-8ba7b48314e3-kube-api-access-9mm2s" pod "f9b6d8fd-c404-4c32-a6ae-8ba7b48314e3" (UID: "f9b6d8fd-c404-4c32-a6ae-8ba7b48314e3"). InnerVolumeSpecName "kube-api-access-9mm2s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:23:58.395982 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:58.395943    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mm2s\" (UniqueName: \"kubernetes.io/projected/f9b6d8fd-c404-4c32-a6ae-8ba7b48314e3-kube-api-access-9mm2s\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:23:58.395982 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:58.395981    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xtkw\" (UniqueName: \"kubernetes.io/projected/4cf48f92-69f5-47aa-8163-26f2d6df1921-kube-api-access-7xtkw\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:23:58.396223 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:58.395995    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9b6d8fd-c404-4c32-a6ae-8ba7b48314e3-maas-api-xks-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:23:58.617900 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:58.617808    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-lm5rm"]
Nov 02 03:23:58.618110 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:58.618032    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-lm5rm" podUID="05c9d201-5455-4a37-8382-8188a407e044" containerName="istio-proxy" containerID="cri-o://598a2e262bba3e5d2d25036e801009386eb99ca593fe30e905e2959a58ba48d1" gracePeriod=30
Nov 02 03:23:58.904779 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:58.904698    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-ed15c1-56c5d6557c-g5df2" event={"ID":"5ac8f265-8646-430c-bca6-36655877def2","Type":"ContainerDied","Data":"d44f6ea2f54188781b12c3e5ff3ae039014e44dcba6bbe53ccd1daa24e38aa2d"}
Nov 02 03:23:58.904779 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:58.904768    4496 scope.go:122] "RemoveContainer" containerID="7240d587c48d2d15da42e1dc5d035be3e2e6eb681b15ea42c5a440183f088e14"
Nov 02 03:23:58.905295 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:58.904704    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-ed15c1-56c5d6557c-g5df2"
Nov 02 03:23:58.906493 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:58.906466    4496 generic.go:356] "Generic (PLEG): container finished" podID="4cf48f92-69f5-47aa-8163-26f2d6df1921" containerID="0d00e0c98d00b1689c7313b1a401b1a59eddbfacd71fd0746e46caae0cd8b578" exitCode=0
Nov 02 03:23:58.906629 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:58.906498    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ed15c1-57bddc8787-l7kbv"
Nov 02 03:23:58.906629 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:58.906595    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ed15c1-57bddc8787-l7kbv" event={"ID":"4cf48f92-69f5-47aa-8163-26f2d6df1921","Type":"ContainerDied","Data":"0d00e0c98d00b1689c7313b1a401b1a59eddbfacd71fd0746e46caae0cd8b578"}
Nov 02 03:23:58.906753 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:58.906648    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ed15c1-57bddc8787-l7kbv" event={"ID":"4cf48f92-69f5-47aa-8163-26f2d6df1921","Type":"ContainerDied","Data":"44d1f3452f4129b602804b1b9ea25b8c677b356633a93af3d24d81c0d2d526f9"}
Nov 02 03:23:58.912411 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:58.912391    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0d26af-7m7t5"
Nov 02 03:23:58.912542 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:58.912426    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0d26af-7m7t5" event={"ID":"f9b6d8fd-c404-4c32-a6ae-8ba7b48314e3","Type":"ContainerDied","Data":"d538cafff563c89cdaa08217fad458e71aae3f09f7b7c174745f1fbaed99df9e"}
Nov 02 03:23:58.912542 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:58.912465    4496 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d538cafff563c89cdaa08217fad458e71aae3f09f7b7c174745f1fbaed99df9e"
Nov 02 03:23:58.918320 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:58.918296    4496 scope.go:122] "RemoveContainer" containerID="0d00e0c98d00b1689c7313b1a401b1a59eddbfacd71fd0746e46caae0cd8b578"
Nov 02 03:23:58.930275 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:58.930259    4496 scope.go:122] "RemoveContainer" containerID="0d00e0c98d00b1689c7313b1a401b1a59eddbfacd71fd0746e46caae0cd8b578"
Nov 02 03:23:58.930550 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:23:58.930529    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d00e0c98d00b1689c7313b1a401b1a59eddbfacd71fd0746e46caae0cd8b578\": container with ID starting with 0d00e0c98d00b1689c7313b1a401b1a59eddbfacd71fd0746e46caae0cd8b578 not found: ID does not exist" containerID="0d00e0c98d00b1689c7313b1a401b1a59eddbfacd71fd0746e46caae0cd8b578"
Nov 02 03:23:58.930601 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:58.930557    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d00e0c98d00b1689c7313b1a401b1a59eddbfacd71fd0746e46caae0cd8b578"} err="failed to get container status \"0d00e0c98d00b1689c7313b1a401b1a59eddbfacd71fd0746e46caae0cd8b578\": rpc error: code = NotFound desc = could not find container \"0d00e0c98d00b1689c7313b1a401b1a59eddbfacd71fd0746e46caae0cd8b578\": container with ID starting with 0d00e0c98d00b1689c7313b1a401b1a59eddbfacd71fd0746e46caae0cd8b578 not found: ID does not exist"
Nov 02 03:23:59.073173 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:59.073135    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ed15c1-57bddc8787-l7kbv"]
Nov 02 03:23:59.079696 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:59.079662    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ed15c1-57bddc8787-l7kbv"]
Nov 02 03:23:59.190283 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:59.190205    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ed15c1-56c5d6557c-g5df2"]
Nov 02 03:23:59.194674 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:59.194652    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ed15c1-56c5d6557c-g5df2"]
Nov 02 03:23:59.579941 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:59.579901    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cf48f92-69f5-47aa-8163-26f2d6df1921" path="/var/lib/kubelet/pods/4cf48f92-69f5-47aa-8163-26f2d6df1921/volumes"
Nov 02 03:23:59.580621 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:23:59.580597    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ac8f265-8646-430c-bca6-36655877def2" path="/var/lib/kubelet/pods/5ac8f265-8646-430c-bca6-36655877def2/volumes"
Nov 02 03:24:00.617950 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:00.617911    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c170e9-67fcf76c9-f4kkt"]
Nov 02 03:24:00.622884 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:00.619294    4496 cpu_manager.go:414] "containerMap: removing container" podUID="5ac8f265-8646-430c-bca6-36655877def2" containerName="payload-processing"
Nov 02 03:24:00.622884 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:00.619314    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ac8f265-8646-430c-bca6-36655877def2" containerName="payload-processing"
Nov 02 03:24:00.622884 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:00.619340    4496 cpu_manager.go:414] "containerMap: removing container" podUID="f9b6d8fd-c404-4c32-a6ae-8ba7b48314e3" containerName="revoke-keys"
Nov 02 03:24:00.622884 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:00.619345    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9b6d8fd-c404-4c32-a6ae-8ba7b48314e3" containerName="revoke-keys"
Nov 02 03:24:00.622884 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:00.619356    4496 cpu_manager.go:414] "containerMap: removing container" podUID="4cf48f92-69f5-47aa-8163-26f2d6df1921" containerName="payload-pre-processing"
Nov 02 03:24:00.622884 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:00.619362    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cf48f92-69f5-47aa-8163-26f2d6df1921" containerName="payload-pre-processing"
Nov 02 03:24:00.622884 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:00.619515    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ac8f265-8646-430c-bca6-36655877def2" containerName="payload-processing"
Nov 02 03:24:00.622884 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:00.619525    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9b6d8fd-c404-4c32-a6ae-8ba7b48314e3" containerName="revoke-keys"
Nov 02 03:24:00.622884 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:00.619532    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cf48f92-69f5-47aa-8163-26f2d6df1921" containerName="payload-pre-processing"
Nov 02 03:24:00.635052 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:00.632701    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-c170e9-67fcf76c9-f4kkt"
Nov 02 03:24:00.697111 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:00.689214    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-c170e9\""
Nov 02 03:24:00.697111 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:00.689675    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c170e9-67fcf76c9-f4kkt"]
Nov 02 03:24:00.697111 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:00.690085    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-c170e9-dockercfg-wbvrv\""
Nov 02 03:24:00.745385 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:00.745230    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhr4z\" (UniqueName: \"kubernetes.io/projected/eaa7c285-859b-4a5c-9f04-cc38b3164f2e-kube-api-access-lhr4z\") pod \"payload-processing-e2e-api-b-c170e9-67fcf76c9-f4kkt\" (UID: \"eaa7c285-859b-4a5c-9f04-cc38b3164f2e\") " pod="openshift-ingress/payload-processing-e2e-api-b-c170e9-67fcf76c9-f4kkt"
Nov 02 03:24:00.745385 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:00.745374    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eaa7c285-859b-4a5c-9f04-cc38b3164f2e-plugins-config-volume\") pod \"payload-processing-e2e-api-b-c170e9-67fcf76c9-f4kkt\" (UID: \"eaa7c285-859b-4a5c-9f04-cc38b3164f2e\") " pod="openshift-ingress/payload-processing-e2e-api-b-c170e9-67fcf76c9-f4kkt"
Nov 02 03:24:00.847107 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:00.847071    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhr4z\" (UniqueName: \"kubernetes.io/projected/eaa7c285-859b-4a5c-9f04-cc38b3164f2e-kube-api-access-lhr4z\") pod \"payload-processing-e2e-api-b-c170e9-67fcf76c9-f4kkt\" (UID: \"eaa7c285-859b-4a5c-9f04-cc38b3164f2e\") " pod="openshift-ingress/payload-processing-e2e-api-b-c170e9-67fcf76c9-f4kkt"
Nov 02 03:24:00.847280 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:00.847181    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eaa7c285-859b-4a5c-9f04-cc38b3164f2e-plugins-config-volume\") pod \"payload-processing-e2e-api-b-c170e9-67fcf76c9-f4kkt\" (UID: \"eaa7c285-859b-4a5c-9f04-cc38b3164f2e\") " pod="openshift-ingress/payload-processing-e2e-api-b-c170e9-67fcf76c9-f4kkt"
Nov 02 03:24:00.848131 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:00.848104    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eaa7c285-859b-4a5c-9f04-cc38b3164f2e-plugins-config-volume\") pod \"payload-processing-e2e-api-b-c170e9-67fcf76c9-f4kkt\" (UID: \"eaa7c285-859b-4a5c-9f04-cc38b3164f2e\") " pod="openshift-ingress/payload-processing-e2e-api-b-c170e9-67fcf76c9-f4kkt"
Nov 02 03:24:00.904177 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:00.904115    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhr4z\" (UniqueName: \"kubernetes.io/projected/eaa7c285-859b-4a5c-9f04-cc38b3164f2e-kube-api-access-lhr4z\") pod \"payload-processing-e2e-api-b-c170e9-67fcf76c9-f4kkt\" (UID: \"eaa7c285-859b-4a5c-9f04-cc38b3164f2e\") " pod="openshift-ingress/payload-processing-e2e-api-b-c170e9-67fcf76c9-f4kkt"
Nov 02 03:24:00.972408 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:00.972379    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-c170e9-67fcf76c9-f4kkt"
Nov 02 03:24:01.359681 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:01.359618    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c170e9-67fcf76c9-f4kkt"]
Nov 02 03:24:01.360294 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:24:01.360262    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaa7c285_859b_4a5c_9f04_cc38b3164f2e.slice/crio-dbcda1d1bd69c317a59a82cdf7f81c4ecfa9079864275e87aab2b8fe8765a3ae WatchSource:0}: Error finding container dbcda1d1bd69c317a59a82cdf7f81c4ecfa9079864275e87aab2b8fe8765a3ae: Status 404 returned error can't find the container with id dbcda1d1bd69c317a59a82cdf7f81c4ecfa9079864275e87aab2b8fe8765a3ae
Nov 02 03:24:01.945299 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:01.945261    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-c170e9-67fcf76c9-f4kkt" event={"ID":"eaa7c285-859b-4a5c-9f04-cc38b3164f2e","Type":"ContainerStarted","Data":"6c8ed99d61ccd8c64e8649ce66d757394a6c159fc0a91d0b252ec12a18d7bfa1"}
Nov 02 03:24:01.945788 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:01.945316    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-c170e9-67fcf76c9-f4kkt" event={"ID":"eaa7c285-859b-4a5c-9f04-cc38b3164f2e","Type":"ContainerStarted","Data":"dbcda1d1bd69c317a59a82cdf7f81c4ecfa9079864275e87aab2b8fe8765a3ae"}
Nov 02 03:24:01.945788 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:01.945395    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-c170e9-67fcf76c9-f4kkt"
Nov 02 03:24:02.069503 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:02.069433    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-c170e9-67fcf76c9-f4kkt" podStartSLOduration=2.0694150159999998 podStartE2EDuration="2.069415016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:24:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:24:02.063056544 +0000 UTC m=+2182.641663925" watchObservedRunningTime="2026-11-02 03:24:02.069415016 +0000 UTC m=+2182.648022397"
Nov 02 03:24:03.529625 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:24:03.529593    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 03:24:04.152258 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:04.145033    4496 generic.go:356] "Generic (PLEG): container finished" podID="05c9d201-5455-4a37-8382-8188a407e044" containerID="598a2e262bba3e5d2d25036e801009386eb99ca593fe30e905e2959a58ba48d1" exitCode=0
Nov 02 03:24:04.152258 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:04.145421    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-lm5rm" event={"ID":"05c9d201-5455-4a37-8382-8188a407e044","Type":"ContainerDied","Data":"598a2e262bba3e5d2d25036e801009386eb99ca593fe30e905e2959a58ba48d1"}
Nov 02 03:24:04.196038 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:04.193559    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-lm5rm"
Nov 02 03:24:04.385550 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:04.385451    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05c9d201-5455-4a37-8382-8188a407e044-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05c9d201-5455-4a37-8382-8188a407e044-workload-certs\") pod \"05c9d201-5455-4a37-8382-8188a407e044\" (UID: \"05c9d201-5455-4a37-8382-8188a407e044\") "
Nov 02 03:24:04.385699 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:04.385568    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05c9d201-5455-4a37-8382-8188a407e044-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05c9d201-5455-4a37-8382-8188a407e044-workload-socket\") pod \"05c9d201-5455-4a37-8382-8188a407e044\" (UID: \"05c9d201-5455-4a37-8382-8188a407e044\") "
Nov 02 03:24:04.385699 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:04.385667    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05c9d201-5455-4a37-8382-8188a407e044-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05c9d201-5455-4a37-8382-8188a407e044-istio-data\") pod \"05c9d201-5455-4a37-8382-8188a407e044\" (UID: \"05c9d201-5455-4a37-8382-8188a407e044\") "
Nov 02 03:24:04.385770 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:04.385744    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/05c9d201-5455-4a37-8382-8188a407e044-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05c9d201-5455-4a37-8382-8188a407e044-istio-podinfo\") pod \"05c9d201-5455-4a37-8382-8188a407e044\" (UID: \"05c9d201-5455-4a37-8382-8188a407e044\") "
Nov 02 03:24:04.385805 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:04.385796    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05c9d201-5455-4a37-8382-8188a407e044-istio-token\" (UniqueName: \"kubernetes.io/projected/05c9d201-5455-4a37-8382-8188a407e044-istio-token\") pod \"05c9d201-5455-4a37-8382-8188a407e044\" (UID: \"05c9d201-5455-4a37-8382-8188a407e044\") "
Nov 02 03:24:04.385964 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:04.385943    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05c9d201-5455-4a37-8382-8188a407e044-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05c9d201-5455-4a37-8382-8188a407e044-credential-socket\") pod \"05c9d201-5455-4a37-8382-8188a407e044\" (UID: \"05c9d201-5455-4a37-8382-8188a407e044\") "
Nov 02 03:24:04.386082 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:04.385997    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05c9d201-5455-4a37-8382-8188a407e044-kube-api-access-c5zjq\" (UniqueName: \"kubernetes.io/projected/05c9d201-5455-4a37-8382-8188a407e044-kube-api-access-c5zjq\") pod \"05c9d201-5455-4a37-8382-8188a407e044\" (UID: \"05c9d201-5455-4a37-8382-8188a407e044\") "
Nov 02 03:24:04.386160 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:04.386098    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05c9d201-5455-4a37-8382-8188a407e044-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05c9d201-5455-4a37-8382-8188a407e044-istiod-ca-cert\") pod \"05c9d201-5455-4a37-8382-8188a407e044\" (UID: \"05c9d201-5455-4a37-8382-8188a407e044\") "
Nov 02 03:24:04.386160 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:04.386132    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05c9d201-5455-4a37-8382-8188a407e044-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05c9d201-5455-4a37-8382-8188a407e044-istio-envoy\") pod \"05c9d201-5455-4a37-8382-8188a407e044\" (UID: \"05c9d201-5455-4a37-8382-8188a407e044\") "
Nov 02 03:24:04.392477 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:04.392439    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05c9d201-5455-4a37-8382-8188a407e044-istio-token" pod "05c9d201-5455-4a37-8382-8188a407e044" (UID: "05c9d201-5455-4a37-8382-8188a407e044"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:24:04.392747 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:04.392723    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05c9d201-5455-4a37-8382-8188a407e044-credential-socket" pod "05c9d201-5455-4a37-8382-8188a407e044" (UID: "05c9d201-5455-4a37-8382-8188a407e044"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:24:04.392850 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:04.392800    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05c9d201-5455-4a37-8382-8188a407e044-istio-envoy" pod "05c9d201-5455-4a37-8382-8188a407e044" (UID: "05c9d201-5455-4a37-8382-8188a407e044"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:24:04.393259 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:04.393079    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05c9d201-5455-4a37-8382-8188a407e044-workload-socket" pod "05c9d201-5455-4a37-8382-8188a407e044" (UID: "05c9d201-5455-4a37-8382-8188a407e044"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:24:04.393621 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:04.393420    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05c9d201-5455-4a37-8382-8188a407e044-workload-certs" pod "05c9d201-5455-4a37-8382-8188a407e044" (UID: "05c9d201-5455-4a37-8382-8188a407e044"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:24:04.393820 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:04.393768    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05c9d201-5455-4a37-8382-8188a407e044-istio-data" pod "05c9d201-5455-4a37-8382-8188a407e044" (UID: "05c9d201-5455-4a37-8382-8188a407e044"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:24:04.394127 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:04.394090    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05c9d201-5455-4a37-8382-8188a407e044-istiod-ca-cert" pod "05c9d201-5455-4a37-8382-8188a407e044" (UID: "05c9d201-5455-4a37-8382-8188a407e044"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:24:04.394872 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:04.394781    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/05c9d201-5455-4a37-8382-8188a407e044-istio-podinfo" pod "05c9d201-5455-4a37-8382-8188a407e044" (UID: "05c9d201-5455-4a37-8382-8188a407e044"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 03:24:04.400179 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:04.400140    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05c9d201-5455-4a37-8382-8188a407e044-kube-api-access-c5zjq" pod "05c9d201-5455-4a37-8382-8188a407e044" (UID: "05c9d201-5455-4a37-8382-8188a407e044"). InnerVolumeSpecName "kube-api-access-c5zjq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:24:04.488789 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:04.488755    4496 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05c9d201-5455-4a37-8382-8188a407e044-istio-token\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:24:04.488789 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:04.488786    4496 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05c9d201-5455-4a37-8382-8188a407e044-credential-socket\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:24:04.488789 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:04.488797    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5zjq\" (UniqueName: \"kubernetes.io/projected/05c9d201-5455-4a37-8382-8188a407e044-kube-api-access-c5zjq\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:24:04.489037 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:04.488806    4496 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05c9d201-5455-4a37-8382-8188a407e044-istiod-ca-cert\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:24:04.489037 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:04.488815    4496 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05c9d201-5455-4a37-8382-8188a407e044-istio-envoy\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:24:04.489037 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:04.488823    4496 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05c9d201-5455-4a37-8382-8188a407e044-workload-certs\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:24:04.489037 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:04.488833    4496 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05c9d201-5455-4a37-8382-8188a407e044-workload-socket\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:24:04.489037 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:04.488843    4496 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05c9d201-5455-4a37-8382-8188a407e044-istio-data\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:24:04.489037 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:04.488851    4496 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05c9d201-5455-4a37-8382-8188a407e044-istio-podinfo\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:24:05.156643 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:05.156606    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-lm5rm"
Nov 02 03:24:05.156643 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:05.156639    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-lm5rm" event={"ID":"05c9d201-5455-4a37-8382-8188a407e044","Type":"ContainerDied","Data":"769d29ff81e1b41a354008a53565d534fa59da400e7a847ead2675a7d0f701ef"}
Nov 02 03:24:05.157264 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:05.156706    4496 scope.go:122] "RemoveContainer" containerID="598a2e262bba3e5d2d25036e801009386eb99ca593fe30e905e2959a58ba48d1"
Nov 02 03:24:05.377591 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:05.377547    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-lm5rm"]
Nov 02 03:24:05.381955 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:05.381923    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-lm5rm"]
Nov 02 03:24:05.577445 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:05.577405    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05c9d201-5455-4a37-8382-8188a407e044" path="/var/lib/kubelet/pods/05c9d201-5455-4a37-8382-8188a407e044/volumes"
Nov 02 03:24:06.004205 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:06.004174    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0d26af-59668b6f65-dz2f7"]
Nov 02 03:24:06.119256 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:06.119194    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0d26af-64f5446f47-qnwqh"]
Nov 02 03:24:11.004624 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:11.004555    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-0d26af-59668b6f65-dz2f7" podUID="d3a1ea4e-eb3e-462b-900c-ea2181bb88d9" containerName="payload-processing" containerID="cri-o://54673afff06f24dbb38b20f242cc2106831690ffc0c35691cf3c26f382abfefa" gracePeriod=25
Nov 02 03:24:11.120428 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:11.120368    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0d26af-64f5446f47-qnwqh" podUID="15d2020a-9cef-4fa1-b3f5-7d1f2f1a9bcf" containerName="payload-pre-processing" containerID="cri-o://2455831f6d1cd11a7a2862b497e58693a935bbb0e8755e7f9ffd9f77659b4988" gracePeriod=25
Nov 02 03:24:11.232904 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:11.224889    4496 generic.go:356] "Generic (PLEG): container finished" podID="d3a1ea4e-eb3e-462b-900c-ea2181bb88d9" containerID="54673afff06f24dbb38b20f242cc2106831690ffc0c35691cf3c26f382abfefa" exitCode=0
Nov 02 03:24:11.232904 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:11.225246    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-0d26af-59668b6f65-dz2f7" event={"ID":"d3a1ea4e-eb3e-462b-900c-ea2181bb88d9","Type":"ContainerDied","Data":"54673afff06f24dbb38b20f242cc2106831690ffc0c35691cf3c26f382abfefa"}
Nov 02 03:24:11.468806 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:11.461978    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-0d26af-59668b6f65-dz2f7"
Nov 02 03:24:11.562102 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:11.562065    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3a1ea4e-eb3e-462b-900c-ea2181bb88d9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3a1ea4e-eb3e-462b-900c-ea2181bb88d9-plugins-config-volume\") pod \"d3a1ea4e-eb3e-462b-900c-ea2181bb88d9\" (UID: \"d3a1ea4e-eb3e-462b-900c-ea2181bb88d9\") "
Nov 02 03:24:11.562343 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:11.562324    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3a1ea4e-eb3e-462b-900c-ea2181bb88d9-kube-api-access-fpd9g\" (UniqueName: \"kubernetes.io/projected/d3a1ea4e-eb3e-462b-900c-ea2181bb88d9-kube-api-access-fpd9g\") pod \"d3a1ea4e-eb3e-462b-900c-ea2181bb88d9\" (UID: \"d3a1ea4e-eb3e-462b-900c-ea2181bb88d9\") "
Nov 02 03:24:11.562899 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:11.562860    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3a1ea4e-eb3e-462b-900c-ea2181bb88d9-plugins-config-volume" pod "d3a1ea4e-eb3e-462b-900c-ea2181bb88d9" (UID: "d3a1ea4e-eb3e-462b-900c-ea2181bb88d9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:24:11.563094 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:11.563071    4496 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3a1ea4e-eb3e-462b-900c-ea2181bb88d9-plugins-config-volume\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:24:11.565676 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:11.565654    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3a1ea4e-eb3e-462b-900c-ea2181bb88d9-kube-api-access-fpd9g" pod "d3a1ea4e-eb3e-462b-900c-ea2181bb88d9" (UID: "d3a1ea4e-eb3e-462b-900c-ea2181bb88d9"). InnerVolumeSpecName "kube-api-access-fpd9g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:24:11.573795 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:11.573769    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0d26af-64f5446f47-qnwqh"
Nov 02 03:24:11.648276 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:24:11.648201    4496 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3a1ea4e_eb3e_462b_900c_ea2181bb88d9.slice/crio-34bc285a738bdbb8cd11ce1fce15f8ce10c34c9ce899b41f2f45ab21c2021b1e\": RecentStats: unable to find data in memory cache]"
Nov 02 03:24:11.664683 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:11.664650    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15d2020a-9cef-4fa1-b3f5-7d1f2f1a9bcf-kube-api-access-vcv7f\" (UniqueName: \"kubernetes.io/projected/15d2020a-9cef-4fa1-b3f5-7d1f2f1a9bcf-kube-api-access-vcv7f\") pod \"15d2020a-9cef-4fa1-b3f5-7d1f2f1a9bcf\" (UID: \"15d2020a-9cef-4fa1-b3f5-7d1f2f1a9bcf\") "
Nov 02 03:24:11.664806 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:11.664793    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15d2020a-9cef-4fa1-b3f5-7d1f2f1a9bcf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15d2020a-9cef-4fa1-b3f5-7d1f2f1a9bcf-plugins-config-volume\") pod \"15d2020a-9cef-4fa1-b3f5-7d1f2f1a9bcf\" (UID: \"15d2020a-9cef-4fa1-b3f5-7d1f2f1a9bcf\") "
Nov 02 03:24:11.665271 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:11.665240    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15d2020a-9cef-4fa1-b3f5-7d1f2f1a9bcf-plugins-config-volume" pod "15d2020a-9cef-4fa1-b3f5-7d1f2f1a9bcf" (UID: "15d2020a-9cef-4fa1-b3f5-7d1f2f1a9bcf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:24:11.665602 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:11.665579    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpd9g\" (UniqueName: \"kubernetes.io/projected/d3a1ea4e-eb3e-462b-900c-ea2181bb88d9-kube-api-access-fpd9g\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:24:11.665698 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:11.665606    4496 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15d2020a-9cef-4fa1-b3f5-7d1f2f1a9bcf-plugins-config-volume\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:24:11.666881 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:11.666858    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15d2020a-9cef-4fa1-b3f5-7d1f2f1a9bcf-kube-api-access-vcv7f" pod "15d2020a-9cef-4fa1-b3f5-7d1f2f1a9bcf" (UID: "15d2020a-9cef-4fa1-b3f5-7d1f2f1a9bcf"). InnerVolumeSpecName "kube-api-access-vcv7f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:24:11.767664 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:11.767628    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcv7f\" (UniqueName: \"kubernetes.io/projected/15d2020a-9cef-4fa1-b3f5-7d1f2f1a9bcf-kube-api-access-vcv7f\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:24:12.235756 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:12.235723    4496 generic.go:356] "Generic (PLEG): container finished" podID="15d2020a-9cef-4fa1-b3f5-7d1f2f1a9bcf" containerID="2455831f6d1cd11a7a2862b497e58693a935bbb0e8755e7f9ffd9f77659b4988" exitCode=0
Nov 02 03:24:12.236271 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:12.235769    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0d26af-64f5446f47-qnwqh"
Nov 02 03:24:12.236271 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:12.235809    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0d26af-64f5446f47-qnwqh" event={"ID":"15d2020a-9cef-4fa1-b3f5-7d1f2f1a9bcf","Type":"ContainerDied","Data":"2455831f6d1cd11a7a2862b497e58693a935bbb0e8755e7f9ffd9f77659b4988"}
Nov 02 03:24:12.236271 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:12.235867    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0d26af-64f5446f47-qnwqh" event={"ID":"15d2020a-9cef-4fa1-b3f5-7d1f2f1a9bcf","Type":"ContainerDied","Data":"2d64d3df7d2c5f4608ccc032f276615247cad841848ef2e4775de003c295fcdc"}
Nov 02 03:24:12.236271 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:12.235901    4496 scope.go:122] "RemoveContainer" containerID="2455831f6d1cd11a7a2862b497e58693a935bbb0e8755e7f9ffd9f77659b4988"
Nov 02 03:24:12.237978 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:12.237953    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-0d26af-59668b6f65-dz2f7"
Nov 02 03:24:12.237978 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:12.237957    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-0d26af-59668b6f65-dz2f7" event={"ID":"d3a1ea4e-eb3e-462b-900c-ea2181bb88d9","Type":"ContainerDied","Data":"34bc285a738bdbb8cd11ce1fce15f8ce10c34c9ce899b41f2f45ab21c2021b1e"}
Nov 02 03:24:12.256853 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:12.256835    4496 scope.go:122] "RemoveContainer" containerID="2455831f6d1cd11a7a2862b497e58693a935bbb0e8755e7f9ffd9f77659b4988"
Nov 02 03:24:12.257246 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:24:12.257226    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2455831f6d1cd11a7a2862b497e58693a935bbb0e8755e7f9ffd9f77659b4988\": container with ID starting with 2455831f6d1cd11a7a2862b497e58693a935bbb0e8755e7f9ffd9f77659b4988 not found: ID does not exist" containerID="2455831f6d1cd11a7a2862b497e58693a935bbb0e8755e7f9ffd9f77659b4988"
Nov 02 03:24:12.257350 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:12.257252    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2455831f6d1cd11a7a2862b497e58693a935bbb0e8755e7f9ffd9f77659b4988"} err="failed to get container status \"2455831f6d1cd11a7a2862b497e58693a935bbb0e8755e7f9ffd9f77659b4988\": rpc error: code = NotFound desc = could not find container \"2455831f6d1cd11a7a2862b497e58693a935bbb0e8755e7f9ffd9f77659b4988\": container with ID starting with 2455831f6d1cd11a7a2862b497e58693a935bbb0e8755e7f9ffd9f77659b4988 not found: ID does not exist"
Nov 02 03:24:12.257350 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:12.257269    4496 scope.go:122] "RemoveContainer" containerID="54673afff06f24dbb38b20f242cc2106831690ffc0c35691cf3c26f382abfefa"
Nov 02 03:24:12.403921 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:12.403810    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0d26af-59668b6f65-dz2f7"]
Nov 02 03:24:12.454838 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:12.454809    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0d26af-59668b6f65-dz2f7"]
Nov 02 03:24:12.565649 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:12.565614    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0d26af-64f5446f47-qnwqh"]
Nov 02 03:24:12.570037 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:12.569991    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0d26af-64f5446f47-qnwqh"]
Nov 02 03:24:12.959216 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:12.959178    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-c170e9-67fcf76c9-f4kkt"
Nov 02 03:24:13.576445 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:13.576403    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15d2020a-9cef-4fa1-b3f5-7d1f2f1a9bcf" path="/var/lib/kubelet/pods/15d2020a-9cef-4fa1-b3f5-7d1f2f1a9bcf/volumes"
Nov 02 03:24:13.577139 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:13.577117    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3a1ea4e-eb3e-462b-900c-ea2181bb88d9" path="/var/lib/kubelet/pods/d3a1ea4e-eb3e-462b-900c-ea2181bb88d9/volumes"
Nov 02 03:24:27.051653 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:27.051562    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5165bfa2-s22rx"]
Nov 02 03:24:27.101530 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:27.101495    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5165bfa2-s22rx"]
Nov 02 03:24:27.576666 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:27.576631    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69be064c-f48a-40b8-8f62-011b916949ee" path="/var/lib/kubelet/pods/69be064c-f48a-40b8-8f62-011b916949ee/volumes"
Nov 02 03:24:33.047517 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:33.047477    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f9a40ef-qrxrd"]
Nov 02 03:24:33.097773 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:33.097738    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f9a40ef-qrxrd"]
Nov 02 03:24:33.553846 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:24:33.553805    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 03:24:33.579052 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:33.578987    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08f82f8d-8213-423d-8df8-08ae031d0f7c" path="/var/lib/kubelet/pods/08f82f8d-8213-423d-8df8-08ae031d0f7c/volumes"
Nov 02 03:24:36.334009 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:36.333973    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b4a230-8w4qw"]
Nov 02 03:24:36.334777 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:36.334761    4496 cpu_manager.go:414] "containerMap: removing container" podUID="d3a1ea4e-eb3e-462b-900c-ea2181bb88d9" containerName="payload-processing"
Nov 02 03:24:36.334777 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:36.334778    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3a1ea4e-eb3e-462b-900c-ea2181bb88d9" containerName="payload-processing"
Nov 02 03:24:36.334887 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:36.334795    4496 cpu_manager.go:414] "containerMap: removing container" podUID="15d2020a-9cef-4fa1-b3f5-7d1f2f1a9bcf" containerName="payload-pre-processing"
Nov 02 03:24:36.334887 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:36.334802    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15d2020a-9cef-4fa1-b3f5-7d1f2f1a9bcf" containerName="payload-pre-processing"
Nov 02 03:24:36.334887 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:36.334833    4496 cpu_manager.go:414] "containerMap: removing container" podUID="05c9d201-5455-4a37-8382-8188a407e044" containerName="istio-proxy"
Nov 02 03:24:36.334887 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:36.334838    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05c9d201-5455-4a37-8382-8188a407e044" containerName="istio-proxy"
Nov 02 03:24:36.335010 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:36.334956    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3a1ea4e-eb3e-462b-900c-ea2181bb88d9" containerName="payload-processing"
Nov 02 03:24:36.335010 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:36.334963    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="15d2020a-9cef-4fa1-b3f5-7d1f2f1a9bcf" containerName="payload-pre-processing"
Nov 02 03:24:36.335010 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:36.334974    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="05c9d201-5455-4a37-8382-8188a407e044" containerName="istio-proxy"
Nov 02 03:24:36.338565 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:36.338549    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b4a230-8w4qw"
Nov 02 03:24:36.395158 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:36.395128    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b4a230-8w4qw"]
Nov 02 03:24:36.439162 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:36.439107    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 03:24:36.439341 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:36.439313    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9z5bk\""
Nov 02 03:24:36.439411 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:36.439352    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 03:24:36.439411 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:36.439373    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 03:24:36.526484 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:36.526447    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aee8a3f1-6405-467c-8b98-84ca26618ffb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-b4a230-8w4qw\" (UID: \"aee8a3f1-6405-467c-8b98-84ca26618ffb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b4a230-8w4qw"
Nov 02 03:24:36.526679 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:36.526512    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsphm\" (UniqueName: \"kubernetes.io/projected/aee8a3f1-6405-467c-8b98-84ca26618ffb-kube-api-access-tsphm\") pod \"maas-api-revoke-keys-e2e-api-a-b4a230-8w4qw\" (UID: \"aee8a3f1-6405-467c-8b98-84ca26618ffb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b4a230-8w4qw"
Nov 02 03:24:36.526679 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:36.526610    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aee8a3f1-6405-467c-8b98-84ca26618ffb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-b4a230-8w4qw\" (UID: \"aee8a3f1-6405-467c-8b98-84ca26618ffb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b4a230-8w4qw"
Nov 02 03:24:36.628159 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:36.628081    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsphm\" (UniqueName: \"kubernetes.io/projected/aee8a3f1-6405-467c-8b98-84ca26618ffb-kube-api-access-tsphm\") pod \"maas-api-revoke-keys-e2e-api-a-b4a230-8w4qw\" (UID: \"aee8a3f1-6405-467c-8b98-84ca26618ffb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b4a230-8w4qw"
Nov 02 03:24:36.628159 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:36.628136    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aee8a3f1-6405-467c-8b98-84ca26618ffb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-b4a230-8w4qw\" (UID: \"aee8a3f1-6405-467c-8b98-84ca26618ffb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b4a230-8w4qw"
Nov 02 03:24:36.628405 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:36.628385    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aee8a3f1-6405-467c-8b98-84ca26618ffb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-b4a230-8w4qw\" (UID: \"aee8a3f1-6405-467c-8b98-84ca26618ffb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b4a230-8w4qw"
Nov 02 03:24:36.629000 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:36.628976    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aee8a3f1-6405-467c-8b98-84ca26618ffb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-b4a230-8w4qw\" (UID: \"aee8a3f1-6405-467c-8b98-84ca26618ffb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b4a230-8w4qw"
Nov 02 03:24:36.630642 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:36.630614    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aee8a3f1-6405-467c-8b98-84ca26618ffb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-b4a230-8w4qw\" (UID: \"aee8a3f1-6405-467c-8b98-84ca26618ffb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b4a230-8w4qw"
Nov 02 03:24:36.685550 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:36.685523    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsphm\" (UniqueName: \"kubernetes.io/projected/aee8a3f1-6405-467c-8b98-84ca26618ffb-kube-api-access-tsphm\") pod \"maas-api-revoke-keys-e2e-api-a-b4a230-8w4qw\" (UID: \"aee8a3f1-6405-467c-8b98-84ca26618ffb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b4a230-8w4qw"
Nov 02 03:24:36.952436 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:36.952358    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b4a230-8w4qw"
Nov 02 03:24:37.330789 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:37.330756    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b4a230-8w4qw"]
Nov 02 03:24:37.331575 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:24:37.331540    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaee8a3f1_6405_467c_8b98_84ca26618ffb.slice/crio-39bd8a55df8e93d8ba4690460f110824d3bf146c0f463a7334b40b5f4ca1d4f1 WatchSource:0}: Error finding container 39bd8a55df8e93d8ba4690460f110824d3bf146c0f463a7334b40b5f4ca1d4f1: Status 404 returned error can't find the container with id 39bd8a55df8e93d8ba4690460f110824d3bf146c0f463a7334b40b5f4ca1d4f1
Nov 02 03:24:37.505883 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:37.505843    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b4a230-8w4qw" event={"ID":"aee8a3f1-6405-467c-8b98-84ca26618ffb","Type":"ContainerStarted","Data":"39bd8a55df8e93d8ba4690460f110824d3bf146c0f463a7334b40b5f4ca1d4f1"}
Nov 02 03:24:38.515829 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:38.515794    4496 generic.go:356] "Generic (PLEG): container finished" podID="aee8a3f1-6405-467c-8b98-84ca26618ffb" containerID="500b65846cb429f34e4b661eb2980f3faa430efbcd9cad4a3bc3b0f46608513b" exitCode=0
Nov 02 03:24:38.516233 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:38.515829    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b4a230-8w4qw" event={"ID":"aee8a3f1-6405-467c-8b98-84ca26618ffb","Type":"ContainerDied","Data":"500b65846cb429f34e4b661eb2980f3faa430efbcd9cad4a3bc3b0f46608513b"}
Nov 02 03:24:39.699475 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:39.699449    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b4a230-8w4qw"
Nov 02 03:24:39.779403 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:39.779305    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aee8a3f1-6405-467c-8b98-84ca26618ffb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aee8a3f1-6405-467c-8b98-84ca26618ffb-maas-api-xks-ca\") pod \"aee8a3f1-6405-467c-8b98-84ca26618ffb\" (UID: \"aee8a3f1-6405-467c-8b98-84ca26618ffb\") "
Nov 02 03:24:39.779403 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:39.779375    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aee8a3f1-6405-467c-8b98-84ca26618ffb-kube-api-access-tsphm\" (UniqueName: \"kubernetes.io/projected/aee8a3f1-6405-467c-8b98-84ca26618ffb-kube-api-access-tsphm\") pod \"aee8a3f1-6405-467c-8b98-84ca26618ffb\" (UID: \"aee8a3f1-6405-467c-8b98-84ca26618ffb\") "
Nov 02 03:24:39.779708 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:39.779469    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aee8a3f1-6405-467c-8b98-84ca26618ffb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aee8a3f1-6405-467c-8b98-84ca26618ffb-maas-api-openshift-ca\") pod \"aee8a3f1-6405-467c-8b98-84ca26618ffb\" (UID: \"aee8a3f1-6405-467c-8b98-84ca26618ffb\") "
Nov 02 03:24:39.780290 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:39.780259    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aee8a3f1-6405-467c-8b98-84ca26618ffb-maas-api-openshift-ca" pod "aee8a3f1-6405-467c-8b98-84ca26618ffb" (UID: "aee8a3f1-6405-467c-8b98-84ca26618ffb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:24:39.782036 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:39.781988    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aee8a3f1-6405-467c-8b98-84ca26618ffb-maas-api-xks-ca" pod "aee8a3f1-6405-467c-8b98-84ca26618ffb" (UID: "aee8a3f1-6405-467c-8b98-84ca26618ffb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 03:24:39.782155 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:39.781989    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aee8a3f1-6405-467c-8b98-84ca26618ffb-kube-api-access-tsphm" pod "aee8a3f1-6405-467c-8b98-84ca26618ffb" (UID: "aee8a3f1-6405-467c-8b98-84ca26618ffb"). InnerVolumeSpecName "kube-api-access-tsphm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:24:39.881903 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:39.881863    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aee8a3f1-6405-467c-8b98-84ca26618ffb-maas-api-xks-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:24:39.881903 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:39.881903    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsphm\" (UniqueName: \"kubernetes.io/projected/aee8a3f1-6405-467c-8b98-84ca26618ffb-kube-api-access-tsphm\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:24:39.882134 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:39.881919    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aee8a3f1-6405-467c-8b98-84ca26618ffb-maas-api-openshift-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:24:40.534245 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:40.534207    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b4a230-8w4qw"
Nov 02 03:24:40.534444 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:40.534241    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b4a230-8w4qw" event={"ID":"aee8a3f1-6405-467c-8b98-84ca26618ffb","Type":"ContainerDied","Data":"39bd8a55df8e93d8ba4690460f110824d3bf146c0f463a7334b40b5f4ca1d4f1"}
Nov 02 03:24:40.534444 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:40.534297    4496 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="39bd8a55df8e93d8ba4690460f110824d3bf146c0f463a7334b40b5f4ca1d4f1"
Nov 02 03:24:40.731930 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:40.731899    4496 scope.go:122] "RemoveContainer" containerID="eb86f9650e2ab1c0cd3176ca5ea0aca8f0d5aa494a72ebb231a616615f33a576"
Nov 02 03:24:40.745215 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:40.745189    4496 scope.go:122] "RemoveContainer" containerID="baafaace65f1db2d444fdf3fff77de463f2ae1fa71435e82d8d40f0e5c224503"
Nov 02 03:24:40.757690 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:40.757666    4496 scope.go:122] "RemoveContainer" containerID="8959e0d0edc25e0de08cecda67df9b27d7cbcc08ab09affc05dc8e21bb17bb25"
Nov 02 03:24:40.767401 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:40.767381    4496 scope.go:122] "RemoveContainer" containerID="369be73cdc34cca261cf0982987e2e7dc485000865fd2ac9cdf7847bc5d9bdbd"
Nov 02 03:24:43.050442 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:43.050405    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fde0b86e-pvbq5"]
Nov 02 03:24:43.098073 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:43.098033    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fde0b86e-pvbq5"]
Nov 02 03:24:43.583331 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:43.583294    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e174f62e-606b-4970-aa05-32df9c3e7fbd" path="/var/lib/kubelet/pods/e174f62e-606b-4970-aa05-32df9c3e7fbd/volumes"
Nov 02 03:24:44.045971 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:44.045936    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-330c7d4b-7jkx4"]
Nov 02 03:24:44.096852 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:44.096818    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-330c7d4b-7jkx4"]
Nov 02 03:24:45.577484 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:45.577422    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50032077-80e1-4c33-8dfd-faf513f9ecc8" path="/var/lib/kubelet/pods/50032077-80e1-4c33-8dfd-faf513f9ecc8/volumes"
Nov 02 03:24:48.480524 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:48.480486    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b4a230-7f49b69f9f-jfcml"]
Nov 02 03:24:48.599228 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:48.599173    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b4a230-5666cf5b6c-vv8cx"]
Nov 02 03:24:53.481146 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:53.481099    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-b4a230-7f49b69f9f-jfcml" podUID="54bac357-122d-43a3-909e-87f1cb110ffa" containerName="payload-processing" containerID="cri-o://8385a1531b982dfb9d00e307092cbae8bd514870b425a685ed91b288b1c062df" gracePeriod=25
Nov 02 03:24:53.600238 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:53.600200    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b4a230-5666cf5b6c-vv8cx" podUID="0dfde42a-42cb-482d-a667-6ee53e804104" containerName="payload-pre-processing" containerID="cri-o://471c15032821a5d41c47d97941e8bf0758308737148ba1e44c307b9d3bce5868" gracePeriod=25
Nov 02 03:24:53.727063 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:53.725922    4496 generic.go:356] "Generic (PLEG): container finished" podID="54bac357-122d-43a3-909e-87f1cb110ffa" containerID="8385a1531b982dfb9d00e307092cbae8bd514870b425a685ed91b288b1c062df" exitCode=0
Nov 02 03:24:53.727063 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:53.726224    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-b4a230-7f49b69f9f-jfcml" event={"ID":"54bac357-122d-43a3-909e-87f1cb110ffa","Type":"ContainerDied","Data":"8385a1531b982dfb9d00e307092cbae8bd514870b425a685ed91b288b1c062df"}
Nov 02 03:24:53.801395 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:53.801360    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-b4a230-7f49b69f9f-jfcml"
Nov 02 03:24:53.896130 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:53.896105    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b4a230-5666cf5b6c-vv8cx"
Nov 02 03:24:53.907404 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:53.907380    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0dfde42a-42cb-482d-a667-6ee53e804104-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0dfde42a-42cb-482d-a667-6ee53e804104-plugins-config-volume\") pod \"0dfde42a-42cb-482d-a667-6ee53e804104\" (UID: \"0dfde42a-42cb-482d-a667-6ee53e804104\") "
Nov 02 03:24:53.907537 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:53.907414    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0dfde42a-42cb-482d-a667-6ee53e804104-kube-api-access-gqn5r\" (UniqueName: \"kubernetes.io/projected/0dfde42a-42cb-482d-a667-6ee53e804104-kube-api-access-gqn5r\") pod \"0dfde42a-42cb-482d-a667-6ee53e804104\" (UID: \"0dfde42a-42cb-482d-a667-6ee53e804104\") "
Nov 02 03:24:53.907537 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:53.907496    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54bac357-122d-43a3-909e-87f1cb110ffa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54bac357-122d-43a3-909e-87f1cb110ffa-plugins-config-volume\") pod \"54bac357-122d-43a3-909e-87f1cb110ffa\" (UID: \"54bac357-122d-43a3-909e-87f1cb110ffa\") "
Nov 02 03:24:53.907616 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:53.907584    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54bac357-122d-43a3-909e-87f1cb110ffa-kube-api-access-bx5qn\" (UniqueName: \"kubernetes.io/projected/54bac357-122d-43a3-909e-87f1cb110ffa-kube-api-access-bx5qn\") pod \"54bac357-122d-43a3-909e-87f1cb110ffa\" (UID: \"54bac357-122d-43a3-909e-87f1cb110ffa\") "
Nov 02 03:24:53.908433 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:53.907872    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dfde42a-42cb-482d-a667-6ee53e804104-plugins-config-volume" pod "0dfde42a-42cb-482d-a667-6ee53e804104" (UID: "0dfde42a-42cb-482d-a667-6ee53e804104"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:24:53.908433 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:53.907929    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54bac357-122d-43a3-909e-87f1cb110ffa-plugins-config-volume" pod "54bac357-122d-43a3-909e-87f1cb110ffa" (UID: "54bac357-122d-43a3-909e-87f1cb110ffa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:24:53.908433 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:53.908315    4496 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0dfde42a-42cb-482d-a667-6ee53e804104-plugins-config-volume\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:24:53.908433 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:53.908339    4496 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54bac357-122d-43a3-909e-87f1cb110ffa-plugins-config-volume\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:24:53.909657 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:53.909637    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dfde42a-42cb-482d-a667-6ee53e804104-kube-api-access-gqn5r" pod "0dfde42a-42cb-482d-a667-6ee53e804104" (UID: "0dfde42a-42cb-482d-a667-6ee53e804104"). InnerVolumeSpecName "kube-api-access-gqn5r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:24:53.909728 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:53.909655    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54bac357-122d-43a3-909e-87f1cb110ffa-kube-api-access-bx5qn" pod "54bac357-122d-43a3-909e-87f1cb110ffa" (UID: "54bac357-122d-43a3-909e-87f1cb110ffa"). InnerVolumeSpecName "kube-api-access-bx5qn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:24:54.010365 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:54.010321    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bx5qn\" (UniqueName: \"kubernetes.io/projected/54bac357-122d-43a3-909e-87f1cb110ffa-kube-api-access-bx5qn\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:24:54.010365 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:54.010362    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqn5r\" (UniqueName: \"kubernetes.io/projected/0dfde42a-42cb-482d-a667-6ee53e804104-kube-api-access-gqn5r\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:24:54.735837 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:54.735798    4496 generic.go:356] "Generic (PLEG): container finished" podID="0dfde42a-42cb-482d-a667-6ee53e804104" containerID="471c15032821a5d41c47d97941e8bf0758308737148ba1e44c307b9d3bce5868" exitCode=0
Nov 02 03:24:54.736363 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:54.735857    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b4a230-5666cf5b6c-vv8cx"
Nov 02 03:24:54.736363 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:54.735884    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b4a230-5666cf5b6c-vv8cx" event={"ID":"0dfde42a-42cb-482d-a667-6ee53e804104","Type":"ContainerDied","Data":"471c15032821a5d41c47d97941e8bf0758308737148ba1e44c307b9d3bce5868"}
Nov 02 03:24:54.736363 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:54.735952    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b4a230-5666cf5b6c-vv8cx" event={"ID":"0dfde42a-42cb-482d-a667-6ee53e804104","Type":"ContainerDied","Data":"c6d04b6db58517e36537698b0fc6105b417bd13d0ca587de0f6e809ad422331e"}
Nov 02 03:24:54.736363 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:54.735981    4496 scope.go:122] "RemoveContainer" containerID="471c15032821a5d41c47d97941e8bf0758308737148ba1e44c307b9d3bce5868"
Nov 02 03:24:54.744038 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:54.738295    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-b4a230-7f49b69f9f-jfcml"
Nov 02 03:24:54.744193 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:54.744159    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-b4a230-7f49b69f9f-jfcml" event={"ID":"54bac357-122d-43a3-909e-87f1cb110ffa","Type":"ContainerDied","Data":"70c387a354e304b2b5fb8fd1a968dc47639fa28493df1140b0b413d8bfdb2864"}
Nov 02 03:24:54.751352 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:54.751333    4496 scope.go:122] "RemoveContainer" containerID="471c15032821a5d41c47d97941e8bf0758308737148ba1e44c307b9d3bce5868"
Nov 02 03:24:54.751750 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:24:54.751725    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"471c15032821a5d41c47d97941e8bf0758308737148ba1e44c307b9d3bce5868\": container with ID starting with 471c15032821a5d41c47d97941e8bf0758308737148ba1e44c307b9d3bce5868 not found: ID does not exist" containerID="471c15032821a5d41c47d97941e8bf0758308737148ba1e44c307b9d3bce5868"
Nov 02 03:24:54.751866 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:54.751758    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"471c15032821a5d41c47d97941e8bf0758308737148ba1e44c307b9d3bce5868"} err="failed to get container status \"471c15032821a5d41c47d97941e8bf0758308737148ba1e44c307b9d3bce5868\": rpc error: code = NotFound desc = could not find container \"471c15032821a5d41c47d97941e8bf0758308737148ba1e44c307b9d3bce5868\": container with ID starting with 471c15032821a5d41c47d97941e8bf0758308737148ba1e44c307b9d3bce5868 not found: ID does not exist"
Nov 02 03:24:54.751866 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:54.751778    4496 scope.go:122] "RemoveContainer" containerID="8385a1531b982dfb9d00e307092cbae8bd514870b425a685ed91b288b1c062df"
Nov 02 03:24:54.907775 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:54.907743    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b4a230-5666cf5b6c-vv8cx"]
Nov 02 03:24:54.957971 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:54.957931    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b4a230-5666cf5b6c-vv8cx"]
Nov 02 03:24:55.023936 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:55.023858    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b4a230-7f49b69f9f-jfcml"]
Nov 02 03:24:55.027818 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:55.027793    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b4a230-7f49b69f9f-jfcml"]
Nov 02 03:24:55.575901 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:55.575865    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0dfde42a-42cb-482d-a667-6ee53e804104" path="/var/lib/kubelet/pods/0dfde42a-42cb-482d-a667-6ee53e804104/volumes"
Nov 02 03:24:55.576400 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:55.576384    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54bac357-122d-43a3-909e-87f1cb110ffa" path="/var/lib/kubelet/pods/54bac357-122d-43a3-909e-87f1cb110ffa/volumes"
Nov 02 03:24:57.049356 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:57.049321    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61cf3d87-jjgxb"]
Nov 02 03:24:57.097385 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:57.097353    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61cf3d87-jjgxb"]
Nov 02 03:24:57.577602 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:24:57.577565    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27f971ed-34be-47b3-ac97-ab4710e5a412" path="/var/lib/kubelet/pods/27f971ed-34be-47b3-ac97-ab4710e5a412/volumes"
Nov 02 03:25:00.767252 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:00.767208    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-ff2fd"]
Nov 02 03:25:00.769744 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:00.767951    4496 cpu_manager.go:414] "containerMap: removing container" podUID="54bac357-122d-43a3-909e-87f1cb110ffa" containerName="payload-processing"
Nov 02 03:25:00.769744 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:00.767966    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54bac357-122d-43a3-909e-87f1cb110ffa" containerName="payload-processing"
Nov 02 03:25:00.769744 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:00.767978    4496 cpu_manager.go:414] "containerMap: removing container" podUID="0dfde42a-42cb-482d-a667-6ee53e804104" containerName="payload-pre-processing"
Nov 02 03:25:00.769744 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:00.767984    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dfde42a-42cb-482d-a667-6ee53e804104" containerName="payload-pre-processing"
Nov 02 03:25:00.769744 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:00.768002    4496 cpu_manager.go:414] "containerMap: removing container" podUID="aee8a3f1-6405-467c-8b98-84ca26618ffb" containerName="revoke-keys"
Nov 02 03:25:00.769744 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:00.768007    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aee8a3f1-6405-467c-8b98-84ca26618ffb" containerName="revoke-keys"
Nov 02 03:25:00.769744 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:00.768125    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="54bac357-122d-43a3-909e-87f1cb110ffa" containerName="payload-processing"
Nov 02 03:25:00.769744 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:00.768136    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="0dfde42a-42cb-482d-a667-6ee53e804104" containerName="payload-pre-processing"
Nov 02 03:25:00.769744 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:00.768146    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="aee8a3f1-6405-467c-8b98-84ca26618ffb" containerName="revoke-keys"
Nov 02 03:25:00.771785 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:00.771761    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-ff2fd"
Nov 02 03:25:00.824569 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:00.824529    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-d5dcca72-openshift-default-dockercfg-glcl8\""
Nov 02 03:25:00.867180 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:00.867144    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e11f3625-7e4e-4124-a232-8cf67769fdfc-istiod-ca-cert\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-ff2fd\" (UID: \"e11f3625-7e4e-4124-a232-8cf67769fdfc\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-ff2fd"
Nov 02 03:25:00.867375 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:00.867286    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e11f3625-7e4e-4124-a232-8cf67769fdfc-istio-token\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-ff2fd\" (UID: \"e11f3625-7e4e-4124-a232-8cf67769fdfc\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-ff2fd"
Nov 02 03:25:00.867450 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:00.867372    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e11f3625-7e4e-4124-a232-8cf67769fdfc-istio-envoy\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-ff2fd\" (UID: \"e11f3625-7e4e-4124-a232-8cf67769fdfc\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-ff2fd"
Nov 02 03:25:00.867505 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:00.867464    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e11f3625-7e4e-4124-a232-8cf67769fdfc-istio-data\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-ff2fd\" (UID: \"e11f3625-7e4e-4124-a232-8cf67769fdfc\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-ff2fd"
Nov 02 03:25:00.867591 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:00.867563    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e11f3625-7e4e-4124-a232-8cf67769fdfc-workload-socket\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-ff2fd\" (UID: \"e11f3625-7e4e-4124-a232-8cf67769fdfc\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-ff2fd"
Nov 02 03:25:00.867642 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:00.867627    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e11f3625-7e4e-4124-a232-8cf67769fdfc-istio-podinfo\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-ff2fd\" (UID: \"e11f3625-7e4e-4124-a232-8cf67769fdfc\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-ff2fd"
Nov 02 03:25:00.867727 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:00.867710    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e11f3625-7e4e-4124-a232-8cf67769fdfc-credential-socket\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-ff2fd\" (UID: \"e11f3625-7e4e-4124-a232-8cf67769fdfc\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-ff2fd"
Nov 02 03:25:00.867802 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:00.867785    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8fzm\" (UniqueName: \"kubernetes.io/projected/e11f3625-7e4e-4124-a232-8cf67769fdfc-kube-api-access-r8fzm\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-ff2fd\" (UID: \"e11f3625-7e4e-4124-a232-8cf67769fdfc\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-ff2fd"
Nov 02 03:25:00.867904 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:00.867890    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e11f3625-7e4e-4124-a232-8cf67769fdfc-workload-certs\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-ff2fd\" (UID: \"e11f3625-7e4e-4124-a232-8cf67769fdfc\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-ff2fd"
Nov 02 03:25:00.969580 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:00.969534    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e11f3625-7e4e-4124-a232-8cf67769fdfc-workload-certs\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-ff2fd\" (UID: \"e11f3625-7e4e-4124-a232-8cf67769fdfc\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-ff2fd"
Nov 02 03:25:00.970346 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:00.970317    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e11f3625-7e4e-4124-a232-8cf67769fdfc-istiod-ca-cert\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-ff2fd\" (UID: \"e11f3625-7e4e-4124-a232-8cf67769fdfc\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-ff2fd"
Nov 02 03:25:00.971322 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:00.971295    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e11f3625-7e4e-4124-a232-8cf67769fdfc-istio-token\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-ff2fd\" (UID: \"e11f3625-7e4e-4124-a232-8cf67769fdfc\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-ff2fd"
Nov 02 03:25:00.971447 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:00.971386    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e11f3625-7e4e-4124-a232-8cf67769fdfc-istio-envoy\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-ff2fd\" (UID: \"e11f3625-7e4e-4124-a232-8cf67769fdfc\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-ff2fd"
Nov 02 03:25:00.971515 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:00.971441    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e11f3625-7e4e-4124-a232-8cf67769fdfc-istio-data\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-ff2fd\" (UID: \"e11f3625-7e4e-4124-a232-8cf67769fdfc\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-ff2fd"
Nov 02 03:25:00.971515 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:00.971484    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e11f3625-7e4e-4124-a232-8cf67769fdfc-workload-socket\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-ff2fd\" (UID: \"e11f3625-7e4e-4124-a232-8cf67769fdfc\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-ff2fd"
Nov 02 03:25:00.971636 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:00.971529    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e11f3625-7e4e-4124-a232-8cf67769fdfc-istio-podinfo\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-ff2fd\" (UID: \"e11f3625-7e4e-4124-a232-8cf67769fdfc\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-ff2fd"
Nov 02 03:25:00.971636 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:00.971599    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e11f3625-7e4e-4124-a232-8cf67769fdfc-credential-socket\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-ff2fd\" (UID: \"e11f3625-7e4e-4124-a232-8cf67769fdfc\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-ff2fd"
Nov 02 03:25:00.971745 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:00.971671    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8fzm\" (UniqueName: \"kubernetes.io/projected/e11f3625-7e4e-4124-a232-8cf67769fdfc-kube-api-access-r8fzm\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-ff2fd\" (UID: \"e11f3625-7e4e-4124-a232-8cf67769fdfc\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-ff2fd"
Nov 02 03:25:00.972377 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:00.972353    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e11f3625-7e4e-4124-a232-8cf67769fdfc-workload-socket\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-ff2fd\" (UID: \"e11f3625-7e4e-4124-a232-8cf67769fdfc\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-ff2fd"
Nov 02 03:25:00.974882 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:00.971201    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e11f3625-7e4e-4124-a232-8cf67769fdfc-istiod-ca-cert\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-ff2fd\" (UID: \"e11f3625-7e4e-4124-a232-8cf67769fdfc\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-ff2fd"
Nov 02 03:25:00.975104 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:00.972911    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e11f3625-7e4e-4124-a232-8cf67769fdfc-istio-data\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-ff2fd\" (UID: \"e11f3625-7e4e-4124-a232-8cf67769fdfc\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-ff2fd"
Nov 02 03:25:00.975216 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:00.970212    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e11f3625-7e4e-4124-a232-8cf67769fdfc-workload-certs\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-ff2fd\" (UID: \"e11f3625-7e4e-4124-a232-8cf67769fdfc\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-ff2fd"
Nov 02 03:25:00.975216 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:00.974085    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e11f3625-7e4e-4124-a232-8cf67769fdfc-credential-socket\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-ff2fd\" (UID: \"e11f3625-7e4e-4124-a232-8cf67769fdfc\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-ff2fd"
Nov 02 03:25:00.975216 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:00.974663    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-ff2fd"]
Nov 02 03:25:00.976614 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:00.976559    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e11f3625-7e4e-4124-a232-8cf67769fdfc-istio-envoy\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-ff2fd\" (UID: \"e11f3625-7e4e-4124-a232-8cf67769fdfc\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-ff2fd"
Nov 02 03:25:00.979438 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:00.977494    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e11f3625-7e4e-4124-a232-8cf67769fdfc-istio-podinfo\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-ff2fd\" (UID: \"e11f3625-7e4e-4124-a232-8cf67769fdfc\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-ff2fd"
Nov 02 03:25:01.034414 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:01.034347    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e11f3625-7e4e-4124-a232-8cf67769fdfc-istio-token\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-ff2fd\" (UID: \"e11f3625-7e4e-4124-a232-8cf67769fdfc\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-ff2fd"
Nov 02 03:25:01.034414 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:01.034356    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8fzm\" (UniqueName: \"kubernetes.io/projected/e11f3625-7e4e-4124-a232-8cf67769fdfc-kube-api-access-r8fzm\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-ff2fd\" (UID: \"e11f3625-7e4e-4124-a232-8cf67769fdfc\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-ff2fd"
Nov 02 03:25:01.087438 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:01.087401    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-ff2fd"
Nov 02 03:25:01.525046 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:25:01.525002    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode11f3625_7e4e_4124_a232_8cf67769fdfc.slice/crio-df84c563f28582af4cbc461f84626fb2b1adc27f01b87e9161c20ade8499c41a WatchSource:0}: Error finding container df84c563f28582af4cbc461f84626fb2b1adc27f01b87e9161c20ade8499c41a: Status 404 returned error can't find the container with id df84c563f28582af4cbc461f84626fb2b1adc27f01b87e9161c20ade8499c41a
Nov 02 03:25:01.525829 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:01.525806    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-ff2fd"]
Nov 02 03:25:01.527808 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:01.527778    4496 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 03:25:01.527888 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:01.527861    4496 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 03:25:01.527953 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:01.527888    4496 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 03:25:01.865651 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:01.865519    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-ff2fd" event={"ID":"e11f3625-7e4e-4124-a232-8cf67769fdfc","Type":"ContainerStarted","Data":"6af658c35919aae30a6a2946ea71e2f7fde491043366d64674a0c1636ab0a37e"}
Nov 02 03:25:01.865651 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:01.865629    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-ff2fd" event={"ID":"e11f3625-7e4e-4124-a232-8cf67769fdfc","Type":"ContainerStarted","Data":"df84c563f28582af4cbc461f84626fb2b1adc27f01b87e9161c20ade8499c41a"}
Nov 02 03:25:01.987035 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:01.986972    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-ff2fd" podStartSLOduration=1.9869572519999998 podStartE2EDuration="1.986957252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:25:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:25:01.983109005 +0000 UTC m=+2242.561716386" watchObservedRunningTime="2026-11-02 03:25:01.986957252 +0000 UTC m=+2242.565564632"
Nov 02 03:25:02.087895 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:02.087862    4496 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-ff2fd"
Nov 02 03:25:02.089387 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:02.089357    4496 patch_prober.go:29] interesting pod/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-ff2fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body=
Nov 02 03:25:02.089521 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:02.089416    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-ff2fd" podUID="e11f3625-7e4e-4124-a232-8cf67769fdfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused"
Nov 02 03:25:03.088216 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:03.088178    4496 patch_prober.go:29] interesting pod/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-ff2fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body=
Nov 02 03:25:03.088736 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:03.088246    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-ff2fd" podUID="e11f3625-7e4e-4124-a232-8cf67769fdfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused"
Nov 02 03:25:03.527547 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:25:03.527506    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 03:25:04.088126 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:04.088079    4496 patch_prober.go:29] interesting pod/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-ff2fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body=
Nov 02 03:25:04.088335 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:04.088184    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-ff2fd" podUID="e11f3625-7e4e-4124-a232-8cf67769fdfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused"
Nov 02 03:25:05.088356 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:05.088318    4496 patch_prober.go:29] interesting pod/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-ff2fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body=
Nov 02 03:25:05.088784 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:05.088372    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-ff2fd" podUID="e11f3625-7e4e-4124-a232-8cf67769fdfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused"
Nov 02 03:25:06.088557 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:06.088515    4496 patch_prober.go:29] interesting pod/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-ff2fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body=
Nov 02 03:25:06.089108 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:06.088582    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-ff2fd" podUID="e11f3625-7e4e-4124-a232-8cf67769fdfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused"
Nov 02 03:25:07.088842 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:07.088801    4496 patch_prober.go:29] interesting pod/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-ff2fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body=
Nov 02 03:25:07.089285 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:07.088858    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-ff2fd" podUID="e11f3625-7e4e-4124-a232-8cf67769fdfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused"
Nov 02 03:25:08.092169 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:08.092137    4496 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-ff2fd"
Nov 02 03:25:08.092632 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:08.092612    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-ff2fd"
Nov 02 03:25:08.093446 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:08.093425    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-ff2fd"
Nov 02 03:25:15.882180 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:15.882136    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-fn4nf"]
Nov 02 03:25:15.888053 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:15.888029    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-fn4nf"
Nov 02 03:25:16.007725 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:16.007678    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90252c4b-9031-4994-86e1-9b7ae966cd7e-istio-data\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-fn4nf\" (UID: \"90252c4b-9031-4994-86e1-9b7ae966cd7e\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-fn4nf"
Nov 02 03:25:16.007925 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:16.007795    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zx25\" (UniqueName: \"kubernetes.io/projected/90252c4b-9031-4994-86e1-9b7ae966cd7e-kube-api-access-9zx25\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-fn4nf\" (UID: \"90252c4b-9031-4994-86e1-9b7ae966cd7e\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-fn4nf"
Nov 02 03:25:16.007981 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:16.007928    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90252c4b-9031-4994-86e1-9b7ae966cd7e-istio-podinfo\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-fn4nf\" (UID: \"90252c4b-9031-4994-86e1-9b7ae966cd7e\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-fn4nf"
Nov 02 03:25:16.008072 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:16.007993    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90252c4b-9031-4994-86e1-9b7ae966cd7e-credential-socket\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-fn4nf\" (UID: \"90252c4b-9031-4994-86e1-9b7ae966cd7e\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-fn4nf"
Nov 02 03:25:16.008072 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:16.008057    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90252c4b-9031-4994-86e1-9b7ae966cd7e-istio-token\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-fn4nf\" (UID: \"90252c4b-9031-4994-86e1-9b7ae966cd7e\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-fn4nf"
Nov 02 03:25:16.008159 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:16.008088    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90252c4b-9031-4994-86e1-9b7ae966cd7e-workload-certs\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-fn4nf\" (UID: \"90252c4b-9031-4994-86e1-9b7ae966cd7e\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-fn4nf"
Nov 02 03:25:16.008216 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:16.008188    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90252c4b-9031-4994-86e1-9b7ae966cd7e-istio-envoy\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-fn4nf\" (UID: \"90252c4b-9031-4994-86e1-9b7ae966cd7e\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-fn4nf"
Nov 02 03:25:16.008315 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:16.008299    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90252c4b-9031-4994-86e1-9b7ae966cd7e-workload-socket\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-fn4nf\" (UID: \"90252c4b-9031-4994-86e1-9b7ae966cd7e\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-fn4nf"
Nov 02 03:25:16.008489 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:16.008470    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90252c4b-9031-4994-86e1-9b7ae966cd7e-istiod-ca-cert\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-fn4nf\" (UID: \"90252c4b-9031-4994-86e1-9b7ae966cd7e\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-fn4nf"
Nov 02 03:25:16.113986 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:16.112405    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90252c4b-9031-4994-86e1-9b7ae966cd7e-workload-socket\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-fn4nf\" (UID: \"90252c4b-9031-4994-86e1-9b7ae966cd7e\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-fn4nf"
Nov 02 03:25:16.114321 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:16.113951    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90252c4b-9031-4994-86e1-9b7ae966cd7e-workload-socket\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-fn4nf\" (UID: \"90252c4b-9031-4994-86e1-9b7ae966cd7e\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-fn4nf"
Nov 02 03:25:16.114611 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:16.114590    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90252c4b-9031-4994-86e1-9b7ae966cd7e-istiod-ca-cert\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-fn4nf\" (UID: \"90252c4b-9031-4994-86e1-9b7ae966cd7e\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-fn4nf"
Nov 02 03:25:16.115649 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:16.115527    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90252c4b-9031-4994-86e1-9b7ae966cd7e-istiod-ca-cert\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-fn4nf\" (UID: \"90252c4b-9031-4994-86e1-9b7ae966cd7e\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-fn4nf"
Nov 02 03:25:16.115903 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:16.115883    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90252c4b-9031-4994-86e1-9b7ae966cd7e-istio-data\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-fn4nf\" (UID: \"90252c4b-9031-4994-86e1-9b7ae966cd7e\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-fn4nf"
Nov 02 03:25:16.116387 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:16.116367    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zx25\" (UniqueName: \"kubernetes.io/projected/90252c4b-9031-4994-86e1-9b7ae966cd7e-kube-api-access-9zx25\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-fn4nf\" (UID: \"90252c4b-9031-4994-86e1-9b7ae966cd7e\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-fn4nf"
Nov 02 03:25:16.116606 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:16.116289    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90252c4b-9031-4994-86e1-9b7ae966cd7e-istio-data\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-fn4nf\" (UID: \"90252c4b-9031-4994-86e1-9b7ae966cd7e\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-fn4nf"
Nov 02 03:25:16.116788 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:16.116761    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90252c4b-9031-4994-86e1-9b7ae966cd7e-istio-podinfo\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-fn4nf\" (UID: \"90252c4b-9031-4994-86e1-9b7ae966cd7e\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-fn4nf"
Nov 02 03:25:16.116914 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:16.116829    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90252c4b-9031-4994-86e1-9b7ae966cd7e-credential-socket\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-fn4nf\" (UID: \"90252c4b-9031-4994-86e1-9b7ae966cd7e\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-fn4nf"
Nov 02 03:25:16.116914 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:16.116859    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90252c4b-9031-4994-86e1-9b7ae966cd7e-istio-token\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-fn4nf\" (UID: \"90252c4b-9031-4994-86e1-9b7ae966cd7e\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-fn4nf"
Nov 02 03:25:16.116914 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:16.116889    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90252c4b-9031-4994-86e1-9b7ae966cd7e-workload-certs\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-fn4nf\" (UID: \"90252c4b-9031-4994-86e1-9b7ae966cd7e\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-fn4nf"
Nov 02 03:25:16.117101 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:16.116922    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90252c4b-9031-4994-86e1-9b7ae966cd7e-istio-envoy\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-fn4nf\" (UID: \"90252c4b-9031-4994-86e1-9b7ae966cd7e\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-fn4nf"
Nov 02 03:25:16.117796 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:16.117773    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90252c4b-9031-4994-86e1-9b7ae966cd7e-credential-socket\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-fn4nf\" (UID: \"90252c4b-9031-4994-86e1-9b7ae966cd7e\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-fn4nf"
Nov 02 03:25:16.117997 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:16.117972    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90252c4b-9031-4994-86e1-9b7ae966cd7e-workload-certs\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-fn4nf\" (UID: \"90252c4b-9031-4994-86e1-9b7ae966cd7e\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-fn4nf"
Nov 02 03:25:16.125175 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:16.125147    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90252c4b-9031-4994-86e1-9b7ae966cd7e-istio-podinfo\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-fn4nf\" (UID: \"90252c4b-9031-4994-86e1-9b7ae966cd7e\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-fn4nf"
Nov 02 03:25:16.125706 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:16.125684    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90252c4b-9031-4994-86e1-9b7ae966cd7e-istio-envoy\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-fn4nf\" (UID: \"90252c4b-9031-4994-86e1-9b7ae966cd7e\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-fn4nf"
Nov 02 03:25:16.131331 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:16.131296    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-fn4nf"]
Nov 02 03:25:16.173939 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:16.173859    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zx25\" (UniqueName: \"kubernetes.io/projected/90252c4b-9031-4994-86e1-9b7ae966cd7e-kube-api-access-9zx25\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-fn4nf\" (UID: \"90252c4b-9031-4994-86e1-9b7ae966cd7e\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-fn4nf"
Nov 02 03:25:16.174112 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:16.174074    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90252c4b-9031-4994-86e1-9b7ae966cd7e-istio-token\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-fn4nf\" (UID: \"90252c4b-9031-4994-86e1-9b7ae966cd7e\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-fn4nf"
Nov 02 03:25:16.204133 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:16.204106    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-fn4nf"
Nov 02 03:25:16.643661 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:16.643630    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-fn4nf"]
Nov 02 03:25:16.644301 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:25:16.644273    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90252c4b_9031_4994_86e1_9b7ae966cd7e.slice/crio-4d3d93136db023f3bf8fa611856c922f37533a1412b5deb894be4dd320080781 WatchSource:0}: Error finding container 4d3d93136db023f3bf8fa611856c922f37533a1412b5deb894be4dd320080781: Status 404 returned error can't find the container with id 4d3d93136db023f3bf8fa611856c922f37533a1412b5deb894be4dd320080781
Nov 02 03:25:16.647640 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:16.647610    4496 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 03:25:16.647730 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:16.647672    4496 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 03:25:16.647730 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:16.647703    4496 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 03:25:17.010536 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:17.010496    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-fn4nf" event={"ID":"90252c4b-9031-4994-86e1-9b7ae966cd7e","Type":"ContainerStarted","Data":"2340ff8ddfae8d2f582513548f272f8bbebb3fda69a164f7541e705420fac92e"}
Nov 02 03:25:17.010916 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:17.010555    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-fn4nf" event={"ID":"90252c4b-9031-4994-86e1-9b7ae966cd7e","Type":"ContainerStarted","Data":"4d3d93136db023f3bf8fa611856c922f37533a1412b5deb894be4dd320080781"}
Nov 02 03:25:17.140968 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:17.140905    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-fn4nf" podStartSLOduration=2.140885821 podStartE2EDuration="2.140885821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:25:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:25:17.13541299 +0000 UTC m=+2257.714020371" watchObservedRunningTime="2026-11-02 03:25:17.140885821 +0000 UTC m=+2257.719493202"
Nov 02 03:25:17.204561 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:17.204527    4496 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-fn4nf"
Nov 02 03:25:17.206402 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:17.206369    4496 patch_prober.go:29] interesting pod/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-fn4nf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body=
Nov 02 03:25:17.206539 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:17.206417    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-fn4nf" podUID="90252c4b-9031-4994-86e1-9b7ae966cd7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused"
Nov 02 03:25:18.204757 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:18.204720    4496 patch_prober.go:29] interesting pod/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-fn4nf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body=
Nov 02 03:25:18.205190 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:18.204786    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-fn4nf" podUID="90252c4b-9031-4994-86e1-9b7ae966cd7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused"
Nov 02 03:25:19.205490 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:19.205437    4496 patch_prober.go:29] interesting pod/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-fn4nf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body=
Nov 02 03:25:19.205910 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:19.205520    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-fn4nf" podUID="90252c4b-9031-4994-86e1-9b7ae966cd7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused"
Nov 02 03:25:20.205306 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:20.205274    4496 patch_prober.go:29] interesting pod/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-fn4nf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body=
Nov 02 03:25:20.205499 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:20.205325    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-fn4nf" podUID="90252c4b-9031-4994-86e1-9b7ae966cd7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused"
Nov 02 03:25:21.204657 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:21.204619    4496 patch_prober.go:29] interesting pod/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-fn4nf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body=
Nov 02 03:25:21.205114 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:21.204677    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-fn4nf" podUID="90252c4b-9031-4994-86e1-9b7ae966cd7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused"
Nov 02 03:25:22.204691 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:22.204652    4496 patch_prober.go:29] interesting pod/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-fn4nf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body=
Nov 02 03:25:22.205092 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:22.204720    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-fn4nf" podUID="90252c4b-9031-4994-86e1-9b7ae966cd7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused"
Nov 02 03:25:23.342325 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:23.342293    4496 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-fn4nf"
Nov 02 03:25:23.342804 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:23.342780    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-fn4nf"
Nov 02 03:25:23.343726 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:23.343703    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-fn4nf"
Nov 02 03:25:28.356823 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:28.356778    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11256-20c1c8"]
Nov 02 03:25:28.361553 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:28.361536    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11256-20c1c8"
Nov 02 03:25:28.412876 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:28.412839    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11256-20c1c8"]
Nov 02 03:25:28.457529 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:28.457481    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-t4r8t\""
Nov 02 03:25:28.501418 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:28.501385    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sp84p\" (UniqueName: \"kubernetes.io/projected/d1faaebc-9cac-4b79-9324-59aeda0cdbfd-kube-api-access-sp84p\") pod \"test-curl-11256-20c1c8\" (UID: \"d1faaebc-9cac-4b79-9324-59aeda0cdbfd\") " pod="opendatahub/test-curl-11256-20c1c8"
Nov 02 03:25:28.602992 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:28.602954    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sp84p\" (UniqueName: \"kubernetes.io/projected/d1faaebc-9cac-4b79-9324-59aeda0cdbfd-kube-api-access-sp84p\") pod \"test-curl-11256-20c1c8\" (UID: \"d1faaebc-9cac-4b79-9324-59aeda0cdbfd\") " pod="opendatahub/test-curl-11256-20c1c8"
Nov 02 03:25:28.660929 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:28.660855    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sp84p\" (UniqueName: \"kubernetes.io/projected/d1faaebc-9cac-4b79-9324-59aeda0cdbfd-kube-api-access-sp84p\") pod \"test-curl-11256-20c1c8\" (UID: \"d1faaebc-9cac-4b79-9324-59aeda0cdbfd\") " pod="opendatahub/test-curl-11256-20c1c8"
Nov 02 03:25:28.674687 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:28.674656    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11256-20c1c8"
Nov 02 03:25:28.720885 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:28.720837    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ed15c1-cfl44"]
Nov 02 03:25:28.726466 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:28.726430    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ed15c1-cfl44"]
Nov 02 03:25:29.060623 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:25:29.060595    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd1faaebc_9cac_4b79_9324_59aeda0cdbfd.slice/crio-e97b22ff2170f2331e39a6088ef0dafd3ecf6a258b9585b1478f7dc8b1ec5231 WatchSource:0}: Error finding container e97b22ff2170f2331e39a6088ef0dafd3ecf6a258b9585b1478f7dc8b1ec5231: Status 404 returned error can't find the container with id e97b22ff2170f2331e39a6088ef0dafd3ecf6a258b9585b1478f7dc8b1ec5231
Nov 02 03:25:29.061266 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:29.061244    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11256-20c1c8"]
Nov 02 03:25:29.151086 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:29.151054    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11256-20c1c8" event={"ID":"d1faaebc-9cac-4b79-9324-59aeda0cdbfd","Type":"ContainerStarted","Data":"2052defa4f40bf4d5fa2fb36a8a2f5b7fafb72462833025041a47ef9314e4c5e"}
Nov 02 03:25:29.151211 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:29.151103    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11256-20c1c8" event={"ID":"d1faaebc-9cac-4b79-9324-59aeda0cdbfd","Type":"ContainerStarted","Data":"e97b22ff2170f2331e39a6088ef0dafd3ecf6a258b9585b1478f7dc8b1ec5231"}
Nov 02 03:25:29.266878 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:29.266816    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11256-20c1c8" podStartSLOduration=1.266800933 podStartE2EDuration="1.266800933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:25:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:25:29.263531158 +0000 UTC m=+2269.842138538" watchObservedRunningTime="2026-11-02 03:25:29.266800933 +0000 UTC m=+2269.845408313"
Nov 02 03:25:29.575573 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:29.575539    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31334cad-3472-45dc-b713-de51a828f9b3" path="/var/lib/kubelet/pods/31334cad-3472-45dc-b713-de51a828f9b3/volumes"
Nov 02 03:25:30.669706 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:30.669676    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11256-20c1c8"]
Nov 02 03:25:30.677408 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:30.677372    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11256-20c1c8"]
Nov 02 03:25:30.988625 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:30.988590    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-t4dvl"]
Nov 02 03:25:30.989368 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:30.989352    4496 cpu_manager.go:414] "containerMap: removing container" podUID="d1faaebc-9cac-4b79-9324-59aeda0cdbfd" containerName="test-curl-11256-20c1c8"
Nov 02 03:25:30.989425 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:30.989368    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1faaebc-9cac-4b79-9324-59aeda0cdbfd" containerName="test-curl-11256-20c1c8"
Nov 02 03:25:30.989518 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:30.989508    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1faaebc-9cac-4b79-9324-59aeda0cdbfd" containerName="test-curl-11256-20c1c8"
Nov 02 03:25:30.993516 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:30.993497    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-t4dvl"
Nov 02 03:25:31.116221 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:31.116185    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-t4dvl"]
Nov 02 03:25:31.138997 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:31.138961    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/013b67e2-aae9-4380-baf6-de93a6a9d18b-istio-envoy\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-t4dvl\" (UID: \"013b67e2-aae9-4380-baf6-de93a6a9d18b\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-t4dvl"
Nov 02 03:25:31.139194 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:31.139005    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/013b67e2-aae9-4380-baf6-de93a6a9d18b-credential-socket\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-t4dvl\" (UID: \"013b67e2-aae9-4380-baf6-de93a6a9d18b\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-t4dvl"
Nov 02 03:25:31.139194 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:31.139094    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/013b67e2-aae9-4380-baf6-de93a6a9d18b-workload-certs\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-t4dvl\" (UID: \"013b67e2-aae9-4380-baf6-de93a6a9d18b\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-t4dvl"
Nov 02 03:25:31.139284 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:31.139214    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/013b67e2-aae9-4380-baf6-de93a6a9d18b-istio-data\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-t4dvl\" (UID: \"013b67e2-aae9-4380-baf6-de93a6a9d18b\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-t4dvl"
Nov 02 03:25:31.139336 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:31.139321    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/013b67e2-aae9-4380-baf6-de93a6a9d18b-istio-podinfo\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-t4dvl\" (UID: \"013b67e2-aae9-4380-baf6-de93a6a9d18b\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-t4dvl"
Nov 02 03:25:31.139390 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:31.139379    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/013b67e2-aae9-4380-baf6-de93a6a9d18b-istiod-ca-cert\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-t4dvl\" (UID: \"013b67e2-aae9-4380-baf6-de93a6a9d18b\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-t4dvl"
Nov 02 03:25:31.139462 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:31.139444    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8sgw\" (UniqueName: \"kubernetes.io/projected/013b67e2-aae9-4380-baf6-de93a6a9d18b-kube-api-access-p8sgw\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-t4dvl\" (UID: \"013b67e2-aae9-4380-baf6-de93a6a9d18b\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-t4dvl"
Nov 02 03:25:31.139545 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:31.139530    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/013b67e2-aae9-4380-baf6-de93a6a9d18b-workload-socket\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-t4dvl\" (UID: \"013b67e2-aae9-4380-baf6-de93a6a9d18b\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-t4dvl"
Nov 02 03:25:31.139602 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:31.139591    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/013b67e2-aae9-4380-baf6-de93a6a9d18b-istio-token\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-t4dvl\" (UID: \"013b67e2-aae9-4380-baf6-de93a6a9d18b\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-t4dvl"
Nov 02 03:25:31.170066 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:31.169986    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11256-20c1c8" podUID="d1faaebc-9cac-4b79-9324-59aeda0cdbfd" containerName="test-curl-11256-20c1c8" containerID="cri-o://2052defa4f40bf4d5fa2fb36a8a2f5b7fafb72462833025041a47ef9314e4c5e" gracePeriod=2
Nov 02 03:25:31.209377 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:31.209339    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d5dcca72-nw5jm"]
Nov 02 03:25:31.214090 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:31.214067    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d5dcca72-nw5jm"
Nov 02 03:25:31.222124 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:25:31.222086    4496 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11256-20c1c8\" is forbidden: User \"system:node:ip-10-0-133-163.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-163.us-west-1.compute.internal' and this object" podUID="d1faaebc-9cac-4b79-9324-59aeda0cdbfd" pod="opendatahub/test-curl-11256-20c1c8"
Nov 02 03:25:31.241098 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:31.240991    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/013b67e2-aae9-4380-baf6-de93a6a9d18b-istio-token\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-t4dvl\" (UID: \"013b67e2-aae9-4380-baf6-de93a6a9d18b\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-t4dvl"
Nov 02 03:25:31.241244 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:31.241118    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/013b67e2-aae9-4380-baf6-de93a6a9d18b-istio-envoy\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-t4dvl\" (UID: \"013b67e2-aae9-4380-baf6-de93a6a9d18b\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-t4dvl"
Nov 02 03:25:31.241244 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:31.241149    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/013b67e2-aae9-4380-baf6-de93a6a9d18b-credential-socket\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-t4dvl\" (UID: \"013b67e2-aae9-4380-baf6-de93a6a9d18b\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-t4dvl"
Nov 02 03:25:31.241244 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:31.241186    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/013b67e2-aae9-4380-baf6-de93a6a9d18b-workload-certs\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-t4dvl\" (UID: \"013b67e2-aae9-4380-baf6-de93a6a9d18b\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-t4dvl"
Nov 02 03:25:31.241418 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:31.241258    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/013b67e2-aae9-4380-baf6-de93a6a9d18b-istio-data\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-t4dvl\" (UID: \"013b67e2-aae9-4380-baf6-de93a6a9d18b\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-t4dvl"
Nov 02 03:25:31.241418 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:31.241375    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/013b67e2-aae9-4380-baf6-de93a6a9d18b-istio-podinfo\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-t4dvl\" (UID: \"013b67e2-aae9-4380-baf6-de93a6a9d18b\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-t4dvl"
Nov 02 03:25:31.241536 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:31.241446    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/013b67e2-aae9-4380-baf6-de93a6a9d18b-istiod-ca-cert\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-t4dvl\" (UID: \"013b67e2-aae9-4380-baf6-de93a6a9d18b\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-t4dvl"
Nov 02 03:25:31.241536 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:31.241502    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8sgw\" (UniqueName: \"kubernetes.io/projected/013b67e2-aae9-4380-baf6-de93a6a9d18b-kube-api-access-p8sgw\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-t4dvl\" (UID: \"013b67e2-aae9-4380-baf6-de93a6a9d18b\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-t4dvl"
Nov 02 03:25:31.241637 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:31.241540    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/013b67e2-aae9-4380-baf6-de93a6a9d18b-credential-socket\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-t4dvl\" (UID: \"013b67e2-aae9-4380-baf6-de93a6a9d18b\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-t4dvl"
Nov 02 03:25:31.241637 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:31.241629    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/013b67e2-aae9-4380-baf6-de93a6a9d18b-workload-socket\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-t4dvl\" (UID: \"013b67e2-aae9-4380-baf6-de93a6a9d18b\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-t4dvl"
Nov 02 03:25:31.241754 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:31.241679    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/013b67e2-aae9-4380-baf6-de93a6a9d18b-workload-certs\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-t4dvl\" (UID: \"013b67e2-aae9-4380-baf6-de93a6a9d18b\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-t4dvl"
Nov 02 03:25:31.241754 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:31.241713    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/013b67e2-aae9-4380-baf6-de93a6a9d18b-istio-data\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-t4dvl\" (UID: \"013b67e2-aae9-4380-baf6-de93a6a9d18b\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-t4dvl"
Nov 02 03:25:31.241978 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:31.241959    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/013b67e2-aae9-4380-baf6-de93a6a9d18b-workload-socket\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-t4dvl\" (UID: \"013b67e2-aae9-4380-baf6-de93a6a9d18b\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-t4dvl"
Nov 02 03:25:31.242262 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:31.242235    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/013b67e2-aae9-4380-baf6-de93a6a9d18b-istiod-ca-cert\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-t4dvl\" (UID: \"013b67e2-aae9-4380-baf6-de93a6a9d18b\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-t4dvl"
Nov 02 03:25:31.243508 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:31.243479    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/013b67e2-aae9-4380-baf6-de93a6a9d18b-istio-envoy\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-t4dvl\" (UID: \"013b67e2-aae9-4380-baf6-de93a6a9d18b\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-t4dvl"
Nov 02 03:25:31.244337 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:31.244306    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/013b67e2-aae9-4380-baf6-de93a6a9d18b-istio-podinfo\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-t4dvl\" (UID: \"013b67e2-aae9-4380-baf6-de93a6a9d18b\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-t4dvl"
Nov 02 03:25:31.267291 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:31.267171    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 03:25:31.267291 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:31.267197    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 03:25:31.267291 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:31.267211    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9z5bk\""
Nov 02 03:25:31.267291 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:31.267227    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 03:25:31.270251 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:31.270228    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d5dcca72-nw5jm"]
Nov 02 03:25:31.298920 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:31.298889    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/013b67e2-aae9-4380-baf6-de93a6a9d18b-istio-token\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-t4dvl\" (UID: \"013b67e2-aae9-4380-baf6-de93a6a9d18b\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-t4dvl"
Nov 02 03:25:31.299183 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:31.299070    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8sgw\" (UniqueName: \"kubernetes.io/projected/013b67e2-aae9-4380-baf6-de93a6a9d18b-kube-api-access-p8sgw\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-t4dvl\" (UID: \"013b67e2-aae9-4380-baf6-de93a6a9d18b\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-t4dvl"
Nov 02 03:25:31.309835 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:31.308555    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-t4dvl"
Nov 02 03:25:31.343241 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:31.343213    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6psrf\" (UniqueName: \"kubernetes.io/projected/42e1a6eb-7b6c-429a-b34e-f4a8f78e5981-kube-api-access-6psrf\") pod \"maas-api-revoke-keys-e2e-mt-d5dcca72-nw5jm\" (UID: \"42e1a6eb-7b6c-429a-b34e-f4a8f78e5981\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d5dcca72-nw5jm"
Nov 02 03:25:31.343571 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:31.343538    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42e1a6eb-7b6c-429a-b34e-f4a8f78e5981-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-d5dcca72-nw5jm\" (UID: \"42e1a6eb-7b6c-429a-b34e-f4a8f78e5981\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d5dcca72-nw5jm"
Nov 02 03:25:31.343702 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:31.343585    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42e1a6eb-7b6c-429a-b34e-f4a8f78e5981-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-d5dcca72-nw5jm\" (UID: \"42e1a6eb-7b6c-429a-b34e-f4a8f78e5981\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d5dcca72-nw5jm"
Nov 02 03:25:31.389480 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:25:31.389370    4496 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11256-20c1c8\" is forbidden: User \"system:node:ip-10-0-133-163.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-163.us-west-1.compute.internal' and this object" podUID="d1faaebc-9cac-4b79-9324-59aeda0cdbfd" pod="opendatahub/test-curl-11256-20c1c8"
Nov 02 03:25:31.445181 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:31.445138    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42e1a6eb-7b6c-429a-b34e-f4a8f78e5981-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-d5dcca72-nw5jm\" (UID: \"42e1a6eb-7b6c-429a-b34e-f4a8f78e5981\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d5dcca72-nw5jm"
Nov 02 03:25:31.445362 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:31.445190    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42e1a6eb-7b6c-429a-b34e-f4a8f78e5981-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-d5dcca72-nw5jm\" (UID: \"42e1a6eb-7b6c-429a-b34e-f4a8f78e5981\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d5dcca72-nw5jm"
Nov 02 03:25:31.445362 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:31.445242    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6psrf\" (UniqueName: \"kubernetes.io/projected/42e1a6eb-7b6c-429a-b34e-f4a8f78e5981-kube-api-access-6psrf\") pod \"maas-api-revoke-keys-e2e-mt-d5dcca72-nw5jm\" (UID: \"42e1a6eb-7b6c-429a-b34e-f4a8f78e5981\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d5dcca72-nw5jm"
Nov 02 03:25:31.447334 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:31.447305    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42e1a6eb-7b6c-429a-b34e-f4a8f78e5981-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-d5dcca72-nw5jm\" (UID: \"42e1a6eb-7b6c-429a-b34e-f4a8f78e5981\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d5dcca72-nw5jm"
Nov 02 03:25:31.449313 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:31.449285    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42e1a6eb-7b6c-429a-b34e-f4a8f78e5981-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-d5dcca72-nw5jm\" (UID: \"42e1a6eb-7b6c-429a-b34e-f4a8f78e5981\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d5dcca72-nw5jm"
Nov 02 03:25:31.502765 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:31.502691    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6psrf\" (UniqueName: \"kubernetes.io/projected/42e1a6eb-7b6c-429a-b34e-f4a8f78e5981-kube-api-access-6psrf\") pod \"maas-api-revoke-keys-e2e-mt-d5dcca72-nw5jm\" (UID: \"42e1a6eb-7b6c-429a-b34e-f4a8f78e5981\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d5dcca72-nw5jm"
Nov 02 03:25:31.528754 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:31.528730    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d5dcca72-nw5jm"
Nov 02 03:25:31.757937 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:25:31.757901    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod013b67e2_aae9_4380_baf6_de93a6a9d18b.slice/crio-029092a660811fdfea4fcacc4ada00088314a01b5cf26e800be64cb1cb13c9bc WatchSource:0}: Error finding container 029092a660811fdfea4fcacc4ada00088314a01b5cf26e800be64cb1cb13c9bc: Status 404 returned error can't find the container with id 029092a660811fdfea4fcacc4ada00088314a01b5cf26e800be64cb1cb13c9bc
Nov 02 03:25:31.758938 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:31.758911    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-t4dvl"]
Nov 02 03:25:31.760666 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:31.760633    4496 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 03:25:31.760742 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:31.760695    4496 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 03:25:31.760742 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:31.760735    4496 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 03:25:31.916268 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:25:31.916233    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42e1a6eb_7b6c_429a_b34e_f4a8f78e5981.slice/crio-3ac96acdfe9138cec05520516f42b0111c8b17f86704888b66deb5fc6f90d2d8 WatchSource:0}: Error finding container 3ac96acdfe9138cec05520516f42b0111c8b17f86704888b66deb5fc6f90d2d8: Status 404 returned error can't find the container with id 3ac96acdfe9138cec05520516f42b0111c8b17f86704888b66deb5fc6f90d2d8
Nov 02 03:25:31.916995 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:31.916967    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d5dcca72-nw5jm"]
Nov 02 03:25:32.180456 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:32.180412    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d5dcca72-nw5jm" event={"ID":"42e1a6eb-7b6c-429a-b34e-f4a8f78e5981","Type":"ContainerStarted","Data":"3ac96acdfe9138cec05520516f42b0111c8b17f86704888b66deb5fc6f90d2d8"}
Nov 02 03:25:32.182483 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:32.182452    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-t4dvl" event={"ID":"013b67e2-aae9-4380-baf6-de93a6a9d18b","Type":"ContainerStarted","Data":"a5c6a4c4b834bd5d7091bedb7a209bfcd4b79b2694f89912363f29f1c05ee425"}
Nov 02 03:25:32.182650 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:32.182513    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-t4dvl" event={"ID":"013b67e2-aae9-4380-baf6-de93a6a9d18b","Type":"ContainerStarted","Data":"029092a660811fdfea4fcacc4ada00088314a01b5cf26e800be64cb1cb13c9bc"}
Nov 02 03:25:32.304374 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:32.304322    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-t4dvl" podStartSLOduration=2.304307628 podStartE2EDuration="2.304307628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:25:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:25:32.299882552 +0000 UTC m=+2272.878489933" watchObservedRunningTime="2026-11-02 03:25:32.304307628 +0000 UTC m=+2272.882915007"
Nov 02 03:25:32.309011 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:32.308982    4496 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-t4dvl"
Nov 02 03:25:32.310510 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:32.310483    4496 patch_prober.go:29] interesting pod/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-t4dvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body=
Nov 02 03:25:32.310700 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:32.310528    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-t4dvl" podUID="013b67e2-aae9-4380-baf6-de93a6a9d18b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused"
Nov 02 03:25:33.193086 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:33.193060    4496 generic.go:356] "Generic (PLEG): container finished" podID="42e1a6eb-7b6c-429a-b34e-f4a8f78e5981" containerID="816c8e4e99ebba8d7ba2e16ac4877f6c12488e11dd2c1e7bbe44fefc9ec54b84" exitCode=0
Nov 02 03:25:33.193453 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:33.193144    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d5dcca72-nw5jm" event={"ID":"42e1a6eb-7b6c-429a-b34e-f4a8f78e5981","Type":"ContainerDied","Data":"816c8e4e99ebba8d7ba2e16ac4877f6c12488e11dd2c1e7bbe44fefc9ec54b84"}
Nov 02 03:25:33.309993 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:33.309898    4496 patch_prober.go:29] interesting pod/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-t4dvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body=
Nov 02 03:25:33.309993 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:33.309958    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-t4dvl" podUID="013b67e2-aae9-4380-baf6-de93a6a9d18b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused"
Nov 02 03:25:33.625109 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:33.611192    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11256-20c1c8"
Nov 02 03:25:33.709041 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:33.707405    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1faaebc-9cac-4b79-9324-59aeda0cdbfd-kube-api-access-sp84p\" (UniqueName: \"kubernetes.io/projected/d1faaebc-9cac-4b79-9324-59aeda0cdbfd-kube-api-access-sp84p\") pod \"d1faaebc-9cac-4b79-9324-59aeda0cdbfd\" (UID: \"d1faaebc-9cac-4b79-9324-59aeda0cdbfd\") "
Nov 02 03:25:33.711274 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:33.711237    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1faaebc-9cac-4b79-9324-59aeda0cdbfd-kube-api-access-sp84p" pod "d1faaebc-9cac-4b79-9324-59aeda0cdbfd" (UID: "d1faaebc-9cac-4b79-9324-59aeda0cdbfd"). InnerVolumeSpecName "kube-api-access-sp84p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:25:33.725853 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:25:33.725074    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 03:25:33.809598 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:33.809563    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sp84p\" (UniqueName: \"kubernetes.io/projected/d1faaebc-9cac-4b79-9324-59aeda0cdbfd-kube-api-access-sp84p\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:25:34.202730 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:34.202698    4496 generic.go:356] "Generic (PLEG): container finished" podID="d1faaebc-9cac-4b79-9324-59aeda0cdbfd" containerID="2052defa4f40bf4d5fa2fb36a8a2f5b7fafb72462833025041a47ef9314e4c5e" exitCode=137
Nov 02 03:25:34.203124 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:34.202898    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11256-20c1c8"
Nov 02 03:25:34.203187 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:34.203177    4496 scope.go:122] "RemoveContainer" containerID="2052defa4f40bf4d5fa2fb36a8a2f5b7fafb72462833025041a47ef9314e4c5e"
Nov 02 03:25:34.216213 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:34.216191    4496 scope.go:122] "RemoveContainer" containerID="2052defa4f40bf4d5fa2fb36a8a2f5b7fafb72462833025041a47ef9314e4c5e"
Nov 02 03:25:34.216510 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:25:34.216487    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2052defa4f40bf4d5fa2fb36a8a2f5b7fafb72462833025041a47ef9314e4c5e\": container with ID starting with 2052defa4f40bf4d5fa2fb36a8a2f5b7fafb72462833025041a47ef9314e4c5e not found: ID does not exist" containerID="2052defa4f40bf4d5fa2fb36a8a2f5b7fafb72462833025041a47ef9314e4c5e"
Nov 02 03:25:34.216608 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:34.216517    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2052defa4f40bf4d5fa2fb36a8a2f5b7fafb72462833025041a47ef9314e4c5e"} err="failed to get container status \"2052defa4f40bf4d5fa2fb36a8a2f5b7fafb72462833025041a47ef9314e4c5e\": rpc error: code = NotFound desc = could not find container \"2052defa4f40bf4d5fa2fb36a8a2f5b7fafb72462833025041a47ef9314e4c5e\": container with ID starting with 2052defa4f40bf4d5fa2fb36a8a2f5b7fafb72462833025041a47ef9314e4c5e not found: ID does not exist"
Nov 02 03:25:34.314146 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:34.313418    4496 patch_prober.go:29] interesting pod/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-t4dvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body=
Nov 02 03:25:34.314146 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:34.313501    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-t4dvl" podUID="013b67e2-aae9-4380-baf6-de93a6a9d18b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused"
Nov 02 03:25:34.480455 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:34.480430    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d5dcca72-nw5jm"
Nov 02 03:25:34.623331 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:34.623305    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42e1a6eb-7b6c-429a-b34e-f4a8f78e5981-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42e1a6eb-7b6c-429a-b34e-f4a8f78e5981-maas-api-xks-ca\") pod \"42e1a6eb-7b6c-429a-b34e-f4a8f78e5981\" (UID: \"42e1a6eb-7b6c-429a-b34e-f4a8f78e5981\") "
Nov 02 03:25:34.623726 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:34.623709    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42e1a6eb-7b6c-429a-b34e-f4a8f78e5981-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42e1a6eb-7b6c-429a-b34e-f4a8f78e5981-maas-api-openshift-ca\") pod \"42e1a6eb-7b6c-429a-b34e-f4a8f78e5981\" (UID: \"42e1a6eb-7b6c-429a-b34e-f4a8f78e5981\") "
Nov 02 03:25:34.623846 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:34.623835    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42e1a6eb-7b6c-429a-b34e-f4a8f78e5981-kube-api-access-6psrf\" (UniqueName: \"kubernetes.io/projected/42e1a6eb-7b6c-429a-b34e-f4a8f78e5981-kube-api-access-6psrf\") pod \"42e1a6eb-7b6c-429a-b34e-f4a8f78e5981\" (UID: \"42e1a6eb-7b6c-429a-b34e-f4a8f78e5981\") "
Nov 02 03:25:34.624853 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:34.624830    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42e1a6eb-7b6c-429a-b34e-f4a8f78e5981-maas-api-openshift-ca" pod "42e1a6eb-7b6c-429a-b34e-f4a8f78e5981" (UID: "42e1a6eb-7b6c-429a-b34e-f4a8f78e5981"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:25:34.626916 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:34.626889    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42e1a6eb-7b6c-429a-b34e-f4a8f78e5981-kube-api-access-6psrf" pod "42e1a6eb-7b6c-429a-b34e-f4a8f78e5981" (UID: "42e1a6eb-7b6c-429a-b34e-f4a8f78e5981"). InnerVolumeSpecName "kube-api-access-6psrf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:25:34.627085 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:34.627067    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42e1a6eb-7b6c-429a-b34e-f4a8f78e5981-maas-api-xks-ca" pod "42e1a6eb-7b6c-429a-b34e-f4a8f78e5981" (UID: "42e1a6eb-7b6c-429a-b34e-f4a8f78e5981"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 03:25:34.726159 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:34.726114    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42e1a6eb-7b6c-429a-b34e-f4a8f78e5981-maas-api-openshift-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:25:34.726159 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:34.726161    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6psrf\" (UniqueName: \"kubernetes.io/projected/42e1a6eb-7b6c-429a-b34e-f4a8f78e5981-kube-api-access-6psrf\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:25:34.726378 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:34.726181    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42e1a6eb-7b6c-429a-b34e-f4a8f78e5981-maas-api-xks-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:25:34.871443 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:34.871353    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0d26af-7m7t5"]
Nov 02 03:25:34.878099 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:34.878069    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0d26af-7m7t5"]
Nov 02 03:25:35.217804 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:35.217649    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d5dcca72-nw5jm"
Nov 02 03:25:35.217804 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:35.217679    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d5dcca72-nw5jm" event={"ID":"42e1a6eb-7b6c-429a-b34e-f4a8f78e5981","Type":"ContainerDied","Data":"3ac96acdfe9138cec05520516f42b0111c8b17f86704888b66deb5fc6f90d2d8"}
Nov 02 03:25:35.217804 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:35.217729    4496 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3ac96acdfe9138cec05520516f42b0111c8b17f86704888b66deb5fc6f90d2d8"
Nov 02 03:25:35.326122 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:35.325229    4496 patch_prober.go:29] interesting pod/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-t4dvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body=
Nov 02 03:25:35.326122 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:35.325299    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-t4dvl" podUID="013b67e2-aae9-4380-baf6-de93a6a9d18b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused"
Nov 02 03:25:35.585964 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:35.585930    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1faaebc-9cac-4b79-9324-59aeda0cdbfd" path="/var/lib/kubelet/pods/d1faaebc-9cac-4b79-9324-59aeda0cdbfd/volumes"
Nov 02 03:25:35.586478 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:35.586463    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9b6d8fd-c404-4c32-a6ae-8ba7b48314e3" path="/var/lib/kubelet/pods/f9b6d8fd-c404-4c32-a6ae-8ba7b48314e3/volumes"
Nov 02 03:25:35.998703 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:35.998669    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d6544b-wdg56"]
Nov 02 03:25:35.999441 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:35.999427    4496 cpu_manager.go:414] "containerMap: removing container" podUID="42e1a6eb-7b6c-429a-b34e-f4a8f78e5981" containerName="revoke-keys"
Nov 02 03:25:35.999494 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:35.999443    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42e1a6eb-7b6c-429a-b34e-f4a8f78e5981" containerName="revoke-keys"
Nov 02 03:25:35.999572 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:35.999563    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="42e1a6eb-7b6c-429a-b34e-f4a8f78e5981" containerName="revoke-keys"
Nov 02 03:25:36.004663 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:36.004646    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d6544b-wdg56"
Nov 02 03:25:36.071111 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:36.070639    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fa36a00-43b2-41d6-a789-fe09585a8917-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d6544b-wdg56\" (UID: \"0fa36a00-43b2-41d6-a789-fe09585a8917\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d6544b-wdg56"
Nov 02 03:25:36.071111 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:36.070773    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmkkx\" (UniqueName: \"kubernetes.io/projected/0fa36a00-43b2-41d6-a789-fe09585a8917-kube-api-access-qmkkx\") pod \"maas-api-revoke-keys-e2e-api-b-d6544b-wdg56\" (UID: \"0fa36a00-43b2-41d6-a789-fe09585a8917\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d6544b-wdg56"
Nov 02 03:25:36.071111 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:36.070866    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fa36a00-43b2-41d6-a789-fe09585a8917-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d6544b-wdg56\" (UID: \"0fa36a00-43b2-41d6-a789-fe09585a8917\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d6544b-wdg56"
Nov 02 03:25:36.075206 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:36.075176    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d6544b-wdg56"]
Nov 02 03:25:36.106336 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:36.106301    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 03:25:36.106336 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:36.106304    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 03:25:36.106336 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:36.106308    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 03:25:36.120102 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:36.120081    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9z5bk\""
Nov 02 03:25:36.173040 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:36.172090    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fa36a00-43b2-41d6-a789-fe09585a8917-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d6544b-wdg56\" (UID: \"0fa36a00-43b2-41d6-a789-fe09585a8917\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d6544b-wdg56"
Nov 02 03:25:36.173040 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:36.172305    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fa36a00-43b2-41d6-a789-fe09585a8917-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d6544b-wdg56\" (UID: \"0fa36a00-43b2-41d6-a789-fe09585a8917\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d6544b-wdg56"
Nov 02 03:25:36.173040 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:36.172422    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmkkx\" (UniqueName: \"kubernetes.io/projected/0fa36a00-43b2-41d6-a789-fe09585a8917-kube-api-access-qmkkx\") pod \"maas-api-revoke-keys-e2e-api-b-d6544b-wdg56\" (UID: \"0fa36a00-43b2-41d6-a789-fe09585a8917\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d6544b-wdg56"
Nov 02 03:25:36.174028 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:36.173936    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fa36a00-43b2-41d6-a789-fe09585a8917-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d6544b-wdg56\" (UID: \"0fa36a00-43b2-41d6-a789-fe09585a8917\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d6544b-wdg56"
Nov 02 03:25:36.176930 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:36.176903    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fa36a00-43b2-41d6-a789-fe09585a8917-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d6544b-wdg56\" (UID: \"0fa36a00-43b2-41d6-a789-fe09585a8917\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d6544b-wdg56"
Nov 02 03:25:36.230188 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:36.230162    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmkkx\" (UniqueName: \"kubernetes.io/projected/0fa36a00-43b2-41d6-a789-fe09585a8917-kube-api-access-qmkkx\") pod \"maas-api-revoke-keys-e2e-api-b-d6544b-wdg56\" (UID: \"0fa36a00-43b2-41d6-a789-fe09585a8917\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d6544b-wdg56"
Nov 02 03:25:36.310086 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:36.309194    4496 patch_prober.go:29] interesting pod/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-t4dvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body=
Nov 02 03:25:36.310086 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:36.309275    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-t4dvl" podUID="013b67e2-aae9-4380-baf6-de93a6a9d18b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused"
Nov 02 03:25:36.318540 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:36.318514    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d6544b-wdg56"
Nov 02 03:25:36.768361 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:36.768240    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d6544b-wdg56"]
Nov 02 03:25:36.770882 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:25:36.770849    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fa36a00_43b2_41d6_a789_fe09585a8917.slice/crio-879782d6533b9b6a52ff1db449dca80a984dc6b9398451a42e8356f5281a2679 WatchSource:0}: Error finding container 879782d6533b9b6a52ff1db449dca80a984dc6b9398451a42e8356f5281a2679: Status 404 returned error can't find the container with id 879782d6533b9b6a52ff1db449dca80a984dc6b9398451a42e8356f5281a2679
Nov 02 03:25:37.260047 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:37.257511    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d6544b-wdg56" event={"ID":"0fa36a00-43b2-41d6-a789-fe09585a8917","Type":"ContainerStarted","Data":"52b6c092bea015c787a59f09cc5577660e29e5bf7bb382a097f596b7143a3af5"}
Nov 02 03:25:37.260047 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:37.257578    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d6544b-wdg56" event={"ID":"0fa36a00-43b2-41d6-a789-fe09585a8917","Type":"ContainerStarted","Data":"879782d6533b9b6a52ff1db449dca80a984dc6b9398451a42e8356f5281a2679"}
Nov 02 03:25:37.309120 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:37.309079    4496 patch_prober.go:29] interesting pod/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-t4dvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body=
Nov 02 03:25:37.309292 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:37.309143    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-t4dvl" podUID="013b67e2-aae9-4380-baf6-de93a6a9d18b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused"
Nov 02 03:25:37.374618 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:37.374551    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d6544b-wdg56" podStartSLOduration=2.374533986 podStartE2EDuration="2.374533986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:25:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:25:37.372110724 +0000 UTC m=+2277.950718108" watchObservedRunningTime="2026-11-02 03:25:37.374533986 +0000 UTC m=+2277.953141365"
Nov 02 03:25:38.272818 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:38.272768    4496 generic.go:356] "Generic (PLEG): container finished" podID="0fa36a00-43b2-41d6-a789-fe09585a8917" containerID="52b6c092bea015c787a59f09cc5577660e29e5bf7bb382a097f596b7143a3af5" exitCode=0
Nov 02 03:25:38.273363 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:38.272950    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d6544b-wdg56" event={"ID":"0fa36a00-43b2-41d6-a789-fe09585a8917","Type":"ContainerDied","Data":"52b6c092bea015c787a59f09cc5577660e29e5bf7bb382a097f596b7143a3af5"}
Nov 02 03:25:38.415832 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:38.415798    4496 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-t4dvl"
Nov 02 03:25:38.416499 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:38.416470    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-t4dvl"
Nov 02 03:25:38.419214 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:38.419189    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-t4dvl"
Nov 02 03:25:39.470086 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:39.470056    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d6544b-wdg56"
Nov 02 03:25:39.533290 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:39.533213    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fa36a00-43b2-41d6-a789-fe09585a8917-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fa36a00-43b2-41d6-a789-fe09585a8917-maas-api-openshift-ca\") pod \"0fa36a00-43b2-41d6-a789-fe09585a8917\" (UID: \"0fa36a00-43b2-41d6-a789-fe09585a8917\") "
Nov 02 03:25:39.533290 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:39.533249    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fa36a00-43b2-41d6-a789-fe09585a8917-kube-api-access-qmkkx\" (UniqueName: \"kubernetes.io/projected/0fa36a00-43b2-41d6-a789-fe09585a8917-kube-api-access-qmkkx\") pod \"0fa36a00-43b2-41d6-a789-fe09585a8917\" (UID: \"0fa36a00-43b2-41d6-a789-fe09585a8917\") "
Nov 02 03:25:39.533539 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:39.533405    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0fa36a00-43b2-41d6-a789-fe09585a8917-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fa36a00-43b2-41d6-a789-fe09585a8917-maas-api-xks-ca\") pod \"0fa36a00-43b2-41d6-a789-fe09585a8917\" (UID: \"0fa36a00-43b2-41d6-a789-fe09585a8917\") "
Nov 02 03:25:39.533656 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:39.533625    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fa36a00-43b2-41d6-a789-fe09585a8917-maas-api-openshift-ca" pod "0fa36a00-43b2-41d6-a789-fe09585a8917" (UID: "0fa36a00-43b2-41d6-a789-fe09585a8917"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:25:39.534177 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:39.534157    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fa36a00-43b2-41d6-a789-fe09585a8917-maas-api-openshift-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:25:39.535750 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:39.535725    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fa36a00-43b2-41d6-a789-fe09585a8917-maas-api-xks-ca" pod "0fa36a00-43b2-41d6-a789-fe09585a8917" (UID: "0fa36a00-43b2-41d6-a789-fe09585a8917"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 03:25:39.535938 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:39.535915    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fa36a00-43b2-41d6-a789-fe09585a8917-kube-api-access-qmkkx" pod "0fa36a00-43b2-41d6-a789-fe09585a8917" (UID: "0fa36a00-43b2-41d6-a789-fe09585a8917"). InnerVolumeSpecName "kube-api-access-qmkkx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:25:39.642101 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:39.636319    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmkkx\" (UniqueName: \"kubernetes.io/projected/0fa36a00-43b2-41d6-a789-fe09585a8917-kube-api-access-qmkkx\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:25:39.642101 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:39.636349    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fa36a00-43b2-41d6-a789-fe09585a8917-maas-api-xks-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:25:40.291900 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:40.291859    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d6544b-wdg56"
Nov 02 03:25:40.291900 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:40.291891    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d6544b-wdg56" event={"ID":"0fa36a00-43b2-41d6-a789-fe09585a8917","Type":"ContainerDied","Data":"879782d6533b9b6a52ff1db449dca80a984dc6b9398451a42e8356f5281a2679"}
Nov 02 03:25:40.292162 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:40.291941    4496 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="879782d6533b9b6a52ff1db449dca80a984dc6b9398451a42e8356f5281a2679"
Nov 02 03:25:40.846593 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:40.846556    4496 scope.go:122] "RemoveContainer" containerID="15661fcf126c68ab7b9a91035d9b2482f4d0245deb28495677d9be78e2374c67"
Nov 02 03:25:40.858150 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:40.858117    4496 scope.go:122] "RemoveContainer" containerID="96c1606d177840247f80a4817ac84aa3611ff96f3ef679fca688a4cd87eedeb6"
Nov 02 03:25:40.869939 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:40.869914    4496 scope.go:122] "RemoveContainer" containerID="0211042c283a59769404ae5580a0facf4b51b060b6f8b7f0bfcd091dc3bdc67a"
Nov 02 03:25:40.882316 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:40.882283    4496 scope.go:122] "RemoveContainer" containerID="2b44f9e64c155f55ed911023b843f0631b20a8eb36843c324fa38f85827de085"
Nov 02 03:25:42.050432 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:42.050394    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-018cfa-94vgj"]
Nov 02 03:25:42.051189 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:42.051171    4496 cpu_manager.go:414] "containerMap: removing container" podUID="0fa36a00-43b2-41d6-a789-fe09585a8917" containerName="revoke-keys"
Nov 02 03:25:42.051189 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:42.051189    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fa36a00-43b2-41d6-a789-fe09585a8917" containerName="revoke-keys"
Nov 02 03:25:42.051297 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:42.051285    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fa36a00-43b2-41d6-a789-fe09585a8917" containerName="revoke-keys"
Nov 02 03:25:42.056941 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:42.056920    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-018cfa-94vgj"
Nov 02 03:25:42.109470 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:42.109428    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-018cfa-94vgj"]
Nov 02 03:25:42.155029 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:42.154976    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 03:25:42.155029 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:42.154987    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 03:25:42.155273 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:42.154988    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 03:25:42.160665 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:42.160639    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9z5bk\""
Nov 02 03:25:42.188423 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:42.188393    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d73da420-2bd5-41a2-b24b-e8601948fbd0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-018cfa-94vgj\" (UID: \"d73da420-2bd5-41a2-b24b-e8601948fbd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-018cfa-94vgj"
Nov 02 03:25:42.188603 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:42.188479    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sms8g\" (UniqueName: \"kubernetes.io/projected/d73da420-2bd5-41a2-b24b-e8601948fbd0-kube-api-access-sms8g\") pod \"maas-api-revoke-keys-e2e-api-b-018cfa-94vgj\" (UID: \"d73da420-2bd5-41a2-b24b-e8601948fbd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-018cfa-94vgj"
Nov 02 03:25:42.188603 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:42.188581    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d73da420-2bd5-41a2-b24b-e8601948fbd0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-018cfa-94vgj\" (UID: \"d73da420-2bd5-41a2-b24b-e8601948fbd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-018cfa-94vgj"
Nov 02 03:25:42.290645 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:42.290614    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sms8g\" (UniqueName: \"kubernetes.io/projected/d73da420-2bd5-41a2-b24b-e8601948fbd0-kube-api-access-sms8g\") pod \"maas-api-revoke-keys-e2e-api-b-018cfa-94vgj\" (UID: \"d73da420-2bd5-41a2-b24b-e8601948fbd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-018cfa-94vgj"
Nov 02 03:25:42.290798 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:42.290668    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d73da420-2bd5-41a2-b24b-e8601948fbd0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-018cfa-94vgj\" (UID: \"d73da420-2bd5-41a2-b24b-e8601948fbd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-018cfa-94vgj"
Nov 02 03:25:42.290856 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:42.290843    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d73da420-2bd5-41a2-b24b-e8601948fbd0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-018cfa-94vgj\" (UID: \"d73da420-2bd5-41a2-b24b-e8601948fbd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-018cfa-94vgj"
Nov 02 03:25:42.291478 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:42.291459    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d73da420-2bd5-41a2-b24b-e8601948fbd0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-018cfa-94vgj\" (UID: \"d73da420-2bd5-41a2-b24b-e8601948fbd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-018cfa-94vgj"
Nov 02 03:25:42.293025 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:42.292999    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d73da420-2bd5-41a2-b24b-e8601948fbd0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-018cfa-94vgj\" (UID: \"d73da420-2bd5-41a2-b24b-e8601948fbd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-018cfa-94vgj"
Nov 02 03:25:42.350577 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:42.350501    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sms8g\" (UniqueName: \"kubernetes.io/projected/d73da420-2bd5-41a2-b24b-e8601948fbd0-kube-api-access-sms8g\") pod \"maas-api-revoke-keys-e2e-api-b-018cfa-94vgj\" (UID: \"d73da420-2bd5-41a2-b24b-e8601948fbd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-018cfa-94vgj"
Nov 02 03:25:42.371463 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:42.371437    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-018cfa-94vgj"
Nov 02 03:25:42.756291 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:25:42.756254    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd73da420_2bd5_41a2_b24b_e8601948fbd0.slice/crio-0eae27e9ad7054445edbe56eee6f3712a70f0d2a130cc8ad7399bdf41f1895cc WatchSource:0}: Error finding container 0eae27e9ad7054445edbe56eee6f3712a70f0d2a130cc8ad7399bdf41f1895cc: Status 404 returned error can't find the container with id 0eae27e9ad7054445edbe56eee6f3712a70f0d2a130cc8ad7399bdf41f1895cc
Nov 02 03:25:42.757539 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:42.757511    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-018cfa-94vgj"]
Nov 02 03:25:43.534847 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:43.534802    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-018cfa-94vgj" event={"ID":"d73da420-2bd5-41a2-b24b-e8601948fbd0","Type":"ContainerStarted","Data":"e3c7e86c2a879a568b82c2d3bdaab03da451416ebe6baae03022c6c6514252ef"}
Nov 02 03:25:43.535338 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:43.534863    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-018cfa-94vgj" event={"ID":"d73da420-2bd5-41a2-b24b-e8601948fbd0","Type":"ContainerStarted","Data":"0eae27e9ad7054445edbe56eee6f3712a70f0d2a130cc8ad7399bdf41f1895cc"}
Nov 02 03:25:43.650549 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:43.650502    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-018cfa-94vgj" podStartSLOduration=1.650485499 podStartE2EDuration="1.650485499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:25:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:25:43.649693367 +0000 UTC m=+2284.228300747" watchObservedRunningTime="2026-11-02 03:25:43.650485499 +0000 UTC m=+2284.229092880"
Nov 02 03:25:43.976763 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:43.976732    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d5dcca72-846bb99569-sjdx4"]
Nov 02 03:25:44.093086 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:44.093031    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d5dcca72-7746cbb699-tmsrl"]
Nov 02 03:25:44.556110 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:44.556069    4496 generic.go:356] "Generic (PLEG): container finished" podID="d73da420-2bd5-41a2-b24b-e8601948fbd0" containerID="e3c7e86c2a879a568b82c2d3bdaab03da451416ebe6baae03022c6c6514252ef" exitCode=0
Nov 02 03:25:44.556576 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:44.556108    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-018cfa-94vgj" event={"ID":"d73da420-2bd5-41a2-b24b-e8601948fbd0","Type":"ContainerDied","Data":"e3c7e86c2a879a568b82c2d3bdaab03da451416ebe6baae03022c6c6514252ef"}
Nov 02 03:25:46.004467 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:46.004436    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-018cfa-94vgj"
Nov 02 03:25:46.149690 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:46.149653    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d73da420-2bd5-41a2-b24b-e8601948fbd0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d73da420-2bd5-41a2-b24b-e8601948fbd0-maas-api-xks-ca\") pod \"d73da420-2bd5-41a2-b24b-e8601948fbd0\" (UID: \"d73da420-2bd5-41a2-b24b-e8601948fbd0\") "
Nov 02 03:25:46.149890 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:46.149781    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d73da420-2bd5-41a2-b24b-e8601948fbd0-kube-api-access-sms8g\" (UniqueName: \"kubernetes.io/projected/d73da420-2bd5-41a2-b24b-e8601948fbd0-kube-api-access-sms8g\") pod \"d73da420-2bd5-41a2-b24b-e8601948fbd0\" (UID: \"d73da420-2bd5-41a2-b24b-e8601948fbd0\") "
Nov 02 03:25:46.149890 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:46.149825    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d73da420-2bd5-41a2-b24b-e8601948fbd0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d73da420-2bd5-41a2-b24b-e8601948fbd0-maas-api-openshift-ca\") pod \"d73da420-2bd5-41a2-b24b-e8601948fbd0\" (UID: \"d73da420-2bd5-41a2-b24b-e8601948fbd0\") "
Nov 02 03:25:46.150802 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:46.150777    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d73da420-2bd5-41a2-b24b-e8601948fbd0-maas-api-openshift-ca" pod "d73da420-2bd5-41a2-b24b-e8601948fbd0" (UID: "d73da420-2bd5-41a2-b24b-e8601948fbd0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:25:46.192373 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:46.192340    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d73da420-2bd5-41a2-b24b-e8601948fbd0-maas-api-xks-ca" pod "d73da420-2bd5-41a2-b24b-e8601948fbd0" (UID: "d73da420-2bd5-41a2-b24b-e8601948fbd0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 03:25:46.193787 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:46.193753    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d73da420-2bd5-41a2-b24b-e8601948fbd0-kube-api-access-sms8g" pod "d73da420-2bd5-41a2-b24b-e8601948fbd0" (UID: "d73da420-2bd5-41a2-b24b-e8601948fbd0"). InnerVolumeSpecName "kube-api-access-sms8g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:25:46.251794 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:46.251761    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d73da420-2bd5-41a2-b24b-e8601948fbd0-maas-api-xks-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:25:46.251794 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:46.251792    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sms8g\" (UniqueName: \"kubernetes.io/projected/d73da420-2bd5-41a2-b24b-e8601948fbd0-kube-api-access-sms8g\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:25:46.251990 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:46.251802    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d73da420-2bd5-41a2-b24b-e8601948fbd0-maas-api-openshift-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:25:46.559957 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:46.559874    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d6544b-5fc969d5f9-7k5sq"]
Nov 02 03:25:46.614709 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:46.614676    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-018cfa-94vgj"
Nov 02 03:25:46.614871 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:46.614751    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-018cfa-94vgj" event={"ID":"d73da420-2bd5-41a2-b24b-e8601948fbd0","Type":"ContainerDied","Data":"0eae27e9ad7054445edbe56eee6f3712a70f0d2a130cc8ad7399bdf41f1895cc"}
Nov 02 03:25:46.614871 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:46.614811    4496 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0eae27e9ad7054445edbe56eee6f3712a70f0d2a130cc8ad7399bdf41f1895cc"
Nov 02 03:25:46.664636 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:46.664599    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d6544b-75f96f97f-6m8n2"]
Nov 02 03:25:49.094120 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:49.094077    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-d5dcca72-7746cbb699-tmsrl" podUID="010d8313-308d-4bce-a4fa-277ead1e6d63" containerName="payload-pre-processing" containerID="cri-o://33b5621da04e1a4b0cc030c775712482ccadb8c8ab960b24c346489f24907235" gracePeriod=25
Nov 02 03:25:49.391612 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:49.391588    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d5dcca72-7746cbb699-tmsrl"
Nov 02 03:25:49.406906 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:49.406873    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/010d8313-308d-4bce-a4fa-277ead1e6d63-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/010d8313-308d-4bce-a4fa-277ead1e6d63-plugins-config-volume\") pod \"010d8313-308d-4bce-a4fa-277ead1e6d63\" (UID: \"010d8313-308d-4bce-a4fa-277ead1e6d63\") "
Nov 02 03:25:49.407065 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:49.406957    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/010d8313-308d-4bce-a4fa-277ead1e6d63-kube-api-access-gfmck\" (UniqueName: \"kubernetes.io/projected/010d8313-308d-4bce-a4fa-277ead1e6d63-kube-api-access-gfmck\") pod \"010d8313-308d-4bce-a4fa-277ead1e6d63\" (UID: \"010d8313-308d-4bce-a4fa-277ead1e6d63\") "
Nov 02 03:25:49.407327 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:49.407303    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/010d8313-308d-4bce-a4fa-277ead1e6d63-plugins-config-volume" pod "010d8313-308d-4bce-a4fa-277ead1e6d63" (UID: "010d8313-308d-4bce-a4fa-277ead1e6d63"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:25:49.407596 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:49.407579    4496 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/010d8313-308d-4bce-a4fa-277ead1e6d63-plugins-config-volume\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:25:49.409495 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:49.409459    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/010d8313-308d-4bce-a4fa-277ead1e6d63-kube-api-access-gfmck" pod "010d8313-308d-4bce-a4fa-277ead1e6d63" (UID: "010d8313-308d-4bce-a4fa-277ead1e6d63"). InnerVolumeSpecName "kube-api-access-gfmck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:25:49.509627 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:49.509590    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfmck\" (UniqueName: \"kubernetes.io/projected/010d8313-308d-4bce-a4fa-277ead1e6d63-kube-api-access-gfmck\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:25:49.557242 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:49.557197    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-d5dcca72-846bb99569-sjdx4" podUID="ba6ad9c9-ab6b-43cd-bd4b-c7042792d7f3" containerName="payload-processing" containerID="cri-o://842acaea42c2fa883c1e06099997759df180b89272c00304f20a2adebc314e5d" gracePeriod=25
Nov 02 03:25:49.641699 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:49.641672    4496 generic.go:356] "Generic (PLEG): container finished" podID="010d8313-308d-4bce-a4fa-277ead1e6d63" containerID="33b5621da04e1a4b0cc030c775712482ccadb8c8ab960b24c346489f24907235" exitCode=0
Nov 02 03:25:49.641829 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:49.641744    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d5dcca72-7746cbb699-tmsrl"
Nov 02 03:25:49.641829 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:49.641768    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d5dcca72-7746cbb699-tmsrl" event={"ID":"010d8313-308d-4bce-a4fa-277ead1e6d63","Type":"ContainerDied","Data":"33b5621da04e1a4b0cc030c775712482ccadb8c8ab960b24c346489f24907235"}
Nov 02 03:25:49.641898 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:49.641835    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d5dcca72-7746cbb699-tmsrl" event={"ID":"010d8313-308d-4bce-a4fa-277ead1e6d63","Type":"ContainerDied","Data":"b849487e2a1a06e97d4e664ff270aceb367d22656da3441a853fbf8c8eccabf8"}
Nov 02 03:25:49.641898 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:49.641866    4496 scope.go:122] "RemoveContainer" containerID="33b5621da04e1a4b0cc030c775712482ccadb8c8ab960b24c346489f24907235"
Nov 02 03:25:49.674223 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:49.674074    4496 scope.go:122] "RemoveContainer" containerID="33b5621da04e1a4b0cc030c775712482ccadb8c8ab960b24c346489f24907235"
Nov 02 03:25:49.680182 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:25:49.680152    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33b5621da04e1a4b0cc030c775712482ccadb8c8ab960b24c346489f24907235\": container with ID starting with 33b5621da04e1a4b0cc030c775712482ccadb8c8ab960b24c346489f24907235 not found: ID does not exist" containerID="33b5621da04e1a4b0cc030c775712482ccadb8c8ab960b24c346489f24907235"
Nov 02 03:25:49.680294 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:49.680190    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33b5621da04e1a4b0cc030c775712482ccadb8c8ab960b24c346489f24907235"} err="failed to get container status \"33b5621da04e1a4b0cc030c775712482ccadb8c8ab960b24c346489f24907235\": rpc error: code = NotFound desc = could not find container \"33b5621da04e1a4b0cc030c775712482ccadb8c8ab960b24c346489f24907235\": container with ID starting with 33b5621da04e1a4b0cc030c775712482ccadb8c8ab960b24c346489f24907235 not found: ID does not exist"
Nov 02 03:25:49.812896 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:49.812836    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d5dcca72-7746cbb699-tmsrl"]
Nov 02 03:25:49.818371 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:49.818347    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d5dcca72-7746cbb699-tmsrl"]
Nov 02 03:25:49.827924 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:49.827897    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d5dcca72-846bb99569-sjdx4"
Nov 02 03:25:49.914074 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:49.913989    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba6ad9c9-ab6b-43cd-bd4b-c7042792d7f3-kube-api-access-gxvgp\" (UniqueName: \"kubernetes.io/projected/ba6ad9c9-ab6b-43cd-bd4b-c7042792d7f3-kube-api-access-gxvgp\") pod \"ba6ad9c9-ab6b-43cd-bd4b-c7042792d7f3\" (UID: \"ba6ad9c9-ab6b-43cd-bd4b-c7042792d7f3\") "
Nov 02 03:25:49.914200 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:49.914122    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba6ad9c9-ab6b-43cd-bd4b-c7042792d7f3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba6ad9c9-ab6b-43cd-bd4b-c7042792d7f3-plugins-config-volume\") pod \"ba6ad9c9-ab6b-43cd-bd4b-c7042792d7f3\" (UID: \"ba6ad9c9-ab6b-43cd-bd4b-c7042792d7f3\") "
Nov 02 03:25:49.914476 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:49.914440    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba6ad9c9-ab6b-43cd-bd4b-c7042792d7f3-plugins-config-volume" pod "ba6ad9c9-ab6b-43cd-bd4b-c7042792d7f3" (UID: "ba6ad9c9-ab6b-43cd-bd4b-c7042792d7f3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:25:49.914624 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:49.914607    4496 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba6ad9c9-ab6b-43cd-bd4b-c7042792d7f3-plugins-config-volume\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:25:49.916111 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:49.916091    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba6ad9c9-ab6b-43cd-bd4b-c7042792d7f3-kube-api-access-gxvgp" pod "ba6ad9c9-ab6b-43cd-bd4b-c7042792d7f3" (UID: "ba6ad9c9-ab6b-43cd-bd4b-c7042792d7f3"). InnerVolumeSpecName "kube-api-access-gxvgp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:25:50.031554 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:50.031469    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxvgp\" (UniqueName: \"kubernetes.io/projected/ba6ad9c9-ab6b-43cd-bd4b-c7042792d7f3-kube-api-access-gxvgp\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:25:50.652846 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:50.652810    4496 generic.go:356] "Generic (PLEG): container finished" podID="ba6ad9c9-ab6b-43cd-bd4b-c7042792d7f3" containerID="842acaea42c2fa883c1e06099997759df180b89272c00304f20a2adebc314e5d" exitCode=0
Nov 02 03:25:50.653346 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:50.652857    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d5dcca72-846bb99569-sjdx4"
Nov 02 03:25:50.653346 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:50.652908    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d5dcca72-846bb99569-sjdx4" event={"ID":"ba6ad9c9-ab6b-43cd-bd4b-c7042792d7f3","Type":"ContainerDied","Data":"842acaea42c2fa883c1e06099997759df180b89272c00304f20a2adebc314e5d"}
Nov 02 03:25:50.653346 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:50.652970    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d5dcca72-846bb99569-sjdx4" event={"ID":"ba6ad9c9-ab6b-43cd-bd4b-c7042792d7f3","Type":"ContainerDied","Data":"a93dfd5db852294bb852807a610c65fff9b94d0ff81c4d44db868d8df8a87a3a"}
Nov 02 03:25:50.653346 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:50.653002    4496 scope.go:122] "RemoveContainer" containerID="842acaea42c2fa883c1e06099997759df180b89272c00304f20a2adebc314e5d"
Nov 02 03:25:50.665069 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:50.665046    4496 scope.go:122] "RemoveContainer" containerID="842acaea42c2fa883c1e06099997759df180b89272c00304f20a2adebc314e5d"
Nov 02 03:25:50.665349 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:25:50.665330    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"842acaea42c2fa883c1e06099997759df180b89272c00304f20a2adebc314e5d\": container with ID starting with 842acaea42c2fa883c1e06099997759df180b89272c00304f20a2adebc314e5d not found: ID does not exist" containerID="842acaea42c2fa883c1e06099997759df180b89272c00304f20a2adebc314e5d"
Nov 02 03:25:50.665431 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:50.665358    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"842acaea42c2fa883c1e06099997759df180b89272c00304f20a2adebc314e5d"} err="failed to get container status \"842acaea42c2fa883c1e06099997759df180b89272c00304f20a2adebc314e5d\": rpc error: code = NotFound desc = could not find container \"842acaea42c2fa883c1e06099997759df180b89272c00304f20a2adebc314e5d\": container with ID starting with 842acaea42c2fa883c1e06099997759df180b89272c00304f20a2adebc314e5d not found: ID does not exist"
Nov 02 03:25:50.821621 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:50.821586    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d5dcca72-846bb99569-sjdx4"]
Nov 02 03:25:50.874643 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:50.874606    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d5dcca72-846bb99569-sjdx4"]
Nov 02 03:25:51.561151 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:51.561114    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-d6544b-5fc969d5f9-7k5sq" podUID="25ab21e1-7f9b-4017-86bd-b3e45bb4d823" containerName="payload-processing" containerID="cri-o://691b36370c3cb7c848f814bd4e66be5e6018279c746d0ce079e5140ba7caee7c" gracePeriod=25
Nov 02 03:25:51.579384 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:51.579350    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="010d8313-308d-4bce-a4fa-277ead1e6d63" path="/var/lib/kubelet/pods/010d8313-308d-4bce-a4fa-277ead1e6d63/volumes"
Nov 02 03:25:51.579974 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:51.579957    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba6ad9c9-ab6b-43cd-bd4b-c7042792d7f3" path="/var/lib/kubelet/pods/ba6ad9c9-ab6b-43cd-bd4b-c7042792d7f3/volumes"
Nov 02 03:25:51.665750 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:51.665710    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d6544b-75f96f97f-6m8n2" podUID="ed8d5844-0ec4-4050-959c-f8864624ebd5" containerName="payload-pre-processing" containerID="cri-o://78ce13fdaaf013b61afcd17ecad524a5dcb9e83ed635961327ee0e48bc138abe" gracePeriod=25
Nov 02 03:25:51.873728 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:51.873694    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d6544b-5fc969d5f9-7k5sq"
Nov 02 03:25:51.960185 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:51.960162    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25ab21e1-7f9b-4017-86bd-b3e45bb4d823-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25ab21e1-7f9b-4017-86bd-b3e45bb4d823-plugins-config-volume\") pod \"25ab21e1-7f9b-4017-86bd-b3e45bb4d823\" (UID: \"25ab21e1-7f9b-4017-86bd-b3e45bb4d823\") "
Nov 02 03:25:51.960312 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:51.960269    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25ab21e1-7f9b-4017-86bd-b3e45bb4d823-kube-api-access-brh4g\" (UniqueName: \"kubernetes.io/projected/25ab21e1-7f9b-4017-86bd-b3e45bb4d823-kube-api-access-brh4g\") pod \"25ab21e1-7f9b-4017-86bd-b3e45bb4d823\" (UID: \"25ab21e1-7f9b-4017-86bd-b3e45bb4d823\") "
Nov 02 03:25:51.960635 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:51.960611    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25ab21e1-7f9b-4017-86bd-b3e45bb4d823-plugins-config-volume" pod "25ab21e1-7f9b-4017-86bd-b3e45bb4d823" (UID: "25ab21e1-7f9b-4017-86bd-b3e45bb4d823"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:25:51.960784 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:51.960765    4496 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25ab21e1-7f9b-4017-86bd-b3e45bb4d823-plugins-config-volume\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:25:51.962572 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:51.962548    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25ab21e1-7f9b-4017-86bd-b3e45bb4d823-kube-api-access-brh4g" pod "25ab21e1-7f9b-4017-86bd-b3e45bb4d823" (UID: "25ab21e1-7f9b-4017-86bd-b3e45bb4d823"). InnerVolumeSpecName "kube-api-access-brh4g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:25:51.984056 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:51.983995    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d6544b-75f96f97f-6m8n2"
Nov 02 03:25:52.062449 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:52.062412    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed8d5844-0ec4-4050-959c-f8864624ebd5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed8d5844-0ec4-4050-959c-f8864624ebd5-plugins-config-volume\") pod \"ed8d5844-0ec4-4050-959c-f8864624ebd5\" (UID: \"ed8d5844-0ec4-4050-959c-f8864624ebd5\") "
Nov 02 03:25:52.062643 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:52.062624    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed8d5844-0ec4-4050-959c-f8864624ebd5-kube-api-access-czmvp\" (UniqueName: \"kubernetes.io/projected/ed8d5844-0ec4-4050-959c-f8864624ebd5-kube-api-access-czmvp\") pod \"ed8d5844-0ec4-4050-959c-f8864624ebd5\" (UID: \"ed8d5844-0ec4-4050-959c-f8864624ebd5\") "
Nov 02 03:25:52.062824 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:52.062800    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed8d5844-0ec4-4050-959c-f8864624ebd5-plugins-config-volume" pod "ed8d5844-0ec4-4050-959c-f8864624ebd5" (UID: "ed8d5844-0ec4-4050-959c-f8864624ebd5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:25:52.063496 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:52.063472    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brh4g\" (UniqueName: \"kubernetes.io/projected/25ab21e1-7f9b-4017-86bd-b3e45bb4d823-kube-api-access-brh4g\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:25:52.063577 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:52.063502    4496 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed8d5844-0ec4-4050-959c-f8864624ebd5-plugins-config-volume\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:25:52.065320 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:52.065297    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed8d5844-0ec4-4050-959c-f8864624ebd5-kube-api-access-czmvp" pod "ed8d5844-0ec4-4050-959c-f8864624ebd5" (UID: "ed8d5844-0ec4-4050-959c-f8864624ebd5"). InnerVolumeSpecName "kube-api-access-czmvp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:25:52.168823 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:52.166189    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czmvp\" (UniqueName: \"kubernetes.io/projected/ed8d5844-0ec4-4050-959c-f8864624ebd5-kube-api-access-czmvp\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:25:52.675883 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:52.675846    4496 generic.go:356] "Generic (PLEG): container finished" podID="25ab21e1-7f9b-4017-86bd-b3e45bb4d823" containerID="691b36370c3cb7c848f814bd4e66be5e6018279c746d0ce079e5140ba7caee7c" exitCode=0
Nov 02 03:25:52.676408 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:52.675900    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d6544b-5fc969d5f9-7k5sq"
Nov 02 03:25:52.676408 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:52.675929    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d6544b-5fc969d5f9-7k5sq" event={"ID":"25ab21e1-7f9b-4017-86bd-b3e45bb4d823","Type":"ContainerDied","Data":"691b36370c3cb7c848f814bd4e66be5e6018279c746d0ce079e5140ba7caee7c"}
Nov 02 03:25:52.676408 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:52.675999    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d6544b-5fc969d5f9-7k5sq" event={"ID":"25ab21e1-7f9b-4017-86bd-b3e45bb4d823","Type":"ContainerDied","Data":"d7bb8d60bd9ed8be60488fcd8f0d660ea53bbe320e1dae547d5342eafe836f00"}
Nov 02 03:25:52.676408 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:52.676047    4496 scope.go:122] "RemoveContainer" containerID="691b36370c3cb7c848f814bd4e66be5e6018279c746d0ce079e5140ba7caee7c"
Nov 02 03:25:52.678794 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:52.678774    4496 generic.go:356] "Generic (PLEG): container finished" podID="ed8d5844-0ec4-4050-959c-f8864624ebd5" containerID="78ce13fdaaf013b61afcd17ecad524a5dcb9e83ed635961327ee0e48bc138abe" exitCode=0
Nov 02 03:25:52.678913 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:52.678823    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d6544b-75f96f97f-6m8n2" event={"ID":"ed8d5844-0ec4-4050-959c-f8864624ebd5","Type":"ContainerDied","Data":"78ce13fdaaf013b61afcd17ecad524a5dcb9e83ed635961327ee0e48bc138abe"}
Nov 02 03:25:52.678913 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:52.678831    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d6544b-75f96f97f-6m8n2"
Nov 02 03:25:52.678913 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:52.678863    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d6544b-75f96f97f-6m8n2" event={"ID":"ed8d5844-0ec4-4050-959c-f8864624ebd5","Type":"ContainerDied","Data":"a565b0ad63ce42b490f8c6811ebc5e50342c640db9549dfcd190d9c69dd8f7fd"}
Nov 02 03:25:52.689980 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:52.689965    4496 scope.go:122] "RemoveContainer" containerID="691b36370c3cb7c848f814bd4e66be5e6018279c746d0ce079e5140ba7caee7c"
Nov 02 03:25:52.690347 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:25:52.690327    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"691b36370c3cb7c848f814bd4e66be5e6018279c746d0ce079e5140ba7caee7c\": container with ID starting with 691b36370c3cb7c848f814bd4e66be5e6018279c746d0ce079e5140ba7caee7c not found: ID does not exist" containerID="691b36370c3cb7c848f814bd4e66be5e6018279c746d0ce079e5140ba7caee7c"
Nov 02 03:25:52.690395 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:52.690353    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"691b36370c3cb7c848f814bd4e66be5e6018279c746d0ce079e5140ba7caee7c"} err="failed to get container status \"691b36370c3cb7c848f814bd4e66be5e6018279c746d0ce079e5140ba7caee7c\": rpc error: code = NotFound desc = could not find container \"691b36370c3cb7c848f814bd4e66be5e6018279c746d0ce079e5140ba7caee7c\": container with ID starting with 691b36370c3cb7c848f814bd4e66be5e6018279c746d0ce079e5140ba7caee7c not found: ID does not exist"
Nov 02 03:25:52.690395 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:52.690374    4496 scope.go:122] "RemoveContainer" containerID="78ce13fdaaf013b61afcd17ecad524a5dcb9e83ed635961327ee0e48bc138abe"
Nov 02 03:25:52.699562 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:52.699546    4496 scope.go:122] "RemoveContainer" containerID="78ce13fdaaf013b61afcd17ecad524a5dcb9e83ed635961327ee0e48bc138abe"
Nov 02 03:25:52.699791 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:25:52.699774    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78ce13fdaaf013b61afcd17ecad524a5dcb9e83ed635961327ee0e48bc138abe\": container with ID starting with 78ce13fdaaf013b61afcd17ecad524a5dcb9e83ed635961327ee0e48bc138abe not found: ID does not exist" containerID="78ce13fdaaf013b61afcd17ecad524a5dcb9e83ed635961327ee0e48bc138abe"
Nov 02 03:25:52.699836 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:52.699798    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78ce13fdaaf013b61afcd17ecad524a5dcb9e83ed635961327ee0e48bc138abe"} err="failed to get container status \"78ce13fdaaf013b61afcd17ecad524a5dcb9e83ed635961327ee0e48bc138abe\": rpc error: code = NotFound desc = could not find container \"78ce13fdaaf013b61afcd17ecad524a5dcb9e83ed635961327ee0e48bc138abe\": container with ID starting with 78ce13fdaaf013b61afcd17ecad524a5dcb9e83ed635961327ee0e48bc138abe not found: ID does not exist"
Nov 02 03:25:52.848273 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:52.848190    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d6544b-75f96f97f-6m8n2"]
Nov 02 03:25:52.896204 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:52.896174    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d6544b-75f96f97f-6m8n2"]
Nov 02 03:25:52.957240 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:52.957209    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d6544b-5fc969d5f9-7k5sq"]
Nov 02 03:25:52.961479 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:52.961443    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d6544b-5fc969d5f9-7k5sq"]
Nov 02 03:25:53.544662 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:53.544623    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-018cfa-74666c99c9-pv5nv"]
Nov 02 03:25:53.577564 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:53.577530    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25ab21e1-7f9b-4017-86bd-b3e45bb4d823" path="/var/lib/kubelet/pods/25ab21e1-7f9b-4017-86bd-b3e45bb4d823/volumes"
Nov 02 03:25:53.578267 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:53.578243    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed8d5844-0ec4-4050-959c-f8864624ebd5" path="/var/lib/kubelet/pods/ed8d5844-0ec4-4050-959c-f8864624ebd5/volumes"
Nov 02 03:25:53.661275 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:53.661241    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-018cfa-7979545787-tsnwm"]
Nov 02 03:25:58.544952 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:58.544894    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-018cfa-74666c99c9-pv5nv" podUID="839528ca-8c68-42e2-9a79-a926450a021d" containerName="payload-processing" containerID="cri-o://a79a4369ff678a6a1d51ea993a1ea7df2d6b694190012e23e5250944a35dac80" gracePeriod=25
Nov 02 03:25:58.662158 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:58.662126    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-018cfa-7979545787-tsnwm" podUID="8db02a90-7174-4696-90b4-5fc024addbc1" containerName="payload-pre-processing" containerID="cri-o://fb9003858c3d7b0464cc6312f1937d366f1bcc2ee76e807047e4db1b834a08ed" gracePeriod=25
Nov 02 03:25:58.735933 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:58.735863    4496 generic.go:356] "Generic (PLEG): container finished" podID="839528ca-8c68-42e2-9a79-a926450a021d" containerID="a79a4369ff678a6a1d51ea993a1ea7df2d6b694190012e23e5250944a35dac80" exitCode=0
Nov 02 03:25:58.736199 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:58.736177    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-018cfa-74666c99c9-pv5nv" event={"ID":"839528ca-8c68-42e2-9a79-a926450a021d","Type":"ContainerDied","Data":"a79a4369ff678a6a1d51ea993a1ea7df2d6b694190012e23e5250944a35dac80"}
Nov 02 03:25:58.932595 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:58.932210    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-018cfa-74666c99c9-pv5nv"
Nov 02 03:25:59.109490 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:59.109457    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-018cfa-7979545787-tsnwm"
Nov 02 03:25:59.114614 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:59.114592    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/839528ca-8c68-42e2-9a79-a926450a021d-kube-api-access-wskzv\" (UniqueName: \"kubernetes.io/projected/839528ca-8c68-42e2-9a79-a926450a021d-kube-api-access-wskzv\") pod \"839528ca-8c68-42e2-9a79-a926450a021d\" (UID: \"839528ca-8c68-42e2-9a79-a926450a021d\") "
Nov 02 03:25:59.114808 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:59.114796    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/839528ca-8c68-42e2-9a79-a926450a021d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/839528ca-8c68-42e2-9a79-a926450a021d-plugins-config-volume\") pod \"839528ca-8c68-42e2-9a79-a926450a021d\" (UID: \"839528ca-8c68-42e2-9a79-a926450a021d\") "
Nov 02 03:25:59.115155 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:59.115131    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/839528ca-8c68-42e2-9a79-a926450a021d-plugins-config-volume" pod "839528ca-8c68-42e2-9a79-a926450a021d" (UID: "839528ca-8c68-42e2-9a79-a926450a021d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:25:59.116667 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:59.116638    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/839528ca-8c68-42e2-9a79-a926450a021d-kube-api-access-wskzv" pod "839528ca-8c68-42e2-9a79-a926450a021d" (UID: "839528ca-8c68-42e2-9a79-a926450a021d"). InnerVolumeSpecName "kube-api-access-wskzv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:25:59.216200 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:59.216172    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8db02a90-7174-4696-90b4-5fc024addbc1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8db02a90-7174-4696-90b4-5fc024addbc1-plugins-config-volume\") pod \"8db02a90-7174-4696-90b4-5fc024addbc1\" (UID: \"8db02a90-7174-4696-90b4-5fc024addbc1\") "
Nov 02 03:25:59.216366 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:59.216208    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8db02a90-7174-4696-90b4-5fc024addbc1-kube-api-access-hzf54\" (UniqueName: \"kubernetes.io/projected/8db02a90-7174-4696-90b4-5fc024addbc1-kube-api-access-hzf54\") pod \"8db02a90-7174-4696-90b4-5fc024addbc1\" (UID: \"8db02a90-7174-4696-90b4-5fc024addbc1\") "
Nov 02 03:25:59.216657 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:59.216625    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8db02a90-7174-4696-90b4-5fc024addbc1-plugins-config-volume" pod "8db02a90-7174-4696-90b4-5fc024addbc1" (UID: "8db02a90-7174-4696-90b4-5fc024addbc1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:25:59.216937 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:59.216920    4496 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/839528ca-8c68-42e2-9a79-a926450a021d-plugins-config-volume\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:25:59.217003 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:59.216938    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wskzv\" (UniqueName: \"kubernetes.io/projected/839528ca-8c68-42e2-9a79-a926450a021d-kube-api-access-wskzv\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:25:59.217003 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:59.216948    4496 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8db02a90-7174-4696-90b4-5fc024addbc1-plugins-config-volume\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:25:59.218511 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:59.218488    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8db02a90-7174-4696-90b4-5fc024addbc1-kube-api-access-hzf54" pod "8db02a90-7174-4696-90b4-5fc024addbc1" (UID: "8db02a90-7174-4696-90b4-5fc024addbc1"). InnerVolumeSpecName "kube-api-access-hzf54". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:25:59.318294 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:59.318197    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzf54\" (UniqueName: \"kubernetes.io/projected/8db02a90-7174-4696-90b4-5fc024addbc1-kube-api-access-hzf54\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:25:59.746387 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:59.746351    4496 generic.go:356] "Generic (PLEG): container finished" podID="8db02a90-7174-4696-90b4-5fc024addbc1" containerID="fb9003858c3d7b0464cc6312f1937d366f1bcc2ee76e807047e4db1b834a08ed" exitCode=0
Nov 02 03:25:59.746801 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:59.746411    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-018cfa-7979545787-tsnwm"
Nov 02 03:25:59.746801 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:59.746433    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-018cfa-7979545787-tsnwm" event={"ID":"8db02a90-7174-4696-90b4-5fc024addbc1","Type":"ContainerDied","Data":"fb9003858c3d7b0464cc6312f1937d366f1bcc2ee76e807047e4db1b834a08ed"}
Nov 02 03:25:59.746801 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:59.746502    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-018cfa-7979545787-tsnwm" event={"ID":"8db02a90-7174-4696-90b4-5fc024addbc1","Type":"ContainerDied","Data":"09ddec3aa75db10608584a694976ec746e3d92fbb968d3213ff0c3e15704e12a"}
Nov 02 03:25:59.746801 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:59.746531    4496 scope.go:122] "RemoveContainer" containerID="fb9003858c3d7b0464cc6312f1937d366f1bcc2ee76e807047e4db1b834a08ed"
Nov 02 03:25:59.748387 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:59.748360    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-018cfa-74666c99c9-pv5nv" event={"ID":"839528ca-8c68-42e2-9a79-a926450a021d","Type":"ContainerDied","Data":"84df445189e616bbc98b6dfd8093f88a820f1c5b47fc4e2e1dbf8c19a558e980"}
Nov 02 03:25:59.748478 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:59.748421    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-018cfa-74666c99c9-pv5nv"
Nov 02 03:25:59.758384 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:59.758362    4496 scope.go:122] "RemoveContainer" containerID="fb9003858c3d7b0464cc6312f1937d366f1bcc2ee76e807047e4db1b834a08ed"
Nov 02 03:25:59.758675 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:25:59.758651    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb9003858c3d7b0464cc6312f1937d366f1bcc2ee76e807047e4db1b834a08ed\": container with ID starting with fb9003858c3d7b0464cc6312f1937d366f1bcc2ee76e807047e4db1b834a08ed not found: ID does not exist" containerID="fb9003858c3d7b0464cc6312f1937d366f1bcc2ee76e807047e4db1b834a08ed"
Nov 02 03:25:59.758737 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:59.758683    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb9003858c3d7b0464cc6312f1937d366f1bcc2ee76e807047e4db1b834a08ed"} err="failed to get container status \"fb9003858c3d7b0464cc6312f1937d366f1bcc2ee76e807047e4db1b834a08ed\": rpc error: code = NotFound desc = could not find container \"fb9003858c3d7b0464cc6312f1937d366f1bcc2ee76e807047e4db1b834a08ed\": container with ID starting with fb9003858c3d7b0464cc6312f1937d366f1bcc2ee76e807047e4db1b834a08ed not found: ID does not exist"
Nov 02 03:25:59.758737 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:59.758699    4496 scope.go:122] "RemoveContainer" containerID="a79a4369ff678a6a1d51ea993a1ea7df2d6b694190012e23e5250944a35dac80"
Nov 02 03:25:59.917397 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:59.917320    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-018cfa-7979545787-tsnwm"]
Nov 02 03:25:59.967603 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:25:59.967562    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-018cfa-7979545787-tsnwm"]
Nov 02 03:26:00.078742 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:00.078705    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-018cfa-74666c99c9-pv5nv"]
Nov 02 03:26:00.082836 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:00.082809    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-018cfa-74666c99c9-pv5nv"]
Nov 02 03:26:01.575800 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:01.575769    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="839528ca-8c68-42e2-9a79-a926450a021d" path="/var/lib/kubelet/pods/839528ca-8c68-42e2-9a79-a926450a021d/volumes"
Nov 02 03:26:01.576242 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:01.576227    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8db02a90-7174-4696-90b4-5fc024addbc1" path="/var/lib/kubelet/pods/8db02a90-7174-4696-90b4-5fc024addbc1/volumes"
Nov 02 03:26:03.632288 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:26:03.632241    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 03:26:04.809147 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:04.809088    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c9d1fc-wcp6v"]
Nov 02 03:26:04.810302 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:04.810277    4496 cpu_manager.go:414] "containerMap: removing container" podUID="25ab21e1-7f9b-4017-86bd-b3e45bb4d823" containerName="payload-processing"
Nov 02 03:26:04.810302 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:04.810299    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25ab21e1-7f9b-4017-86bd-b3e45bb4d823" containerName="payload-processing"
Nov 02 03:26:04.810580 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:04.810340    4496 cpu_manager.go:414] "containerMap: removing container" podUID="839528ca-8c68-42e2-9a79-a926450a021d" containerName="payload-processing"
Nov 02 03:26:04.810580 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:04.810348    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="839528ca-8c68-42e2-9a79-a926450a021d" containerName="payload-processing"
Nov 02 03:26:04.810580 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:04.810379    4496 cpu_manager.go:414] "containerMap: removing container" podUID="010d8313-308d-4bce-a4fa-277ead1e6d63" containerName="payload-pre-processing"
Nov 02 03:26:04.810580 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:04.810385    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="010d8313-308d-4bce-a4fa-277ead1e6d63" containerName="payload-pre-processing"
Nov 02 03:26:04.810580 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:04.810392    4496 cpu_manager.go:414] "containerMap: removing container" podUID="8db02a90-7174-4696-90b4-5fc024addbc1" containerName="payload-pre-processing"
Nov 02 03:26:04.810580 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:04.810397    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8db02a90-7174-4696-90b4-5fc024addbc1" containerName="payload-pre-processing"
Nov 02 03:26:04.810580 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:04.810410    4496 cpu_manager.go:414] "containerMap: removing container" podUID="ed8d5844-0ec4-4050-959c-f8864624ebd5" containerName="payload-pre-processing"
Nov 02 03:26:04.810580 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:04.810419    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed8d5844-0ec4-4050-959c-f8864624ebd5" containerName="payload-pre-processing"
Nov 02 03:26:04.810580 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:04.810428    4496 cpu_manager.go:414] "containerMap: removing container" podUID="ba6ad9c9-ab6b-43cd-bd4b-c7042792d7f3" containerName="payload-processing"
Nov 02 03:26:04.810580 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:04.810434    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba6ad9c9-ab6b-43cd-bd4b-c7042792d7f3" containerName="payload-processing"
Nov 02 03:26:04.810580 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:04.810440    4496 cpu_manager.go:414] "containerMap: removing container" podUID="d73da420-2bd5-41a2-b24b-e8601948fbd0" containerName="revoke-keys"
Nov 02 03:26:04.810580 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:04.810444    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d73da420-2bd5-41a2-b24b-e8601948fbd0" containerName="revoke-keys"
Nov 02 03:26:04.811312 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:04.810596    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="010d8313-308d-4bce-a4fa-277ead1e6d63" containerName="payload-pre-processing"
Nov 02 03:26:04.811312 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:04.810613    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="8db02a90-7174-4696-90b4-5fc024addbc1" containerName="payload-pre-processing"
Nov 02 03:26:04.811312 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:04.810624    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed8d5844-0ec4-4050-959c-f8864624ebd5" containerName="payload-pre-processing"
Nov 02 03:26:04.811312 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:04.810635    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba6ad9c9-ab6b-43cd-bd4b-c7042792d7f3" containerName="payload-processing"
Nov 02 03:26:04.811312 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:04.810645    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="d73da420-2bd5-41a2-b24b-e8601948fbd0" containerName="revoke-keys"
Nov 02 03:26:04.811312 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:04.810656    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="25ab21e1-7f9b-4017-86bd-b3e45bb4d823" containerName="payload-processing"
Nov 02 03:26:04.811312 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:04.810675    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="839528ca-8c68-42e2-9a79-a926450a021d" containerName="payload-processing"
Nov 02 03:26:04.818899 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:04.818871    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c9d1fc-wcp6v"
Nov 02 03:26:04.865460 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:04.865423    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c9d1fc-wcp6v"]
Nov 02 03:26:04.908709 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:04.908679    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 03:26:04.908709 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:04.908680    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 03:26:04.908914 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:04.908688    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 03:26:04.923181 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:04.923159    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9z5bk\""
Nov 02 03:26:04.934758 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:04.934728    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdvlp\" (UniqueName: \"kubernetes.io/projected/b6aa56d2-87d0-4ba8-8b82-069d2d1d2acc-kube-api-access-pdvlp\") pod \"maas-api-revoke-keys-e2e-api-a-c9d1fc-wcp6v\" (UID: \"b6aa56d2-87d0-4ba8-8b82-069d2d1d2acc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c9d1fc-wcp6v"
Nov 02 03:26:04.934889 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:04.934774    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6aa56d2-87d0-4ba8-8b82-069d2d1d2acc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c9d1fc-wcp6v\" (UID: \"b6aa56d2-87d0-4ba8-8b82-069d2d1d2acc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c9d1fc-wcp6v"
Nov 02 03:26:04.934889 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:04.934853    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6aa56d2-87d0-4ba8-8b82-069d2d1d2acc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c9d1fc-wcp6v\" (UID: \"b6aa56d2-87d0-4ba8-8b82-069d2d1d2acc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c9d1fc-wcp6v"
Nov 02 03:26:05.036417 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:05.036385    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdvlp\" (UniqueName: \"kubernetes.io/projected/b6aa56d2-87d0-4ba8-8b82-069d2d1d2acc-kube-api-access-pdvlp\") pod \"maas-api-revoke-keys-e2e-api-a-c9d1fc-wcp6v\" (UID: \"b6aa56d2-87d0-4ba8-8b82-069d2d1d2acc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c9d1fc-wcp6v"
Nov 02 03:26:05.036566 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:05.036429    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6aa56d2-87d0-4ba8-8b82-069d2d1d2acc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c9d1fc-wcp6v\" (UID: \"b6aa56d2-87d0-4ba8-8b82-069d2d1d2acc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c9d1fc-wcp6v"
Nov 02 03:26:05.036566 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:05.036470    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6aa56d2-87d0-4ba8-8b82-069d2d1d2acc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c9d1fc-wcp6v\" (UID: \"b6aa56d2-87d0-4ba8-8b82-069d2d1d2acc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c9d1fc-wcp6v"
Nov 02 03:26:05.037271 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:05.037243    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6aa56d2-87d0-4ba8-8b82-069d2d1d2acc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c9d1fc-wcp6v\" (UID: \"b6aa56d2-87d0-4ba8-8b82-069d2d1d2acc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c9d1fc-wcp6v"
Nov 02 03:26:05.038790 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:05.038770    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6aa56d2-87d0-4ba8-8b82-069d2d1d2acc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c9d1fc-wcp6v\" (UID: \"b6aa56d2-87d0-4ba8-8b82-069d2d1d2acc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c9d1fc-wcp6v"
Nov 02 03:26:05.093875 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:05.093818    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdvlp\" (UniqueName: \"kubernetes.io/projected/b6aa56d2-87d0-4ba8-8b82-069d2d1d2acc-kube-api-access-pdvlp\") pod \"maas-api-revoke-keys-e2e-api-a-c9d1fc-wcp6v\" (UID: \"b6aa56d2-87d0-4ba8-8b82-069d2d1d2acc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c9d1fc-wcp6v"
Nov 02 03:26:05.148516 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:05.148488    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c9d1fc-wcp6v"
Nov 02 03:26:05.531462 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:26:05.531429    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6aa56d2_87d0_4ba8_8b82_069d2d1d2acc.slice/crio-ba60ffd0be8bb4f205a4e52bd5b4b5d3520925aa4bbf0064d5b731e60f7e8dee WatchSource:0}: Error finding container ba60ffd0be8bb4f205a4e52bd5b4b5d3520925aa4bbf0064d5b731e60f7e8dee: Status 404 returned error can't find the container with id ba60ffd0be8bb4f205a4e52bd5b4b5d3520925aa4bbf0064d5b731e60f7e8dee
Nov 02 03:26:05.532593 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:05.532564    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c9d1fc-wcp6v"]
Nov 02 03:26:05.829497 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:05.829402    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c9d1fc-wcp6v" event={"ID":"b6aa56d2-87d0-4ba8-8b82-069d2d1d2acc","Type":"ContainerStarted","Data":"ba60ffd0be8bb4f205a4e52bd5b4b5d3520925aa4bbf0064d5b731e60f7e8dee"}
Nov 02 03:26:06.842003 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:06.841968    4496 generic.go:356] "Generic (PLEG): container finished" podID="b6aa56d2-87d0-4ba8-8b82-069d2d1d2acc" containerID="84057766c252316ad4078d71c351e0dc98e136fca66cf5887088b9b5ef328009" exitCode=0
Nov 02 03:26:06.842445 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:06.842070    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c9d1fc-wcp6v" event={"ID":"b6aa56d2-87d0-4ba8-8b82-069d2d1d2acc","Type":"ContainerDied","Data":"84057766c252316ad4078d71c351e0dc98e136fca66cf5887088b9b5ef328009"}
Nov 02 03:26:08.020155 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:08.020123    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c9d1fc-wcp6v"
Nov 02 03:26:08.182062 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:08.181967    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6aa56d2-87d0-4ba8-8b82-069d2d1d2acc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6aa56d2-87d0-4ba8-8b82-069d2d1d2acc-maas-api-openshift-ca\") pod \"b6aa56d2-87d0-4ba8-8b82-069d2d1d2acc\" (UID: \"b6aa56d2-87d0-4ba8-8b82-069d2d1d2acc\") "
Nov 02 03:26:08.182062 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:08.182046    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6aa56d2-87d0-4ba8-8b82-069d2d1d2acc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6aa56d2-87d0-4ba8-8b82-069d2d1d2acc-maas-api-xks-ca\") pod \"b6aa56d2-87d0-4ba8-8b82-069d2d1d2acc\" (UID: \"b6aa56d2-87d0-4ba8-8b82-069d2d1d2acc\") "
Nov 02 03:26:08.182234 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:08.182135    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6aa56d2-87d0-4ba8-8b82-069d2d1d2acc-kube-api-access-pdvlp\" (UniqueName: \"kubernetes.io/projected/b6aa56d2-87d0-4ba8-8b82-069d2d1d2acc-kube-api-access-pdvlp\") pod \"b6aa56d2-87d0-4ba8-8b82-069d2d1d2acc\" (UID: \"b6aa56d2-87d0-4ba8-8b82-069d2d1d2acc\") "
Nov 02 03:26:08.182445 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:08.182416    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6aa56d2-87d0-4ba8-8b82-069d2d1d2acc-maas-api-openshift-ca" pod "b6aa56d2-87d0-4ba8-8b82-069d2d1d2acc" (UID: "b6aa56d2-87d0-4ba8-8b82-069d2d1d2acc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:26:08.182627 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:08.182607    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6aa56d2-87d0-4ba8-8b82-069d2d1d2acc-maas-api-openshift-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:26:08.184239 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:08.184215    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6aa56d2-87d0-4ba8-8b82-069d2d1d2acc-maas-api-xks-ca" pod "b6aa56d2-87d0-4ba8-8b82-069d2d1d2acc" (UID: "b6aa56d2-87d0-4ba8-8b82-069d2d1d2acc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 03:26:08.184337 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:08.184298    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6aa56d2-87d0-4ba8-8b82-069d2d1d2acc-kube-api-access-pdvlp" pod "b6aa56d2-87d0-4ba8-8b82-069d2d1d2acc" (UID: "b6aa56d2-87d0-4ba8-8b82-069d2d1d2acc"). InnerVolumeSpecName "kube-api-access-pdvlp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:26:08.284486 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:08.284445    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6aa56d2-87d0-4ba8-8b82-069d2d1d2acc-maas-api-xks-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:26:08.284486 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:08.284485    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdvlp\" (UniqueName: \"kubernetes.io/projected/b6aa56d2-87d0-4ba8-8b82-069d2d1d2acc-kube-api-access-pdvlp\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:26:08.862594 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:08.862560    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c9d1fc-wcp6v"
Nov 02 03:26:08.862778 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:08.862593    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c9d1fc-wcp6v" event={"ID":"b6aa56d2-87d0-4ba8-8b82-069d2d1d2acc","Type":"ContainerDied","Data":"ba60ffd0be8bb4f205a4e52bd5b4b5d3520925aa4bbf0064d5b731e60f7e8dee"}
Nov 02 03:26:08.862778 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:08.862655    4496 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba60ffd0be8bb4f205a4e52bd5b4b5d3520925aa4bbf0064d5b731e60f7e8dee"
Nov 02 03:26:16.234145 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:16.234116    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c9d1fc-ffdc6cb67-6nwgq"]
Nov 02 03:26:16.345794 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:16.345762    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c9d1fc-5b9c6db48c-lkdss"]
Nov 02 03:26:20.961303 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:20.961262    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d5dcca72-nw5jm"]
Nov 02 03:26:21.011311 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:21.011275    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d5dcca72-nw5jm"]
Nov 02 03:26:21.234727 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:21.234650    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-c9d1fc-ffdc6cb67-6nwgq" podUID="2371a226-dcd2-49f3-a966-41d7da7cafa8" containerName="payload-processing" containerID="cri-o://e34ac0a1cacb1d594227b40f5c1dcf4a8d9a5d7a98e5ed186306c3d0b68c8505" gracePeriod=25
Nov 02 03:26:21.347060 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:21.346958    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c9d1fc-5b9c6db48c-lkdss" podUID="2d399530-92b8-40ea-b7d0-05e30081f5f2" containerName="payload-pre-processing" containerID="cri-o://fbd0fe60d51e45bdaac91bbfe255f5da59b7917e4b971291493616ce74527777" gracePeriod=25
Nov 02 03:26:21.564446 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:21.564419    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c9d1fc-ffdc6cb67-6nwgq"
Nov 02 03:26:21.594323 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:21.594284    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42e1a6eb-7b6c-429a-b34e-f4a8f78e5981" path="/var/lib/kubelet/pods/42e1a6eb-7b6c-429a-b34e-f4a8f78e5981/volumes"
Nov 02 03:26:21.700371 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:21.700342    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c9d1fc-5b9c6db48c-lkdss"
Nov 02 03:26:21.711912 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:21.711884    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2371a226-dcd2-49f3-a966-41d7da7cafa8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2371a226-dcd2-49f3-a966-41d7da7cafa8-plugins-config-volume\") pod \"2371a226-dcd2-49f3-a966-41d7da7cafa8\" (UID: \"2371a226-dcd2-49f3-a966-41d7da7cafa8\") "
Nov 02 03:26:21.712076 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:21.712005    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2371a226-dcd2-49f3-a966-41d7da7cafa8-kube-api-access-6spbm\" (UniqueName: \"kubernetes.io/projected/2371a226-dcd2-49f3-a966-41d7da7cafa8-kube-api-access-6spbm\") pod \"2371a226-dcd2-49f3-a966-41d7da7cafa8\" (UID: \"2371a226-dcd2-49f3-a966-41d7da7cafa8\") "
Nov 02 03:26:21.712260 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:21.712238    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2371a226-dcd2-49f3-a966-41d7da7cafa8-plugins-config-volume" pod "2371a226-dcd2-49f3-a966-41d7da7cafa8" (UID: "2371a226-dcd2-49f3-a966-41d7da7cafa8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:26:21.712683 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:21.712617    4496 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2371a226-dcd2-49f3-a966-41d7da7cafa8-plugins-config-volume\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:26:21.714170 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:21.714145    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2371a226-dcd2-49f3-a966-41d7da7cafa8-kube-api-access-6spbm" pod "2371a226-dcd2-49f3-a966-41d7da7cafa8" (UID: "2371a226-dcd2-49f3-a966-41d7da7cafa8"). InnerVolumeSpecName "kube-api-access-6spbm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:26:21.813436 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:21.813350    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d399530-92b8-40ea-b7d0-05e30081f5f2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d399530-92b8-40ea-b7d0-05e30081f5f2-plugins-config-volume\") pod \"2d399530-92b8-40ea-b7d0-05e30081f5f2\" (UID: \"2d399530-92b8-40ea-b7d0-05e30081f5f2\") "
Nov 02 03:26:21.813436 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:21.813426    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d399530-92b8-40ea-b7d0-05e30081f5f2-kube-api-access-9rw8k\" (UniqueName: \"kubernetes.io/projected/2d399530-92b8-40ea-b7d0-05e30081f5f2-kube-api-access-9rw8k\") pod \"2d399530-92b8-40ea-b7d0-05e30081f5f2\" (UID: \"2d399530-92b8-40ea-b7d0-05e30081f5f2\") "
Nov 02 03:26:21.813730 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:21.813708    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d399530-92b8-40ea-b7d0-05e30081f5f2-plugins-config-volume" pod "2d399530-92b8-40ea-b7d0-05e30081f5f2" (UID: "2d399530-92b8-40ea-b7d0-05e30081f5f2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:26:21.813797 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:21.813783    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6spbm\" (UniqueName: \"kubernetes.io/projected/2371a226-dcd2-49f3-a966-41d7da7cafa8-kube-api-access-6spbm\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:26:21.816091 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:21.816061    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d399530-92b8-40ea-b7d0-05e30081f5f2-kube-api-access-9rw8k" pod "2d399530-92b8-40ea-b7d0-05e30081f5f2" (UID: "2d399530-92b8-40ea-b7d0-05e30081f5f2"). InnerVolumeSpecName "kube-api-access-9rw8k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:26:21.914960 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:21.914926    4496 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d399530-92b8-40ea-b7d0-05e30081f5f2-plugins-config-volume\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:26:21.914960 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:21.914956    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rw8k\" (UniqueName: \"kubernetes.io/projected/2d399530-92b8-40ea-b7d0-05e30081f5f2-kube-api-access-9rw8k\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:26:21.993791 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:21.993756    4496 generic.go:356] "Generic (PLEG): container finished" podID="2371a226-dcd2-49f3-a966-41d7da7cafa8" containerID="e34ac0a1cacb1d594227b40f5c1dcf4a8d9a5d7a98e5ed186306c3d0b68c8505" exitCode=0
Nov 02 03:26:21.994289 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:21.993815    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c9d1fc-ffdc6cb67-6nwgq"
Nov 02 03:26:21.994289 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:21.993838    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c9d1fc-ffdc6cb67-6nwgq" event={"ID":"2371a226-dcd2-49f3-a966-41d7da7cafa8","Type":"ContainerDied","Data":"e34ac0a1cacb1d594227b40f5c1dcf4a8d9a5d7a98e5ed186306c3d0b68c8505"}
Nov 02 03:26:21.994289 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:21.993902    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c9d1fc-ffdc6cb67-6nwgq" event={"ID":"2371a226-dcd2-49f3-a966-41d7da7cafa8","Type":"ContainerDied","Data":"a6173ee9223410d462d1e031fd11ce3b543bf94658eaa06d9b54aaa945bdf6d4"}
Nov 02 03:26:21.994289 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:21.993924    4496 scope.go:122] "RemoveContainer" containerID="e34ac0a1cacb1d594227b40f5c1dcf4a8d9a5d7a98e5ed186306c3d0b68c8505"
Nov 02 03:26:21.995309 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:21.995291    4496 generic.go:356] "Generic (PLEG): container finished" podID="2d399530-92b8-40ea-b7d0-05e30081f5f2" containerID="fbd0fe60d51e45bdaac91bbfe255f5da59b7917e4b971291493616ce74527777" exitCode=0
Nov 02 03:26:21.995409 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:21.995317    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c9d1fc-5b9c6db48c-lkdss" event={"ID":"2d399530-92b8-40ea-b7d0-05e30081f5f2","Type":"ContainerDied","Data":"fbd0fe60d51e45bdaac91bbfe255f5da59b7917e4b971291493616ce74527777"}
Nov 02 03:26:21.995409 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:21.995339    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c9d1fc-5b9c6db48c-lkdss"
Nov 02 03:26:21.995409 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:21.995369    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c9d1fc-5b9c6db48c-lkdss" event={"ID":"2d399530-92b8-40ea-b7d0-05e30081f5f2","Type":"ContainerDied","Data":"9a185dfab07ec381347648b0326f1d6e1496185abe0f681243f9f61abc383ac8"}
Nov 02 03:26:22.006360 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:22.006344    4496 scope.go:122] "RemoveContainer" containerID="e34ac0a1cacb1d594227b40f5c1dcf4a8d9a5d7a98e5ed186306c3d0b68c8505"
Nov 02 03:26:22.006623 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:26:22.006600    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e34ac0a1cacb1d594227b40f5c1dcf4a8d9a5d7a98e5ed186306c3d0b68c8505\": container with ID starting with e34ac0a1cacb1d594227b40f5c1dcf4a8d9a5d7a98e5ed186306c3d0b68c8505 not found: ID does not exist" containerID="e34ac0a1cacb1d594227b40f5c1dcf4a8d9a5d7a98e5ed186306c3d0b68c8505"
Nov 02 03:26:22.006694 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:22.006634    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e34ac0a1cacb1d594227b40f5c1dcf4a8d9a5d7a98e5ed186306c3d0b68c8505"} err="failed to get container status \"e34ac0a1cacb1d594227b40f5c1dcf4a8d9a5d7a98e5ed186306c3d0b68c8505\": rpc error: code = NotFound desc = could not find container \"e34ac0a1cacb1d594227b40f5c1dcf4a8d9a5d7a98e5ed186306c3d0b68c8505\": container with ID starting with e34ac0a1cacb1d594227b40f5c1dcf4a8d9a5d7a98e5ed186306c3d0b68c8505 not found: ID does not exist"
Nov 02 03:26:22.006694 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:22.006660    4496 scope.go:122] "RemoveContainer" containerID="fbd0fe60d51e45bdaac91bbfe255f5da59b7917e4b971291493616ce74527777"
Nov 02 03:26:22.015953 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:22.015930    4496 scope.go:122] "RemoveContainer" containerID="fbd0fe60d51e45bdaac91bbfe255f5da59b7917e4b971291493616ce74527777"
Nov 02 03:26:22.016232 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:26:22.016212    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbd0fe60d51e45bdaac91bbfe255f5da59b7917e4b971291493616ce74527777\": container with ID starting with fbd0fe60d51e45bdaac91bbfe255f5da59b7917e4b971291493616ce74527777 not found: ID does not exist" containerID="fbd0fe60d51e45bdaac91bbfe255f5da59b7917e4b971291493616ce74527777"
Nov 02 03:26:22.016289 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:22.016237    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbd0fe60d51e45bdaac91bbfe255f5da59b7917e4b971291493616ce74527777"} err="failed to get container status \"fbd0fe60d51e45bdaac91bbfe255f5da59b7917e4b971291493616ce74527777\": rpc error: code = NotFound desc = could not find container \"fbd0fe60d51e45bdaac91bbfe255f5da59b7917e4b971291493616ce74527777\": container with ID starting with fbd0fe60d51e45bdaac91bbfe255f5da59b7917e4b971291493616ce74527777 not found: ID does not exist"
Nov 02 03:26:22.162189 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:22.162088    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c9d1fc-ffdc6cb67-6nwgq"]
Nov 02 03:26:22.166001 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:22.165973    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c9d1fc-ffdc6cb67-6nwgq"]
Nov 02 03:26:22.273657 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:22.273622    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c9d1fc-5b9c6db48c-lkdss"]
Nov 02 03:26:22.277658 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:22.277630    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c9d1fc-5b9c6db48c-lkdss"]
Nov 02 03:26:23.577351 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:23.577314    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2371a226-dcd2-49f3-a966-41d7da7cafa8" path="/var/lib/kubelet/pods/2371a226-dcd2-49f3-a966-41d7da7cafa8/volumes"
Nov 02 03:26:23.577948 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:23.577925    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d399530-92b8-40ea-b7d0-05e30081f5f2" path="/var/lib/kubelet/pods/2d399530-92b8-40ea-b7d0-05e30081f5f2/volumes"
Nov 02 03:26:27.172855 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:27.172813    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b4a230-8w4qw"]
Nov 02 03:26:27.222920 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:27.222871    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b4a230-8w4qw"]
Nov 02 03:26:27.574795 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:27.574759    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aee8a3f1-6405-467c-8b98-84ca26618ffb" path="/var/lib/kubelet/pods/aee8a3f1-6405-467c-8b98-84ca26618ffb/volumes"
Nov 02 03:26:33.549062 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:26:33.548995    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 03:26:34.366661 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:34.366618    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a27da5-wsjdx"]
Nov 02 03:26:34.367372 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:34.367355    4496 cpu_manager.go:414] "containerMap: removing container" podUID="b6aa56d2-87d0-4ba8-8b82-069d2d1d2acc" containerName="revoke-keys"
Nov 02 03:26:34.367430 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:34.367373    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6aa56d2-87d0-4ba8-8b82-069d2d1d2acc" containerName="revoke-keys"
Nov 02 03:26:34.367430 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:34.367383    4496 cpu_manager.go:414] "containerMap: removing container" podUID="2371a226-dcd2-49f3-a966-41d7da7cafa8" containerName="payload-processing"
Nov 02 03:26:34.367430 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:34.367388    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2371a226-dcd2-49f3-a966-41d7da7cafa8" containerName="payload-processing"
Nov 02 03:26:34.367430 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:34.367414    4496 cpu_manager.go:414] "containerMap: removing container" podUID="2d399530-92b8-40ea-b7d0-05e30081f5f2" containerName="payload-pre-processing"
Nov 02 03:26:34.367430 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:34.367418    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d399530-92b8-40ea-b7d0-05e30081f5f2" containerName="payload-pre-processing"
Nov 02 03:26:34.367584 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:34.367521    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d399530-92b8-40ea-b7d0-05e30081f5f2" containerName="payload-pre-processing"
Nov 02 03:26:34.367584 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:34.367537    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6aa56d2-87d0-4ba8-8b82-069d2d1d2acc" containerName="revoke-keys"
Nov 02 03:26:34.367584 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:34.367543    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="2371a226-dcd2-49f3-a966-41d7da7cafa8" containerName="payload-processing"
Nov 02 03:26:34.372663 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:34.372644    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a27da5-wsjdx"
Nov 02 03:26:34.431288 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:34.431256    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a27da5-wsjdx"]
Nov 02 03:26:34.449374 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:34.449338    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7c2d7\" (UniqueName: \"kubernetes.io/projected/a62ebf0f-f2fd-404d-aa26-6f57836432dd-kube-api-access-7c2d7\") pod \"maas-api-revoke-keys-e2e-api-b-a27da5-wsjdx\" (UID: \"a62ebf0f-f2fd-404d-aa26-6f57836432dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a27da5-wsjdx"
Nov 02 03:26:34.449560 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:34.449474    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a62ebf0f-f2fd-404d-aa26-6f57836432dd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a27da5-wsjdx\" (UID: \"a62ebf0f-f2fd-404d-aa26-6f57836432dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a27da5-wsjdx"
Nov 02 03:26:34.449637 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:34.449592    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a62ebf0f-f2fd-404d-aa26-6f57836432dd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a27da5-wsjdx\" (UID: \"a62ebf0f-f2fd-404d-aa26-6f57836432dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a27da5-wsjdx"
Nov 02 03:26:34.477046 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:34.476975    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 03:26:34.477208 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:34.477079    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9z5bk\""
Nov 02 03:26:34.527787 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:34.527753    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 03:26:34.527787 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:34.527783    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 03:26:34.550565 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:34.550539    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7c2d7\" (UniqueName: \"kubernetes.io/projected/a62ebf0f-f2fd-404d-aa26-6f57836432dd-kube-api-access-7c2d7\") pod \"maas-api-revoke-keys-e2e-api-b-a27da5-wsjdx\" (UID: \"a62ebf0f-f2fd-404d-aa26-6f57836432dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a27da5-wsjdx"
Nov 02 03:26:34.551000 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:34.550628    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a62ebf0f-f2fd-404d-aa26-6f57836432dd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a27da5-wsjdx\" (UID: \"a62ebf0f-f2fd-404d-aa26-6f57836432dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a27da5-wsjdx"
Nov 02 03:26:34.551000 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:34.550749    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a62ebf0f-f2fd-404d-aa26-6f57836432dd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a27da5-wsjdx\" (UID: \"a62ebf0f-f2fd-404d-aa26-6f57836432dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a27da5-wsjdx"
Nov 02 03:26:34.551416 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:34.551393    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a62ebf0f-f2fd-404d-aa26-6f57836432dd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a27da5-wsjdx\" (UID: \"a62ebf0f-f2fd-404d-aa26-6f57836432dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a27da5-wsjdx"
Nov 02 03:26:34.553124 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:34.553103    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a62ebf0f-f2fd-404d-aa26-6f57836432dd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a27da5-wsjdx\" (UID: \"a62ebf0f-f2fd-404d-aa26-6f57836432dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a27da5-wsjdx"
Nov 02 03:26:34.608502 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:34.608480    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7c2d7\" (UniqueName: \"kubernetes.io/projected/a62ebf0f-f2fd-404d-aa26-6f57836432dd-kube-api-access-7c2d7\") pod \"maas-api-revoke-keys-e2e-api-b-a27da5-wsjdx\" (UID: \"a62ebf0f-f2fd-404d-aa26-6f57836432dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a27da5-wsjdx"
Nov 02 03:26:34.688994 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:34.688921    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a27da5-wsjdx"
Nov 02 03:26:35.069792 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:26:35.069752    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda62ebf0f_f2fd_404d_aa26_6f57836432dd.slice/crio-d7fb4bcf305dbaf5de684a18a0bd9af8321caa0374328e9e702420d1e42ea4d6 WatchSource:0}: Error finding container d7fb4bcf305dbaf5de684a18a0bd9af8321caa0374328e9e702420d1e42ea4d6: Status 404 returned error can't find the container with id d7fb4bcf305dbaf5de684a18a0bd9af8321caa0374328e9e702420d1e42ea4d6
Nov 02 03:26:35.070855 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:35.070808    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a27da5-wsjdx"]
Nov 02 03:26:35.182577 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:35.182543    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a27da5-wsjdx" event={"ID":"a62ebf0f-f2fd-404d-aa26-6f57836432dd","Type":"ContainerStarted","Data":"d7fb4bcf305dbaf5de684a18a0bd9af8321caa0374328e9e702420d1e42ea4d6"}
Nov 02 03:26:36.191917 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:36.191886    4496 generic.go:356] "Generic (PLEG): container finished" podID="a62ebf0f-f2fd-404d-aa26-6f57836432dd" containerID="969edc3ab36084d081a94c359c09108acb2507149716b733ba329d3da5607e53" exitCode=0
Nov 02 03:26:36.192287 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:36.191937    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a27da5-wsjdx" event={"ID":"a62ebf0f-f2fd-404d-aa26-6f57836432dd","Type":"ContainerDied","Data":"969edc3ab36084d081a94c359c09108acb2507149716b733ba329d3da5607e53"}
Nov 02 03:26:37.371612 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:37.371587    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a27da5-wsjdx"
Nov 02 03:26:37.494035 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:37.493992    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a62ebf0f-f2fd-404d-aa26-6f57836432dd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a62ebf0f-f2fd-404d-aa26-6f57836432dd-maas-api-xks-ca\") pod \"a62ebf0f-f2fd-404d-aa26-6f57836432dd\" (UID: \"a62ebf0f-f2fd-404d-aa26-6f57836432dd\") "
Nov 02 03:26:37.494205 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:37.494107    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a62ebf0f-f2fd-404d-aa26-6f57836432dd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a62ebf0f-f2fd-404d-aa26-6f57836432dd-maas-api-openshift-ca\") pod \"a62ebf0f-f2fd-404d-aa26-6f57836432dd\" (UID: \"a62ebf0f-f2fd-404d-aa26-6f57836432dd\") "
Nov 02 03:26:37.494381 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:37.494360    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a62ebf0f-f2fd-404d-aa26-6f57836432dd-kube-api-access-7c2d7\" (UniqueName: \"kubernetes.io/projected/a62ebf0f-f2fd-404d-aa26-6f57836432dd-kube-api-access-7c2d7\") pod \"a62ebf0f-f2fd-404d-aa26-6f57836432dd\" (UID: \"a62ebf0f-f2fd-404d-aa26-6f57836432dd\") "
Nov 02 03:26:37.494507 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:37.494489    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a62ebf0f-f2fd-404d-aa26-6f57836432dd-maas-api-openshift-ca" pod "a62ebf0f-f2fd-404d-aa26-6f57836432dd" (UID: "a62ebf0f-f2fd-404d-aa26-6f57836432dd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:26:37.494878 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:37.494857    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a62ebf0f-f2fd-404d-aa26-6f57836432dd-maas-api-openshift-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:26:37.496172 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:37.496141    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a62ebf0f-f2fd-404d-aa26-6f57836432dd-maas-api-xks-ca" pod "a62ebf0f-f2fd-404d-aa26-6f57836432dd" (UID: "a62ebf0f-f2fd-404d-aa26-6f57836432dd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 03:26:37.496316 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:37.496291    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a62ebf0f-f2fd-404d-aa26-6f57836432dd-kube-api-access-7c2d7" pod "a62ebf0f-f2fd-404d-aa26-6f57836432dd" (UID: "a62ebf0f-f2fd-404d-aa26-6f57836432dd"). InnerVolumeSpecName "kube-api-access-7c2d7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:26:37.595990 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:37.595959    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7c2d7\" (UniqueName: \"kubernetes.io/projected/a62ebf0f-f2fd-404d-aa26-6f57836432dd-kube-api-access-7c2d7\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:26:37.595990 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:37.595991    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a62ebf0f-f2fd-404d-aa26-6f57836432dd-maas-api-xks-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:26:38.209225 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:38.209197    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a27da5-wsjdx"
Nov 02 03:26:38.209407 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:38.209239    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a27da5-wsjdx" event={"ID":"a62ebf0f-f2fd-404d-aa26-6f57836432dd","Type":"ContainerDied","Data":"d7fb4bcf305dbaf5de684a18a0bd9af8321caa0374328e9e702420d1e42ea4d6"}
Nov 02 03:26:38.209407 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:38.209295    4496 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d7fb4bcf305dbaf5de684a18a0bd9af8321caa0374328e9e702420d1e42ea4d6"
Nov 02 03:26:40.954956 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:40.954916    4496 scope.go:122] "RemoveContainer" containerID="ad3a7ba90784f96b6884f905ba1cfec591dea359924d401d0f24f418d26166ab"
Nov 02 03:26:45.856435 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:45.856399    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a27da5-74bf9f44b5-xdcnr"]
Nov 02 03:26:45.968677 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:45.968638    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a27da5-85ccf9bfcf-mhsrh"]
Nov 02 03:26:50.857619 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:50.857573    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-a27da5-74bf9f44b5-xdcnr" podUID="020a5526-3ffd-46fb-8015-63d0e7657d37" containerName="payload-processing" containerID="cri-o://9a3285a39f385b7c7bc1a1c821480f143057d25bf2945a9b4c61893030924917" gracePeriod=25
Nov 02 03:26:50.970111 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:50.969999    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a27da5-85ccf9bfcf-mhsrh" podUID="b8658350-dded-408d-9904-40b3403119c9" containerName="payload-pre-processing" containerID="cri-o://75b08842f147c8834a2c225eea9932ce212eb052a8594129606df98086aee582" gracePeriod=25
Nov 02 03:26:51.172239 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:51.172208    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a27da5-74bf9f44b5-xdcnr"
Nov 02 03:26:51.234414 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:51.234368    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/020a5526-3ffd-46fb-8015-63d0e7657d37-kube-api-access-42ccr\" (UniqueName: \"kubernetes.io/projected/020a5526-3ffd-46fb-8015-63d0e7657d37-kube-api-access-42ccr\") pod \"020a5526-3ffd-46fb-8015-63d0e7657d37\" (UID: \"020a5526-3ffd-46fb-8015-63d0e7657d37\") "
Nov 02 03:26:51.234729 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:51.234707    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/020a5526-3ffd-46fb-8015-63d0e7657d37-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/020a5526-3ffd-46fb-8015-63d0e7657d37-plugins-config-volume\") pod \"020a5526-3ffd-46fb-8015-63d0e7657d37\" (UID: \"020a5526-3ffd-46fb-8015-63d0e7657d37\") "
Nov 02 03:26:51.251044 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:51.250062    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/020a5526-3ffd-46fb-8015-63d0e7657d37-plugins-config-volume" pod "020a5526-3ffd-46fb-8015-63d0e7657d37" (UID: "020a5526-3ffd-46fb-8015-63d0e7657d37"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:26:51.273710 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:51.273677    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/020a5526-3ffd-46fb-8015-63d0e7657d37-kube-api-access-42ccr" pod "020a5526-3ffd-46fb-8015-63d0e7657d37" (UID: "020a5526-3ffd-46fb-8015-63d0e7657d37"). InnerVolumeSpecName "kube-api-access-42ccr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:26:51.302895 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:51.302868    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a27da5-85ccf9bfcf-mhsrh"
Nov 02 03:26:51.344740 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:51.344702    4496 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/020a5526-3ffd-46fb-8015-63d0e7657d37-plugins-config-volume\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:26:51.344740 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:51.344740    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42ccr\" (UniqueName: \"kubernetes.io/projected/020a5526-3ffd-46fb-8015-63d0e7657d37-kube-api-access-42ccr\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:26:51.423344 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:51.423259    4496 generic.go:356] "Generic (PLEG): container finished" podID="b8658350-dded-408d-9904-40b3403119c9" containerID="75b08842f147c8834a2c225eea9932ce212eb052a8594129606df98086aee582" exitCode=0
Nov 02 03:26:51.423344 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:51.423310    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a27da5-85ccf9bfcf-mhsrh"
Nov 02 03:26:51.423566 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:51.423336    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a27da5-85ccf9bfcf-mhsrh" event={"ID":"b8658350-dded-408d-9904-40b3403119c9","Type":"ContainerDied","Data":"75b08842f147c8834a2c225eea9932ce212eb052a8594129606df98086aee582"}
Nov 02 03:26:51.423566 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:51.423426    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a27da5-85ccf9bfcf-mhsrh" event={"ID":"b8658350-dded-408d-9904-40b3403119c9","Type":"ContainerDied","Data":"4e7395ca8e88191a5dcbb288305be219701a363d357083184727227051f12734"}
Nov 02 03:26:51.423566 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:51.423459    4496 scope.go:122] "RemoveContainer" containerID="75b08842f147c8834a2c225eea9932ce212eb052a8594129606df98086aee582"
Nov 02 03:26:51.425332 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:51.425312    4496 generic.go:356] "Generic (PLEG): container finished" podID="020a5526-3ffd-46fb-8015-63d0e7657d37" containerID="9a3285a39f385b7c7bc1a1c821480f143057d25bf2945a9b4c61893030924917" exitCode=0
Nov 02 03:26:51.425436 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:51.425374    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a27da5-74bf9f44b5-xdcnr"
Nov 02 03:26:51.425436 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:51.425403    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a27da5-74bf9f44b5-xdcnr" event={"ID":"020a5526-3ffd-46fb-8015-63d0e7657d37","Type":"ContainerDied","Data":"9a3285a39f385b7c7bc1a1c821480f143057d25bf2945a9b4c61893030924917"}
Nov 02 03:26:51.425552 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:51.425449    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a27da5-74bf9f44b5-xdcnr" event={"ID":"020a5526-3ffd-46fb-8015-63d0e7657d37","Type":"ContainerDied","Data":"99da226b3c60bac02f92822f16fd4e042292defbc0a482683915b3203dce6eff"}
Nov 02 03:26:51.439800 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:51.439783    4496 scope.go:122] "RemoveContainer" containerID="75b08842f147c8834a2c225eea9932ce212eb052a8594129606df98086aee582"
Nov 02 03:26:51.440172 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:26:51.440145    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75b08842f147c8834a2c225eea9932ce212eb052a8594129606df98086aee582\": container with ID starting with 75b08842f147c8834a2c225eea9932ce212eb052a8594129606df98086aee582 not found: ID does not exist" containerID="75b08842f147c8834a2c225eea9932ce212eb052a8594129606df98086aee582"
Nov 02 03:26:51.440298 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:51.440186    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75b08842f147c8834a2c225eea9932ce212eb052a8594129606df98086aee582"} err="failed to get container status \"75b08842f147c8834a2c225eea9932ce212eb052a8594129606df98086aee582\": rpc error: code = NotFound desc = could not find container \"75b08842f147c8834a2c225eea9932ce212eb052a8594129606df98086aee582\": container with ID starting with 75b08842f147c8834a2c225eea9932ce212eb052a8594129606df98086aee582 not found: ID does not exist"
Nov 02 03:26:51.440298 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:51.440208    4496 scope.go:122] "RemoveContainer" containerID="9a3285a39f385b7c7bc1a1c821480f143057d25bf2945a9b4c61893030924917"
Nov 02 03:26:51.446160 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:51.446138    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8658350-dded-408d-9904-40b3403119c9-kube-api-access-rsrpd\" (UniqueName: \"kubernetes.io/projected/b8658350-dded-408d-9904-40b3403119c9-kube-api-access-rsrpd\") pod \"b8658350-dded-408d-9904-40b3403119c9\" (UID: \"b8658350-dded-408d-9904-40b3403119c9\") "
Nov 02 03:26:51.446419 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:51.446397    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8658350-dded-408d-9904-40b3403119c9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8658350-dded-408d-9904-40b3403119c9-plugins-config-volume\") pod \"b8658350-dded-408d-9904-40b3403119c9\" (UID: \"b8658350-dded-408d-9904-40b3403119c9\") "
Nov 02 03:26:51.447594 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:51.447570    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8658350-dded-408d-9904-40b3403119c9-plugins-config-volume" pod "b8658350-dded-408d-9904-40b3403119c9" (UID: "b8658350-dded-408d-9904-40b3403119c9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:26:51.450393 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:51.450360    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8658350-dded-408d-9904-40b3403119c9-kube-api-access-rsrpd" pod "b8658350-dded-408d-9904-40b3403119c9" (UID: "b8658350-dded-408d-9904-40b3403119c9"). InnerVolumeSpecName "kube-api-access-rsrpd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:26:51.452552 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:51.452535    4496 scope.go:122] "RemoveContainer" containerID="9a3285a39f385b7c7bc1a1c821480f143057d25bf2945a9b4c61893030924917"
Nov 02 03:26:51.452921 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:26:51.452900    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a3285a39f385b7c7bc1a1c821480f143057d25bf2945a9b4c61893030924917\": container with ID starting with 9a3285a39f385b7c7bc1a1c821480f143057d25bf2945a9b4c61893030924917 not found: ID does not exist" containerID="9a3285a39f385b7c7bc1a1c821480f143057d25bf2945a9b4c61893030924917"
Nov 02 03:26:51.453001 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:51.452929    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a3285a39f385b7c7bc1a1c821480f143057d25bf2945a9b4c61893030924917"} err="failed to get container status \"9a3285a39f385b7c7bc1a1c821480f143057d25bf2945a9b4c61893030924917\": rpc error: code = NotFound desc = could not find container \"9a3285a39f385b7c7bc1a1c821480f143057d25bf2945a9b4c61893030924917\": container with ID starting with 9a3285a39f385b7c7bc1a1c821480f143057d25bf2945a9b4c61893030924917 not found: ID does not exist"
Nov 02 03:26:51.548280 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:51.548250    4496 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8658350-dded-408d-9904-40b3403119c9-plugins-config-volume\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:26:51.548280 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:51.548276    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsrpd\" (UniqueName: \"kubernetes.io/projected/b8658350-dded-408d-9904-40b3403119c9-kube-api-access-rsrpd\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:26:51.593120 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:51.593092    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a27da5-74bf9f44b5-xdcnr"]
Nov 02 03:26:51.597447 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:51.597421    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a27da5-74bf9f44b5-xdcnr"]
Nov 02 03:26:51.843680 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:51.843647    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a27da5-85ccf9bfcf-mhsrh"]
Nov 02 03:26:51.847667 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:51.847638    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a27da5-85ccf9bfcf-mhsrh"]
Nov 02 03:26:53.576058 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:53.576002    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="020a5526-3ffd-46fb-8015-63d0e7657d37" path="/var/lib/kubelet/pods/020a5526-3ffd-46fb-8015-63d0e7657d37/volumes"
Nov 02 03:26:53.576561 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:53.576509    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8658350-dded-408d-9904-40b3403119c9" path="/var/lib/kubelet/pods/b8658350-dded-408d-9904-40b3403119c9/volumes"
Nov 02 03:26:55.073080 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:55.073051    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-x5gwb"]
Nov 02 03:26:55.073477 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:55.073278    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-x5gwb" podUID="938e8e17-80ba-4b93-9f96-29e68f73f927" containerName="istio-proxy" containerID="cri-o://843543e593513d43dd923e6a55ddc2f2bb1a95c51f7b86f5ab4ff1a6cae60e03" gracePeriod=30
Nov 02 03:26:56.265434 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:56.265398    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xstsp"]
Nov 02 03:26:56.265886 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:56.265634    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xstsp" podUID="0031b60a-f888-4c85-8999-07bddc61a513" containerName="istio-proxy" containerID="cri-o://a0fbd673106ccac658e925ad94554076f70660e4f737063374097d2828d37ad8" gracePeriod=30
Nov 02 03:26:56.639761 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:56.639682    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-277pb"]
Nov 02 03:26:56.639912 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:56.639886    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-277pb" podUID="3051f935-d87a-422e-9073-909b2c2768ac" containerName="istio-proxy" containerID="cri-o://f508d79f63179aee6899d89526a45e19a8f3725b8c9ed7be843528ac401a5a4c" gracePeriod=30
Nov 02 03:26:58.047071 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:58.047031    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef76bd9a-jkjjp"]
Nov 02 03:26:58.097659 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:58.097624    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef76bd9a-jkjjp"]
Nov 02 03:26:59.574645 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:26:59.574602    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4451fd8f-ab45-4c9b-8e91-c9b6f0a780fe" path="/var/lib/kubelet/pods/4451fd8f-ab45-4c9b-8e91-c9b6f0a780fe/volumes"
Nov 02 03:27:00.291919 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:00.291881    4496 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-277pb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 03:27:00.292106 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:00.291946    4496 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-277pb" podUID="3051f935-d87a-422e-9073-909b2c2768ac" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 03:27:00.344398 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:00.344372    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-x5gwb"
Nov 02 03:27:00.404330 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:00.404246    4496 patch_prober.go:29] interesting pod/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xstsp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 03:27:00.404330 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:00.404312    4496 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xstsp" podUID="0031b60a-f888-4c85-8999-07bddc61a513" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 03:27:00.470825 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:00.470788    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/938e8e17-80ba-4b93-9f96-29e68f73f927-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/938e8e17-80ba-4b93-9f96-29e68f73f927-istiod-ca-cert\") pod \"938e8e17-80ba-4b93-9f96-29e68f73f927\" (UID: \"938e8e17-80ba-4b93-9f96-29e68f73f927\") "
Nov 02 03:27:00.470995 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:00.470839    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/938e8e17-80ba-4b93-9f96-29e68f73f927-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/938e8e17-80ba-4b93-9f96-29e68f73f927-credential-socket\") pod \"938e8e17-80ba-4b93-9f96-29e68f73f927\" (UID: \"938e8e17-80ba-4b93-9f96-29e68f73f927\") "
Nov 02 03:27:00.470995 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:00.470872    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/938e8e17-80ba-4b93-9f96-29e68f73f927-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/938e8e17-80ba-4b93-9f96-29e68f73f927-istio-data\") pod \"938e8e17-80ba-4b93-9f96-29e68f73f927\" (UID: \"938e8e17-80ba-4b93-9f96-29e68f73f927\") "
Nov 02 03:27:00.470995 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:00.470933    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/938e8e17-80ba-4b93-9f96-29e68f73f927-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/938e8e17-80ba-4b93-9f96-29e68f73f927-istio-podinfo\") pod \"938e8e17-80ba-4b93-9f96-29e68f73f927\" (UID: \"938e8e17-80ba-4b93-9f96-29e68f73f927\") "
Nov 02 03:27:00.470995 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:00.470967    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/938e8e17-80ba-4b93-9f96-29e68f73f927-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/938e8e17-80ba-4b93-9f96-29e68f73f927-workload-certs\") pod \"938e8e17-80ba-4b93-9f96-29e68f73f927\" (UID: \"938e8e17-80ba-4b93-9f96-29e68f73f927\") "
Nov 02 03:27:00.471271 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:00.471004    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/938e8e17-80ba-4b93-9f96-29e68f73f927-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/938e8e17-80ba-4b93-9f96-29e68f73f927-istio-envoy\") pod \"938e8e17-80ba-4b93-9f96-29e68f73f927\" (UID: \"938e8e17-80ba-4b93-9f96-29e68f73f927\") "
Nov 02 03:27:00.471271 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:00.471055    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/938e8e17-80ba-4b93-9f96-29e68f73f927-kube-api-access-68b6p\" (UniqueName: \"kubernetes.io/projected/938e8e17-80ba-4b93-9f96-29e68f73f927-kube-api-access-68b6p\") pod \"938e8e17-80ba-4b93-9f96-29e68f73f927\" (UID: \"938e8e17-80ba-4b93-9f96-29e68f73f927\") "
Nov 02 03:27:00.471271 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:00.471093    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/938e8e17-80ba-4b93-9f96-29e68f73f927-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/938e8e17-80ba-4b93-9f96-29e68f73f927-workload-socket\") pod \"938e8e17-80ba-4b93-9f96-29e68f73f927\" (UID: \"938e8e17-80ba-4b93-9f96-29e68f73f927\") "
Nov 02 03:27:00.471271 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:00.471240    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/938e8e17-80ba-4b93-9f96-29e68f73f927-istio-token\" (UniqueName: \"kubernetes.io/projected/938e8e17-80ba-4b93-9f96-29e68f73f927-istio-token\") pod \"938e8e17-80ba-4b93-9f96-29e68f73f927\" (UID: \"938e8e17-80ba-4b93-9f96-29e68f73f927\") "
Nov 02 03:27:00.472575 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:00.472543    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/938e8e17-80ba-4b93-9f96-29e68f73f927-workload-certs" pod "938e8e17-80ba-4b93-9f96-29e68f73f927" (UID: "938e8e17-80ba-4b93-9f96-29e68f73f927"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:27:00.473730 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:00.473436    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/938e8e17-80ba-4b93-9f96-29e68f73f927-credential-socket" pod "938e8e17-80ba-4b93-9f96-29e68f73f927" (UID: "938e8e17-80ba-4b93-9f96-29e68f73f927"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:27:00.473730 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:00.473471    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/938e8e17-80ba-4b93-9f96-29e68f73f927-istio-data" pod "938e8e17-80ba-4b93-9f96-29e68f73f927" (UID: "938e8e17-80ba-4b93-9f96-29e68f73f927"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:27:00.473898 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:00.473808    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/938e8e17-80ba-4b93-9f96-29e68f73f927-istiod-ca-cert" pod "938e8e17-80ba-4b93-9f96-29e68f73f927" (UID: "938e8e17-80ba-4b93-9f96-29e68f73f927"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:27:00.474058 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:00.473996    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/938e8e17-80ba-4b93-9f96-29e68f73f927-workload-socket" pod "938e8e17-80ba-4b93-9f96-29e68f73f927" (UID: "938e8e17-80ba-4b93-9f96-29e68f73f927"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:27:00.474195 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:00.474158    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/938e8e17-80ba-4b93-9f96-29e68f73f927-istio-podinfo" pod "938e8e17-80ba-4b93-9f96-29e68f73f927" (UID: "938e8e17-80ba-4b93-9f96-29e68f73f927"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 03:27:00.475221 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:00.475178    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/938e8e17-80ba-4b93-9f96-29e68f73f927-istio-envoy" pod "938e8e17-80ba-4b93-9f96-29e68f73f927" (UID: "938e8e17-80ba-4b93-9f96-29e68f73f927"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:27:00.476762 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:00.476740    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/938e8e17-80ba-4b93-9f96-29e68f73f927-kube-api-access-68b6p" pod "938e8e17-80ba-4b93-9f96-29e68f73f927" (UID: "938e8e17-80ba-4b93-9f96-29e68f73f927"). InnerVolumeSpecName "kube-api-access-68b6p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:27:00.476988 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:00.476963    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/938e8e17-80ba-4b93-9f96-29e68f73f927-istio-token" pod "938e8e17-80ba-4b93-9f96-29e68f73f927" (UID: "938e8e17-80ba-4b93-9f96-29e68f73f927"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:27:00.519809 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:00.519778    4496 generic.go:356] "Generic (PLEG): container finished" podID="938e8e17-80ba-4b93-9f96-29e68f73f927" containerID="843543e593513d43dd923e6a55ddc2f2bb1a95c51f7b86f5ab4ff1a6cae60e03" exitCode=0
Nov 02 03:27:00.519978 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:00.519845    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-x5gwb"
Nov 02 03:27:00.519978 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:00.519865    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-x5gwb" event={"ID":"938e8e17-80ba-4b93-9f96-29e68f73f927","Type":"ContainerDied","Data":"843543e593513d43dd923e6a55ddc2f2bb1a95c51f7b86f5ab4ff1a6cae60e03"}
Nov 02 03:27:00.519978 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:00.519926    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-x5gwb" event={"ID":"938e8e17-80ba-4b93-9f96-29e68f73f927","Type":"ContainerDied","Data":"4e6387c2c10ac5e347b2768084216de3d372c8c95bf3930ab5e250d7c4ed9ca5"}
Nov 02 03:27:00.519978 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:00.519960    4496 scope.go:122] "RemoveContainer" containerID="843543e593513d43dd923e6a55ddc2f2bb1a95c51f7b86f5ab4ff1a6cae60e03"
Nov 02 03:27:00.534205 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:00.534115    4496 scope.go:122] "RemoveContainer" containerID="843543e593513d43dd923e6a55ddc2f2bb1a95c51f7b86f5ab4ff1a6cae60e03"
Nov 02 03:27:00.534473 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:27:00.534443    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"843543e593513d43dd923e6a55ddc2f2bb1a95c51f7b86f5ab4ff1a6cae60e03\": container with ID starting with 843543e593513d43dd923e6a55ddc2f2bb1a95c51f7b86f5ab4ff1a6cae60e03 not found: ID does not exist" containerID="843543e593513d43dd923e6a55ddc2f2bb1a95c51f7b86f5ab4ff1a6cae60e03"
Nov 02 03:27:00.534569 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:00.534483    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"843543e593513d43dd923e6a55ddc2f2bb1a95c51f7b86f5ab4ff1a6cae60e03"} err="failed to get container status \"843543e593513d43dd923e6a55ddc2f2bb1a95c51f7b86f5ab4ff1a6cae60e03\": rpc error: code = NotFound desc = could not find container \"843543e593513d43dd923e6a55ddc2f2bb1a95c51f7b86f5ab4ff1a6cae60e03\": container with ID starting with 843543e593513d43dd923e6a55ddc2f2bb1a95c51f7b86f5ab4ff1a6cae60e03 not found: ID does not exist"
Nov 02 03:27:00.573780 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:00.573746    4496 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/938e8e17-80ba-4b93-9f96-29e68f73f927-istio-token\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:27:00.574102 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:00.574075    4496 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/938e8e17-80ba-4b93-9f96-29e68f73f927-istiod-ca-cert\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:27:00.574263 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:00.574128    4496 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/938e8e17-80ba-4b93-9f96-29e68f73f927-credential-socket\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:27:00.574263 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:00.574145    4496 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/938e8e17-80ba-4b93-9f96-29e68f73f927-istio-data\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:27:00.574263 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:00.574159    4496 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/938e8e17-80ba-4b93-9f96-29e68f73f927-istio-podinfo\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:27:00.574263 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:00.574173    4496 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/938e8e17-80ba-4b93-9f96-29e68f73f927-workload-certs\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:27:00.574263 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:00.574190    4496 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/938e8e17-80ba-4b93-9f96-29e68f73f927-istio-envoy\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:27:00.574263 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:00.574205    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68b6p\" (UniqueName: \"kubernetes.io/projected/938e8e17-80ba-4b93-9f96-29e68f73f927-kube-api-access-68b6p\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:27:00.574263 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:00.574222    4496 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/938e8e17-80ba-4b93-9f96-29e68f73f927-workload-socket\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:27:00.689926 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:00.689847    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-x5gwb"]
Nov 02 03:27:00.696080 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:00.696054    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-x5gwb"]
Nov 02 03:27:01.533396 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:01.533267    4496 generic.go:356] "Generic (PLEG): container finished" podID="0031b60a-f888-4c85-8999-07bddc61a513" containerID="a0fbd673106ccac658e925ad94554076f70660e4f737063374097d2828d37ad8" exitCode=0
Nov 02 03:27:01.533396 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:01.533316    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xstsp" event={"ID":"0031b60a-f888-4c85-8999-07bddc61a513","Type":"ContainerDied","Data":"a0fbd673106ccac658e925ad94554076f70660e4f737063374097d2828d37ad8"}
Nov 02 03:27:01.555595 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:01.555572    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xstsp"
Nov 02 03:27:01.575462 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:01.575432    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="938e8e17-80ba-4b93-9f96-29e68f73f927" path="/var/lib/kubelet/pods/938e8e17-80ba-4b93-9f96-29e68f73f927/volumes"
Nov 02 03:27:01.616992 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:01.587943    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0031b60a-f888-4c85-8999-07bddc61a513-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0031b60a-f888-4c85-8999-07bddc61a513-istio-podinfo\") pod \"0031b60a-f888-4c85-8999-07bddc61a513\" (UID: \"0031b60a-f888-4c85-8999-07bddc61a513\") "
Nov 02 03:27:01.616992 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:01.588042    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0031b60a-f888-4c85-8999-07bddc61a513-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0031b60a-f888-4c85-8999-07bddc61a513-credential-socket\") pod \"0031b60a-f888-4c85-8999-07bddc61a513\" (UID: \"0031b60a-f888-4c85-8999-07bddc61a513\") "
Nov 02 03:27:01.616992 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:01.594424    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0031b60a-f888-4c85-8999-07bddc61a513-credential-socket" pod "0031b60a-f888-4c85-8999-07bddc61a513" (UID: "0031b60a-f888-4c85-8999-07bddc61a513"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:27:01.616992 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:01.600713    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0031b60a-f888-4c85-8999-07bddc61a513-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0031b60a-f888-4c85-8999-07bddc61a513-workload-socket\") pod \"0031b60a-f888-4c85-8999-07bddc61a513\" (UID: \"0031b60a-f888-4c85-8999-07bddc61a513\") "
Nov 02 03:27:01.616992 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:01.600906    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0031b60a-f888-4c85-8999-07bddc61a513-kube-api-access-2ltmb\" (UniqueName: \"kubernetes.io/projected/0031b60a-f888-4c85-8999-07bddc61a513-kube-api-access-2ltmb\") pod \"0031b60a-f888-4c85-8999-07bddc61a513\" (UID: \"0031b60a-f888-4c85-8999-07bddc61a513\") "
Nov 02 03:27:01.616992 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:01.600944    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0031b60a-f888-4c85-8999-07bddc61a513-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0031b60a-f888-4c85-8999-07bddc61a513-istio-envoy\") pod \"0031b60a-f888-4c85-8999-07bddc61a513\" (UID: \"0031b60a-f888-4c85-8999-07bddc61a513\") "
Nov 02 03:27:01.616992 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:01.600995    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0031b60a-f888-4c85-8999-07bddc61a513-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0031b60a-f888-4c85-8999-07bddc61a513-istiod-ca-cert\") pod \"0031b60a-f888-4c85-8999-07bddc61a513\" (UID: \"0031b60a-f888-4c85-8999-07bddc61a513\") "
Nov 02 03:27:01.616992 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:01.601141    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0031b60a-f888-4c85-8999-07bddc61a513-istio-token\" (UniqueName: \"kubernetes.io/projected/0031b60a-f888-4c85-8999-07bddc61a513-istio-token\") pod \"0031b60a-f888-4c85-8999-07bddc61a513\" (UID: \"0031b60a-f888-4c85-8999-07bddc61a513\") "
Nov 02 03:27:01.616992 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:01.601202    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0031b60a-f888-4c85-8999-07bddc61a513-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0031b60a-f888-4c85-8999-07bddc61a513-workload-certs\") pod \"0031b60a-f888-4c85-8999-07bddc61a513\" (UID: \"0031b60a-f888-4c85-8999-07bddc61a513\") "
Nov 02 03:27:01.616992 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:01.601243    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0031b60a-f888-4c85-8999-07bddc61a513-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0031b60a-f888-4c85-8999-07bddc61a513-istio-data\") pod \"0031b60a-f888-4c85-8999-07bddc61a513\" (UID: \"0031b60a-f888-4c85-8999-07bddc61a513\") "
Nov 02 03:27:01.616992 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:01.602538    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0031b60a-f888-4c85-8999-07bddc61a513-istiod-ca-cert" pod "0031b60a-f888-4c85-8999-07bddc61a513" (UID: "0031b60a-f888-4c85-8999-07bddc61a513"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:27:01.616992 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:01.602798    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0031b60a-f888-4c85-8999-07bddc61a513-workload-socket" pod "0031b60a-f888-4c85-8999-07bddc61a513" (UID: "0031b60a-f888-4c85-8999-07bddc61a513"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:27:01.616992 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:01.603494    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0031b60a-f888-4c85-8999-07bddc61a513-istio-data" pod "0031b60a-f888-4c85-8999-07bddc61a513" (UID: "0031b60a-f888-4c85-8999-07bddc61a513"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:27:01.616992 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:01.603633    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0031b60a-f888-4c85-8999-07bddc61a513-workload-certs" pod "0031b60a-f888-4c85-8999-07bddc61a513" (UID: "0031b60a-f888-4c85-8999-07bddc61a513"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:27:01.616992 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:01.607440    4496 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0031b60a-f888-4c85-8999-07bddc61a513-istiod-ca-cert\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:27:01.616992 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:01.607465    4496 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0031b60a-f888-4c85-8999-07bddc61a513-workload-certs\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:27:01.616992 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:01.607481    4496 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0031b60a-f888-4c85-8999-07bddc61a513-istio-data\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:27:01.616992 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:01.607490    4496 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0031b60a-f888-4c85-8999-07bddc61a513-credential-socket\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:27:01.616992 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:01.607499    4496 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0031b60a-f888-4c85-8999-07bddc61a513-workload-socket\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:27:01.629217 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:01.619099    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0031b60a-f888-4c85-8999-07bddc61a513-istio-podinfo" pod "0031b60a-f888-4c85-8999-07bddc61a513" (UID: "0031b60a-f888-4c85-8999-07bddc61a513"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 03:27:01.629217 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:01.619143    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0031b60a-f888-4c85-8999-07bddc61a513-istio-token" pod "0031b60a-f888-4c85-8999-07bddc61a513" (UID: "0031b60a-f888-4c85-8999-07bddc61a513"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:27:01.629217 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:01.622861    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0031b60a-f888-4c85-8999-07bddc61a513-istio-envoy" pod "0031b60a-f888-4c85-8999-07bddc61a513" (UID: "0031b60a-f888-4c85-8999-07bddc61a513"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:27:01.633679 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:01.633646    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0031b60a-f888-4c85-8999-07bddc61a513-kube-api-access-2ltmb" pod "0031b60a-f888-4c85-8999-07bddc61a513" (UID: "0031b60a-f888-4c85-8999-07bddc61a513"). InnerVolumeSpecName "kube-api-access-2ltmb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:27:01.709286 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:01.709253    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ltmb\" (UniqueName: \"kubernetes.io/projected/0031b60a-f888-4c85-8999-07bddc61a513-kube-api-access-2ltmb\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:27:01.709286 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:01.709289    4496 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0031b60a-f888-4c85-8999-07bddc61a513-istio-envoy\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:27:01.709779 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:01.709300    4496 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0031b60a-f888-4c85-8999-07bddc61a513-istio-token\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:27:01.709779 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:01.709309    4496 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0031b60a-f888-4c85-8999-07bddc61a513-istio-podinfo\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:27:01.907648 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:01.907621    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-277pb"
Nov 02 03:27:02.012942 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:02.012908    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3051f935-d87a-422e-9073-909b2c2768ac-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3051f935-d87a-422e-9073-909b2c2768ac-istio-podinfo\") pod \"3051f935-d87a-422e-9073-909b2c2768ac\" (UID: \"3051f935-d87a-422e-9073-909b2c2768ac\") "
Nov 02 03:27:02.013130 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:02.013038    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3051f935-d87a-422e-9073-909b2c2768ac-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3051f935-d87a-422e-9073-909b2c2768ac-istio-data\") pod \"3051f935-d87a-422e-9073-909b2c2768ac\" (UID: \"3051f935-d87a-422e-9073-909b2c2768ac\") "
Nov 02 03:27:02.013183 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:02.013124    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3051f935-d87a-422e-9073-909b2c2768ac-kube-api-access-p62vw\" (UniqueName: \"kubernetes.io/projected/3051f935-d87a-422e-9073-909b2c2768ac-kube-api-access-p62vw\") pod \"3051f935-d87a-422e-9073-909b2c2768ac\" (UID: \"3051f935-d87a-422e-9073-909b2c2768ac\") "
Nov 02 03:27:02.013183 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:02.013169    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3051f935-d87a-422e-9073-909b2c2768ac-istio-token\" (UniqueName: \"kubernetes.io/projected/3051f935-d87a-422e-9073-909b2c2768ac-istio-token\") pod \"3051f935-d87a-422e-9073-909b2c2768ac\" (UID: \"3051f935-d87a-422e-9073-909b2c2768ac\") "
Nov 02 03:27:02.013285 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:02.013232    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3051f935-d87a-422e-9073-909b2c2768ac-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3051f935-d87a-422e-9073-909b2c2768ac-credential-socket\") pod \"3051f935-d87a-422e-9073-909b2c2768ac\" (UID: \"3051f935-d87a-422e-9073-909b2c2768ac\") "
Nov 02 03:27:02.013337 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:02.013297    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3051f935-d87a-422e-9073-909b2c2768ac-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3051f935-d87a-422e-9073-909b2c2768ac-workload-socket\") pod \"3051f935-d87a-422e-9073-909b2c2768ac\" (UID: \"3051f935-d87a-422e-9073-909b2c2768ac\") "
Nov 02 03:27:02.013403 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:02.013349    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3051f935-d87a-422e-9073-909b2c2768ac-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3051f935-d87a-422e-9073-909b2c2768ac-workload-certs\") pod \"3051f935-d87a-422e-9073-909b2c2768ac\" (UID: \"3051f935-d87a-422e-9073-909b2c2768ac\") "
Nov 02 03:27:02.013403 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:02.013386    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3051f935-d87a-422e-9073-909b2c2768ac-istio-data" pod "3051f935-d87a-422e-9073-909b2c2768ac" (UID: "3051f935-d87a-422e-9073-909b2c2768ac"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:27:02.013526 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:02.013440    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3051f935-d87a-422e-9073-909b2c2768ac-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3051f935-d87a-422e-9073-909b2c2768ac-istiod-ca-cert\") pod \"3051f935-d87a-422e-9073-909b2c2768ac\" (UID: \"3051f935-d87a-422e-9073-909b2c2768ac\") "
Nov 02 03:27:02.013526 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:02.013512    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3051f935-d87a-422e-9073-909b2c2768ac-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3051f935-d87a-422e-9073-909b2c2768ac-istio-envoy\") pod \"3051f935-d87a-422e-9073-909b2c2768ac\" (UID: \"3051f935-d87a-422e-9073-909b2c2768ac\") "
Nov 02 03:27:02.013712 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:02.013628    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3051f935-d87a-422e-9073-909b2c2768ac-workload-socket" pod "3051f935-d87a-422e-9073-909b2c2768ac" (UID: "3051f935-d87a-422e-9073-909b2c2768ac"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:27:02.013809 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:02.013701    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3051f935-d87a-422e-9073-909b2c2768ac-workload-certs" pod "3051f935-d87a-422e-9073-909b2c2768ac" (UID: "3051f935-d87a-422e-9073-909b2c2768ac"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:27:02.013912 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:02.013888    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3051f935-d87a-422e-9073-909b2c2768ac-credential-socket" pod "3051f935-d87a-422e-9073-909b2c2768ac" (UID: "3051f935-d87a-422e-9073-909b2c2768ac"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:27:02.013966 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:02.013924    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3051f935-d87a-422e-9073-909b2c2768ac-istiod-ca-cert" pod "3051f935-d87a-422e-9073-909b2c2768ac" (UID: "3051f935-d87a-422e-9073-909b2c2768ac"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:27:02.014718 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:02.014690    4496 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3051f935-d87a-422e-9073-909b2c2768ac-istio-data\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:27:02.014839 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:02.014721    4496 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3051f935-d87a-422e-9073-909b2c2768ac-credential-socket\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:27:02.014839 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:02.014737    4496 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3051f935-d87a-422e-9073-909b2c2768ac-workload-socket\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:27:02.014839 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:02.014753    4496 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3051f935-d87a-422e-9073-909b2c2768ac-workload-certs\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:27:02.014839 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:02.014767    4496 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3051f935-d87a-422e-9073-909b2c2768ac-istiod-ca-cert\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:27:02.015570 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:02.015542    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3051f935-d87a-422e-9073-909b2c2768ac-istio-token" pod "3051f935-d87a-422e-9073-909b2c2768ac" (UID: "3051f935-d87a-422e-9073-909b2c2768ac"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:27:02.015662 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:02.015635    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3051f935-d87a-422e-9073-909b2c2768ac-istio-podinfo" pod "3051f935-d87a-422e-9073-909b2c2768ac" (UID: "3051f935-d87a-422e-9073-909b2c2768ac"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 03:27:02.015662 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:02.015640    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3051f935-d87a-422e-9073-909b2c2768ac-kube-api-access-p62vw" pod "3051f935-d87a-422e-9073-909b2c2768ac" (UID: "3051f935-d87a-422e-9073-909b2c2768ac"). InnerVolumeSpecName "kube-api-access-p62vw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:27:02.015842 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:02.015822    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3051f935-d87a-422e-9073-909b2c2768ac-istio-envoy" pod "3051f935-d87a-422e-9073-909b2c2768ac" (UID: "3051f935-d87a-422e-9073-909b2c2768ac"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:27:02.116564 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:02.116473    4496 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3051f935-d87a-422e-9073-909b2c2768ac-istio-envoy\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:27:02.116564 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:02.116515    4496 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3051f935-d87a-422e-9073-909b2c2768ac-istio-podinfo\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:27:02.116564 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:02.116531    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p62vw\" (UniqueName: \"kubernetes.io/projected/3051f935-d87a-422e-9073-909b2c2768ac-kube-api-access-p62vw\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:27:02.116564 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:02.116546    4496 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3051f935-d87a-422e-9073-909b2c2768ac-istio-token\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:27:02.542627 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:02.542590    4496 generic.go:356] "Generic (PLEG): container finished" podID="3051f935-d87a-422e-9073-909b2c2768ac" containerID="f508d79f63179aee6899d89526a45e19a8f3725b8c9ed7be843528ac401a5a4c" exitCode=0
Nov 02 03:27:02.542829 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:02.542636    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-277pb"
Nov 02 03:27:02.542829 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:02.542683    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-277pb" event={"ID":"3051f935-d87a-422e-9073-909b2c2768ac","Type":"ContainerDied","Data":"f508d79f63179aee6899d89526a45e19a8f3725b8c9ed7be843528ac401a5a4c"}
Nov 02 03:27:02.542829 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:02.542776    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-277pb" event={"ID":"3051f935-d87a-422e-9073-909b2c2768ac","Type":"ContainerDied","Data":"a219aca5ef632fbebf00cc4f9491fd47b079cb3a912ebd68670f37e3375b821b"}
Nov 02 03:27:02.542829 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:02.542807    4496 scope.go:122] "RemoveContainer" containerID="f508d79f63179aee6899d89526a45e19a8f3725b8c9ed7be843528ac401a5a4c"
Nov 02 03:27:02.545283 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:02.545254    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xstsp"
Nov 02 03:27:02.545403 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:02.545276    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xstsp" event={"ID":"0031b60a-f888-4c85-8999-07bddc61a513","Type":"ContainerDied","Data":"e3c56b1411f6453bf70fecc9dfd7bd8414a73414691a4308a3ca62970c04cd74"}
Nov 02 03:27:02.556629 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:02.556612    4496 scope.go:122] "RemoveContainer" containerID="f508d79f63179aee6899d89526a45e19a8f3725b8c9ed7be843528ac401a5a4c"
Nov 02 03:27:02.557038 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:27:02.556926    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f508d79f63179aee6899d89526a45e19a8f3725b8c9ed7be843528ac401a5a4c\": container with ID starting with f508d79f63179aee6899d89526a45e19a8f3725b8c9ed7be843528ac401a5a4c not found: ID does not exist" containerID="f508d79f63179aee6899d89526a45e19a8f3725b8c9ed7be843528ac401a5a4c"
Nov 02 03:27:02.557038 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:02.556961    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f508d79f63179aee6899d89526a45e19a8f3725b8c9ed7be843528ac401a5a4c"} err="failed to get container status \"f508d79f63179aee6899d89526a45e19a8f3725b8c9ed7be843528ac401a5a4c\": rpc error: code = NotFound desc = could not find container \"f508d79f63179aee6899d89526a45e19a8f3725b8c9ed7be843528ac401a5a4c\": container with ID starting with f508d79f63179aee6899d89526a45e19a8f3725b8c9ed7be843528ac401a5a4c not found: ID does not exist"
Nov 02 03:27:02.557038 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:02.556987    4496 scope.go:122] "RemoveContainer" containerID="a0fbd673106ccac658e925ad94554076f70660e4f737063374097d2828d37ad8"
Nov 02 03:27:02.762220 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:02.762182    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xstsp"]
Nov 02 03:27:02.765788 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:02.765762    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xstsp"]
Nov 02 03:27:02.875175 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:02.875095    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-277pb"]
Nov 02 03:27:02.878717 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:02.878691    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-277pb"]
Nov 02 03:27:03.586044 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:27:03.577043    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 03:27:03.683091 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:03.671818    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0031b60a-f888-4c85-8999-07bddc61a513" path="/var/lib/kubelet/pods/0031b60a-f888-4c85-8999-07bddc61a513/volumes"
Nov 02 03:27:03.683091 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:03.672552    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3051f935-d87a-422e-9073-909b2c2768ac" path="/var/lib/kubelet/pods/3051f935-d87a-422e-9073-909b2c2768ac/volumes"
Nov 02 03:27:07.841456 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:07.841418    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d6544b-wdg56"]
Nov 02 03:27:07.866932 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:07.866893    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d6544b-wdg56"]
Nov 02 03:27:09.231117 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:09.231083    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-5fp6v"]
Nov 02 03:27:09.231518 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:09.231284    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-5fp6v" podUID="6b2e2446-f387-4ea4-8787-1fa5f47e18e6" containerName="istio-proxy" containerID="cri-o://61a831abd73b508ee30c4060eacb78400bc5818568d8c7e9517c12efadf55d88" gracePeriod=30
Nov 02 03:27:09.590496 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:09.590422    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fa36a00-43b2-41d6-a789-fe09585a8917" path="/var/lib/kubelet/pods/0fa36a00-43b2-41d6-a789-fe09585a8917/volumes"
Nov 02 03:27:09.796382 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:09.796347    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-fn4nf"]
Nov 02 03:27:09.796576 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:09.796553    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-fn4nf" podUID="90252c4b-9031-4994-86e1-9b7ae966cd7e" containerName="istio-proxy" containerID="cri-o://2340ff8ddfae8d2f582513548f272f8bbebb3fda69a164f7541e705420fac92e" gracePeriod=30
Nov 02 03:27:09.832601 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:09.832561    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-ff2fd"]
Nov 02 03:27:09.832875 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:09.832848    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-ff2fd" podUID="e11f3625-7e4e-4124-a232-8cf67769fdfc" containerName="istio-proxy" containerID="cri-o://6af658c35919aae30a6a2946ea71e2f7fde491043366d64674a0c1636ab0a37e" gracePeriod=30
Nov 02 03:27:09.838136 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:09.838104    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-t4dvl"]
Nov 02 03:27:09.838371 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:09.838345    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-t4dvl" podUID="013b67e2-aae9-4380-baf6-de93a6a9d18b" containerName="istio-proxy" containerID="cri-o://a5c6a4c4b834bd5d7091bedb7a209bfcd4b79b2694f89912363f29f1c05ee425" gracePeriod=30
Nov 02 03:27:13.924335 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:13.924291    4496 patch_prober.go:29] interesting pod/e2e-worker-w3-858236-openshift-default-74fcd64bf5-5fp6v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 03:27:13.924767 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:13.924376    4496 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-5fp6v" podUID="6b2e2446-f387-4ea4-8787-1fa5f47e18e6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 03:27:14.254806 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:14.254772    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c9d1fc-wcp6v"]
Nov 02 03:27:14.306861 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:14.306829    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c9d1fc-wcp6v"]
Nov 02 03:27:14.577752 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:14.577727    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-5fp6v"
Nov 02 03:27:14.654909 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:14.654862    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b2e2446-f387-4ea4-8787-1fa5f47e18e6-istio-token\" (UniqueName: \"kubernetes.io/projected/6b2e2446-f387-4ea4-8787-1fa5f47e18e6-istio-token\") pod \"6b2e2446-f387-4ea4-8787-1fa5f47e18e6\" (UID: \"6b2e2446-f387-4ea4-8787-1fa5f47e18e6\") "
Nov 02 03:27:14.654909 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:14.654913    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b2e2446-f387-4ea4-8787-1fa5f47e18e6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b2e2446-f387-4ea4-8787-1fa5f47e18e6-istio-envoy\") pod \"6b2e2446-f387-4ea4-8787-1fa5f47e18e6\" (UID: \"6b2e2446-f387-4ea4-8787-1fa5f47e18e6\") "
Nov 02 03:27:14.655206 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:14.654990    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b2e2446-f387-4ea4-8787-1fa5f47e18e6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b2e2446-f387-4ea4-8787-1fa5f47e18e6-workload-socket\") pod \"6b2e2446-f387-4ea4-8787-1fa5f47e18e6\" (UID: \"6b2e2446-f387-4ea4-8787-1fa5f47e18e6\") "
Nov 02 03:27:14.655206 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:14.655041    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b2e2446-f387-4ea4-8787-1fa5f47e18e6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b2e2446-f387-4ea4-8787-1fa5f47e18e6-credential-socket\") pod \"6b2e2446-f387-4ea4-8787-1fa5f47e18e6\" (UID: \"6b2e2446-f387-4ea4-8787-1fa5f47e18e6\") "
Nov 02 03:27:14.655206 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:14.655074    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b2e2446-f387-4ea4-8787-1fa5f47e18e6-kube-api-access-9ggms\" (UniqueName: \"kubernetes.io/projected/6b2e2446-f387-4ea4-8787-1fa5f47e18e6-kube-api-access-9ggms\") pod \"6b2e2446-f387-4ea4-8787-1fa5f47e18e6\" (UID: \"6b2e2446-f387-4ea4-8787-1fa5f47e18e6\") "
Nov 02 03:27:14.655206 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:14.655122    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b2e2446-f387-4ea4-8787-1fa5f47e18e6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b2e2446-f387-4ea4-8787-1fa5f47e18e6-istio-data\") pod \"6b2e2446-f387-4ea4-8787-1fa5f47e18e6\" (UID: \"6b2e2446-f387-4ea4-8787-1fa5f47e18e6\") "
Nov 02 03:27:14.655206 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:14.655183    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b2e2446-f387-4ea4-8787-1fa5f47e18e6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b2e2446-f387-4ea4-8787-1fa5f47e18e6-workload-certs\") pod \"6b2e2446-f387-4ea4-8787-1fa5f47e18e6\" (UID: \"6b2e2446-f387-4ea4-8787-1fa5f47e18e6\") "
Nov 02 03:27:14.655471 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:14.655260    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b2e2446-f387-4ea4-8787-1fa5f47e18e6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b2e2446-f387-4ea4-8787-1fa5f47e18e6-istiod-ca-cert\") pod \"6b2e2446-f387-4ea4-8787-1fa5f47e18e6\" (UID: \"6b2e2446-f387-4ea4-8787-1fa5f47e18e6\") "
Nov 02 03:27:14.655471 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:14.655333    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b2e2446-f387-4ea4-8787-1fa5f47e18e6-workload-socket" pod "6b2e2446-f387-4ea4-8787-1fa5f47e18e6" (UID: "6b2e2446-f387-4ea4-8787-1fa5f47e18e6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:27:14.655471 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:14.655381    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6b2e2446-f387-4ea4-8787-1fa5f47e18e6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b2e2446-f387-4ea4-8787-1fa5f47e18e6-istio-podinfo\") pod \"6b2e2446-f387-4ea4-8787-1fa5f47e18e6\" (UID: \"6b2e2446-f387-4ea4-8787-1fa5f47e18e6\") "
Nov 02 03:27:14.655471 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:14.655404    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b2e2446-f387-4ea4-8787-1fa5f47e18e6-credential-socket" pod "6b2e2446-f387-4ea4-8787-1fa5f47e18e6" (UID: "6b2e2446-f387-4ea4-8787-1fa5f47e18e6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:27:14.655670 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:14.655546    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b2e2446-f387-4ea4-8787-1fa5f47e18e6-workload-certs" pod "6b2e2446-f387-4ea4-8787-1fa5f47e18e6" (UID: "6b2e2446-f387-4ea4-8787-1fa5f47e18e6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:27:14.655670 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:14.655648    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b2e2446-f387-4ea4-8787-1fa5f47e18e6-istiod-ca-cert" pod "6b2e2446-f387-4ea4-8787-1fa5f47e18e6" (UID: "6b2e2446-f387-4ea4-8787-1fa5f47e18e6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:27:14.655786 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:14.655770    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b2e2446-f387-4ea4-8787-1fa5f47e18e6-istio-data" pod "6b2e2446-f387-4ea4-8787-1fa5f47e18e6" (UID: "6b2e2446-f387-4ea4-8787-1fa5f47e18e6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:27:14.656134 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:14.656108    4496 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b2e2446-f387-4ea4-8787-1fa5f47e18e6-workload-socket\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:27:14.656243 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:14.656138    4496 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b2e2446-f387-4ea4-8787-1fa5f47e18e6-credential-socket\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:27:14.656243 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:14.656155    4496 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b2e2446-f387-4ea4-8787-1fa5f47e18e6-istio-data\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:27:14.656243 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:14.656170    4496 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b2e2446-f387-4ea4-8787-1fa5f47e18e6-workload-certs\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:27:14.656243 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:14.656186    4496 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b2e2446-f387-4ea4-8787-1fa5f47e18e6-istiod-ca-cert\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:27:14.657436 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:14.657409    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b2e2446-f387-4ea4-8787-1fa5f47e18e6-istio-envoy" pod "6b2e2446-f387-4ea4-8787-1fa5f47e18e6" (UID: "6b2e2446-f387-4ea4-8787-1fa5f47e18e6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:27:14.657537 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:14.657476    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b2e2446-f387-4ea4-8787-1fa5f47e18e6-kube-api-access-9ggms" pod "6b2e2446-f387-4ea4-8787-1fa5f47e18e6" (UID: "6b2e2446-f387-4ea4-8787-1fa5f47e18e6"). InnerVolumeSpecName "kube-api-access-9ggms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:27:14.657537 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:14.657489    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6b2e2446-f387-4ea4-8787-1fa5f47e18e6-istio-podinfo" pod "6b2e2446-f387-4ea4-8787-1fa5f47e18e6" (UID: "6b2e2446-f387-4ea4-8787-1fa5f47e18e6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 03:27:14.657537 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:14.657501    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b2e2446-f387-4ea4-8787-1fa5f47e18e6-istio-token" pod "6b2e2446-f387-4ea4-8787-1fa5f47e18e6" (UID: "6b2e2446-f387-4ea4-8787-1fa5f47e18e6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:27:14.757636 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:14.757587    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ggms\" (UniqueName: \"kubernetes.io/projected/6b2e2446-f387-4ea4-8787-1fa5f47e18e6-kube-api-access-9ggms\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:27:14.757636 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:14.757630    4496 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b2e2446-f387-4ea4-8787-1fa5f47e18e6-istio-podinfo\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:27:14.757636 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:14.757645    4496 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b2e2446-f387-4ea4-8787-1fa5f47e18e6-istio-token\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:27:14.757861 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:14.757657    4496 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b2e2446-f387-4ea4-8787-1fa5f47e18e6-istio-envoy\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:27:14.792577 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:14.792543    4496 generic.go:356] "Generic (PLEG): container finished" podID="6b2e2446-f387-4ea4-8787-1fa5f47e18e6" containerID="61a831abd73b508ee30c4060eacb78400bc5818568d8c7e9517c12efadf55d88" exitCode=0
Nov 02 03:27:14.792749 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:14.792608    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-5fp6v"
Nov 02 03:27:14.792749 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:14.792624    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-5fp6v" event={"ID":"6b2e2446-f387-4ea4-8787-1fa5f47e18e6","Type":"ContainerDied","Data":"61a831abd73b508ee30c4060eacb78400bc5818568d8c7e9517c12efadf55d88"}
Nov 02 03:27:14.792749 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:14.792701    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-5fp6v" event={"ID":"6b2e2446-f387-4ea4-8787-1fa5f47e18e6","Type":"ContainerDied","Data":"d04c141255b9dbdeba41491fa0251e0081f9db3c4a9ad11d46fab49bcbb711b3"}
Nov 02 03:27:14.792749 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:14.792724    4496 scope.go:122] "RemoveContainer" containerID="61a831abd73b508ee30c4060eacb78400bc5818568d8c7e9517c12efadf55d88"
Nov 02 03:27:14.807785 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:14.807755    4496 scope.go:122] "RemoveContainer" containerID="61a831abd73b508ee30c4060eacb78400bc5818568d8c7e9517c12efadf55d88"
Nov 02 03:27:14.808426 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:27:14.808134    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61a831abd73b508ee30c4060eacb78400bc5818568d8c7e9517c12efadf55d88\": container with ID starting with 61a831abd73b508ee30c4060eacb78400bc5818568d8c7e9517c12efadf55d88 not found: ID does not exist" containerID="61a831abd73b508ee30c4060eacb78400bc5818568d8c7e9517c12efadf55d88"
Nov 02 03:27:14.808426 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:14.808172    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61a831abd73b508ee30c4060eacb78400bc5818568d8c7e9517c12efadf55d88"} err="failed to get container status \"61a831abd73b508ee30c4060eacb78400bc5818568d8c7e9517c12efadf55d88\": rpc error: code = NotFound desc = could not find container \"61a831abd73b508ee30c4060eacb78400bc5818568d8c7e9517c12efadf55d88\": container with ID starting with 61a831abd73b508ee30c4060eacb78400bc5818568d8c7e9517c12efadf55d88 not found: ID does not exist"
Nov 02 03:27:14.966258 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:14.966232    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-5fp6v"]
Nov 02 03:27:14.970609 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:14.970582    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-5fp6v"]
Nov 02 03:27:15.155186 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:15.155156    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-fn4nf"
Nov 02 03:27:15.209769 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:15.209743    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-ff2fd"
Nov 02 03:27:15.220820 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:15.220800    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-t4dvl"
Nov 02 03:27:15.264799 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:15.264765    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/90252c4b-9031-4994-86e1-9b7ae966cd7e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90252c4b-9031-4994-86e1-9b7ae966cd7e-istio-podinfo\") pod \"90252c4b-9031-4994-86e1-9b7ae966cd7e\" (UID: \"90252c4b-9031-4994-86e1-9b7ae966cd7e\") "
Nov 02 03:27:15.264917 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:15.264904    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e11f3625-7e4e-4124-a232-8cf67769fdfc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e11f3625-7e4e-4124-a232-8cf67769fdfc-istio-data\") pod \"e11f3625-7e4e-4124-a232-8cf67769fdfc\" (UID: \"e11f3625-7e4e-4124-a232-8cf67769fdfc\") "
Nov 02 03:27:15.264976 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:15.264939    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e11f3625-7e4e-4124-a232-8cf67769fdfc-kube-api-access-r8fzm\" (UniqueName: \"kubernetes.io/projected/e11f3625-7e4e-4124-a232-8cf67769fdfc-kube-api-access-r8fzm\") pod \"e11f3625-7e4e-4124-a232-8cf67769fdfc\" (UID: \"e11f3625-7e4e-4124-a232-8cf67769fdfc\") "
Nov 02 03:27:15.265071 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:15.264986    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90252c4b-9031-4994-86e1-9b7ae966cd7e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90252c4b-9031-4994-86e1-9b7ae966cd7e-istio-envoy\") pod \"90252c4b-9031-4994-86e1-9b7ae966cd7e\" (UID: \"90252c4b-9031-4994-86e1-9b7ae966cd7e\") "
Nov 02 03:27:15.265133 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:15.265070    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e11f3625-7e4e-4124-a232-8cf67769fdfc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e11f3625-7e4e-4124-a232-8cf67769fdfc-istio-envoy\") pod \"e11f3625-7e4e-4124-a232-8cf67769fdfc\" (UID: \"e11f3625-7e4e-4124-a232-8cf67769fdfc\") "
Nov 02 03:27:15.265133 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:15.265105    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90252c4b-9031-4994-86e1-9b7ae966cd7e-istio-token\" (UniqueName: \"kubernetes.io/projected/90252c4b-9031-4994-86e1-9b7ae966cd7e-istio-token\") pod \"90252c4b-9031-4994-86e1-9b7ae966cd7e\" (UID: \"90252c4b-9031-4994-86e1-9b7ae966cd7e\") "
Nov 02 03:27:15.265243 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:15.265165    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90252c4b-9031-4994-86e1-9b7ae966cd7e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90252c4b-9031-4994-86e1-9b7ae966cd7e-workload-socket\") pod \"90252c4b-9031-4994-86e1-9b7ae966cd7e\" (UID: \"90252c4b-9031-4994-86e1-9b7ae966cd7e\") "
Nov 02 03:27:15.265243 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:15.265187    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e11f3625-7e4e-4124-a232-8cf67769fdfc-istio-data" pod "e11f3625-7e4e-4124-a232-8cf67769fdfc" (UID: "e11f3625-7e4e-4124-a232-8cf67769fdfc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:27:15.265243 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:15.265200    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90252c4b-9031-4994-86e1-9b7ae966cd7e-kube-api-access-9zx25\" (UniqueName: \"kubernetes.io/projected/90252c4b-9031-4994-86e1-9b7ae966cd7e-kube-api-access-9zx25\") pod \"90252c4b-9031-4994-86e1-9b7ae966cd7e\" (UID: \"90252c4b-9031-4994-86e1-9b7ae966cd7e\") "
Nov 02 03:27:15.265399 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:15.265264    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e11f3625-7e4e-4124-a232-8cf67769fdfc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e11f3625-7e4e-4124-a232-8cf67769fdfc-istio-podinfo\") pod \"e11f3625-7e4e-4124-a232-8cf67769fdfc\" (UID: \"e11f3625-7e4e-4124-a232-8cf67769fdfc\") "
Nov 02 03:27:15.265399 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:15.265309    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e11f3625-7e4e-4124-a232-8cf67769fdfc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e11f3625-7e4e-4124-a232-8cf67769fdfc-istiod-ca-cert\") pod \"e11f3625-7e4e-4124-a232-8cf67769fdfc\" (UID: \"e11f3625-7e4e-4124-a232-8cf67769fdfc\") "
Nov 02 03:27:15.265399 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:15.265349    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e11f3625-7e4e-4124-a232-8cf67769fdfc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e11f3625-7e4e-4124-a232-8cf67769fdfc-workload-certs\") pod \"e11f3625-7e4e-4124-a232-8cf67769fdfc\" (UID: \"e11f3625-7e4e-4124-a232-8cf67769fdfc\") "
Nov 02 03:27:15.265550 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:15.265440    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90252c4b-9031-4994-86e1-9b7ae966cd7e-workload-socket" pod "90252c4b-9031-4994-86e1-9b7ae966cd7e" (UID: "90252c4b-9031-4994-86e1-9b7ae966cd7e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:27:15.265550 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:15.265457    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90252c4b-9031-4994-86e1-9b7ae966cd7e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90252c4b-9031-4994-86e1-9b7ae966cd7e-credential-socket\") pod \"90252c4b-9031-4994-86e1-9b7ae966cd7e\" (UID: \"90252c4b-9031-4994-86e1-9b7ae966cd7e\") "
Nov 02 03:27:15.265550 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:15.265512    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e11f3625-7e4e-4124-a232-8cf67769fdfc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e11f3625-7e4e-4124-a232-8cf67769fdfc-credential-socket\") pod \"e11f3625-7e4e-4124-a232-8cf67769fdfc\" (UID: \"e11f3625-7e4e-4124-a232-8cf67769fdfc\") "
Nov 02 03:27:15.265706 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:15.265552    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90252c4b-9031-4994-86e1-9b7ae966cd7e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90252c4b-9031-4994-86e1-9b7ae966cd7e-istio-data\") pod \"90252c4b-9031-4994-86e1-9b7ae966cd7e\" (UID: \"90252c4b-9031-4994-86e1-9b7ae966cd7e\") "
Nov 02 03:27:15.265706 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:15.265592    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e11f3625-7e4e-4124-a232-8cf67769fdfc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e11f3625-7e4e-4124-a232-8cf67769fdfc-workload-socket\") pod \"e11f3625-7e4e-4124-a232-8cf67769fdfc\" (UID: \"e11f3625-7e4e-4124-a232-8cf67769fdfc\") "
Nov 02 03:27:15.265706 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:15.265635    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e11f3625-7e4e-4124-a232-8cf67769fdfc-istio-token\" (UniqueName: \"kubernetes.io/projected/e11f3625-7e4e-4124-a232-8cf67769fdfc-istio-token\") pod \"e11f3625-7e4e-4124-a232-8cf67769fdfc\" (UID: \"e11f3625-7e4e-4124-a232-8cf67769fdfc\") "
Nov 02 03:27:15.265706 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:15.265672    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90252c4b-9031-4994-86e1-9b7ae966cd7e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90252c4b-9031-4994-86e1-9b7ae966cd7e-workload-certs\") pod \"90252c4b-9031-4994-86e1-9b7ae966cd7e\" (UID: \"90252c4b-9031-4994-86e1-9b7ae966cd7e\") "
Nov 02 03:27:15.265906 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:15.265725    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90252c4b-9031-4994-86e1-9b7ae966cd7e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90252c4b-9031-4994-86e1-9b7ae966cd7e-istiod-ca-cert\") pod \"90252c4b-9031-4994-86e1-9b7ae966cd7e\" (UID: \"90252c4b-9031-4994-86e1-9b7ae966cd7e\") "
Nov 02 03:27:15.265963 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:15.265908    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e11f3625-7e4e-4124-a232-8cf67769fdfc-istiod-ca-cert" pod "e11f3625-7e4e-4124-a232-8cf67769fdfc" (UID: "e11f3625-7e4e-4124-a232-8cf67769fdfc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:27:15.266383 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:15.266358    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90252c4b-9031-4994-86e1-9b7ae966cd7e-istiod-ca-cert" pod "90252c4b-9031-4994-86e1-9b7ae966cd7e" (UID: "90252c4b-9031-4994-86e1-9b7ae966cd7e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:27:15.266541 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:15.266472    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90252c4b-9031-4994-86e1-9b7ae966cd7e-credential-socket" pod "90252c4b-9031-4994-86e1-9b7ae966cd7e" (UID: "90252c4b-9031-4994-86e1-9b7ae966cd7e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:27:15.266634 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:15.266582    4496 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e11f3625-7e4e-4124-a232-8cf67769fdfc-istio-data\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:27:15.266736 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:15.266722    4496 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90252c4b-9031-4994-86e1-9b7ae966cd7e-workload-socket\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:27:15.266863 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:15.266847    4496 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e11f3625-7e4e-4124-a232-8cf67769fdfc-istiod-ca-cert\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:27:15.266981 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:15.266637    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90252c4b-9031-4994-86e1-9b7ae966cd7e-istio-data" pod "90252c4b-9031-4994-86e1-9b7ae966cd7e" (UID: "90252c4b-9031-4994-86e1-9b7ae966cd7e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:27:15.267146 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:15.266774    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e11f3625-7e4e-4124-a232-8cf67769fdfc-workload-certs" pod "e11f3625-7e4e-4124-a232-8cf67769fdfc" (UID: "e11f3625-7e4e-4124-a232-8cf67769fdfc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:27:15.267259 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:15.266784    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90252c4b-9031-4994-86e1-9b7ae966cd7e-workload-certs" pod "90252c4b-9031-4994-86e1-9b7ae966cd7e" (UID: "90252c4b-9031-4994-86e1-9b7ae966cd7e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:27:15.267379 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:15.266906    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e11f3625-7e4e-4124-a232-8cf67769fdfc-credential-socket" pod "e11f3625-7e4e-4124-a232-8cf67769fdfc" (UID: "e11f3625-7e4e-4124-a232-8cf67769fdfc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:27:15.267379 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:15.267086    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e11f3625-7e4e-4124-a232-8cf67769fdfc-workload-socket" pod "e11f3625-7e4e-4124-a232-8cf67769fdfc" (UID: "e11f3625-7e4e-4124-a232-8cf67769fdfc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:27:15.268200 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:15.268169    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/90252c4b-9031-4994-86e1-9b7ae966cd7e-istio-podinfo" pod "90252c4b-9031-4994-86e1-9b7ae966cd7e" (UID: "90252c4b-9031-4994-86e1-9b7ae966cd7e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 03:27:15.268532 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:15.268490    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90252c4b-9031-4994-86e1-9b7ae966cd7e-istio-envoy" pod "90252c4b-9031-4994-86e1-9b7ae966cd7e" (UID: "90252c4b-9031-4994-86e1-9b7ae966cd7e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:27:15.268605 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:15.268538    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e11f3625-7e4e-4124-a232-8cf67769fdfc-kube-api-access-r8fzm" pod "e11f3625-7e4e-4124-a232-8cf67769fdfc" (UID: "e11f3625-7e4e-4124-a232-8cf67769fdfc"). InnerVolumeSpecName "kube-api-access-r8fzm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:27:15.268737 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:15.268714    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90252c4b-9031-4994-86e1-9b7ae966cd7e-kube-api-access-9zx25" pod "90252c4b-9031-4994-86e1-9b7ae966cd7e" (UID: "90252c4b-9031-4994-86e1-9b7ae966cd7e"). InnerVolumeSpecName "kube-api-access-9zx25". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:27:15.268815 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:15.268761    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90252c4b-9031-4994-86e1-9b7ae966cd7e-istio-token" pod "90252c4b-9031-4994-86e1-9b7ae966cd7e" (UID: "90252c4b-9031-4994-86e1-9b7ae966cd7e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:27:15.269173 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:15.269152    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e11f3625-7e4e-4124-a232-8cf67769fdfc-istio-envoy" pod "e11f3625-7e4e-4124-a232-8cf67769fdfc" (UID: "e11f3625-7e4e-4124-a232-8cf67769fdfc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:27:15.269267 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:15.269151    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e11f3625-7e4e-4124-a232-8cf67769fdfc-istio-podinfo" pod "e11f3625-7e4e-4124-a232-8cf67769fdfc" (UID: "e11f3625-7e4e-4124-a232-8cf67769fdfc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 03:27:15.269392 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:15.269372    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e11f3625-7e4e-4124-a232-8cf67769fdfc-istio-token" pod "e11f3625-7e4e-4124-a232-8cf67769fdfc" (UID: "e11f3625-7e4e-4124-a232-8cf67769fdfc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:27:15.392376 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:15.389533    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/013b67e2-aae9-4380-baf6-de93a6a9d18b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/013b67e2-aae9-4380-baf6-de93a6a9d18b-workload-socket\") pod \"013b67e2-aae9-4380-baf6-de93a6a9d18b\" (UID: \"013b67e2-aae9-4380-baf6-de93a6a9d18b\") "
Nov 02 03:27:15.392376 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:15.389619    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/013b67e2-aae9-4380-baf6-de93a6a9d18b-kube-api-access-p8sgw\" (UniqueName: \"kubernetes.io/projected/013b67e2-aae9-4380-baf6-de93a6a9d18b-kube-api-access-p8sgw\") pod \"013b67e2-aae9-4380-baf6-de93a6a9d18b\" (UID: \"013b67e2-aae9-4380-baf6-de93a6a9d18b\") "
Nov 02 03:27:15.392376 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:15.389740    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/013b67e2-aae9-4380-baf6-de93a6a9d18b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/013b67e2-aae9-4380-baf6-de93a6a9d18b-credential-socket\") pod \"013b67e2-aae9-4380-baf6-de93a6a9d18b\" (UID: \"013b67e2-aae9-4380-baf6-de93a6a9d18b\") "
Nov 02 03:27:15.392376 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:15.389795    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/013b67e2-aae9-4380-baf6-de93a6a9d18b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/013b67e2-aae9-4380-baf6-de93a6a9d18b-istio-data\") pod \"013b67e2-aae9-4380-baf6-de93a6a9d18b\" (UID: \"013b67e2-aae9-4380-baf6-de93a6a9d18b\") "
Nov 02 03:27:15.392376 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:15.389831    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/013b67e2-aae9-4380-baf6-de93a6a9d18b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/013b67e2-aae9-4380-baf6-de93a6a9d18b-istio-podinfo\") pod \"013b67e2-aae9-4380-baf6-de93a6a9d18b\" (UID: \"013b67e2-aae9-4380-baf6-de93a6a9d18b\") "
Nov 02 03:27:15.392376 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:15.389885    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/013b67e2-aae9-4380-baf6-de93a6a9d18b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/013b67e2-aae9-4380-baf6-de93a6a9d18b-istiod-ca-cert\") pod \"013b67e2-aae9-4380-baf6-de93a6a9d18b\" (UID: \"013b67e2-aae9-4380-baf6-de93a6a9d18b\") "
Nov 02 03:27:15.392376 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:15.389966    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/013b67e2-aae9-4380-baf6-de93a6a9d18b-istio-token\" (UniqueName: \"kubernetes.io/projected/013b67e2-aae9-4380-baf6-de93a6a9d18b-istio-token\") pod \"013b67e2-aae9-4380-baf6-de93a6a9d18b\" (UID: \"013b67e2-aae9-4380-baf6-de93a6a9d18b\") "
Nov 02 03:27:15.392376 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:15.390031    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/013b67e2-aae9-4380-baf6-de93a6a9d18b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/013b67e2-aae9-4380-baf6-de93a6a9d18b-workload-certs\") pod \"013b67e2-aae9-4380-baf6-de93a6a9d18b\" (UID: \"013b67e2-aae9-4380-baf6-de93a6a9d18b\") "
Nov 02 03:27:15.392376 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:15.390116    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/013b67e2-aae9-4380-baf6-de93a6a9d18b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/013b67e2-aae9-4380-baf6-de93a6a9d18b-istio-envoy\") pod \"013b67e2-aae9-4380-baf6-de93a6a9d18b\" (UID: \"013b67e2-aae9-4380-baf6-de93a6a9d18b\") "
Nov 02 03:27:15.392376 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:15.390711    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8fzm\" (UniqueName: \"kubernetes.io/projected/e11f3625-7e4e-4124-a232-8cf67769fdfc-kube-api-access-r8fzm\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:27:15.392376 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:15.390737    4496 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90252c4b-9031-4994-86e1-9b7ae966cd7e-istio-envoy\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:27:15.392376 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:15.390754    4496 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e11f3625-7e4e-4124-a232-8cf67769fdfc-istio-envoy\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:27:15.392376 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:15.390769    4496 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90252c4b-9031-4994-86e1-9b7ae966cd7e-istio-token\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:27:15.392376 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:15.390783    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zx25\" (UniqueName: \"kubernetes.io/projected/90252c4b-9031-4994-86e1-9b7ae966cd7e-kube-api-access-9zx25\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:27:15.392376 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:15.390797    4496 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e11f3625-7e4e-4124-a232-8cf67769fdfc-istio-podinfo\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:27:15.392376 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:15.390813    4496 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e11f3625-7e4e-4124-a232-8cf67769fdfc-workload-certs\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:27:15.392376 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:15.390827    4496 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90252c4b-9031-4994-86e1-9b7ae966cd7e-credential-socket\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:27:15.392376 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:15.390843    4496 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e11f3625-7e4e-4124-a232-8cf67769fdfc-credential-socket\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:27:15.392376 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:15.390857    4496 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90252c4b-9031-4994-86e1-9b7ae966cd7e-istio-data\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:27:15.392376 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:15.390873    4496 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e11f3625-7e4e-4124-a232-8cf67769fdfc-workload-socket\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:27:15.392376 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:15.390887    4496 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e11f3625-7e4e-4124-a232-8cf67769fdfc-istio-token\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:27:15.392376 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:15.390901    4496 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90252c4b-9031-4994-86e1-9b7ae966cd7e-workload-certs\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:27:15.392376 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:15.390916    4496 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90252c4b-9031-4994-86e1-9b7ae966cd7e-istiod-ca-cert\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:27:15.392376 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:15.390932    4496 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90252c4b-9031-4994-86e1-9b7ae966cd7e-istio-podinfo\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:27:15.392376 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:15.391270    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/013b67e2-aae9-4380-baf6-de93a6a9d18b-istio-data" pod "013b67e2-aae9-4380-baf6-de93a6a9d18b" (UID: "013b67e2-aae9-4380-baf6-de93a6a9d18b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:27:15.392376 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:15.391503    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/013b67e2-aae9-4380-baf6-de93a6a9d18b-workload-socket" pod "013b67e2-aae9-4380-baf6-de93a6a9d18b" (UID: "013b67e2-aae9-4380-baf6-de93a6a9d18b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:27:15.392376 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:15.391878    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/013b67e2-aae9-4380-baf6-de93a6a9d18b-istiod-ca-cert" pod "013b67e2-aae9-4380-baf6-de93a6a9d18b" (UID: "013b67e2-aae9-4380-baf6-de93a6a9d18b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:27:15.396033 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:15.392732    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/013b67e2-aae9-4380-baf6-de93a6a9d18b-credential-socket" pod "013b67e2-aae9-4380-baf6-de93a6a9d18b" (UID: "013b67e2-aae9-4380-baf6-de93a6a9d18b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:27:15.396033 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:15.392948    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/013b67e2-aae9-4380-baf6-de93a6a9d18b-workload-certs" pod "013b67e2-aae9-4380-baf6-de93a6a9d18b" (UID: "013b67e2-aae9-4380-baf6-de93a6a9d18b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:27:15.415593 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:15.408731    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/013b67e2-aae9-4380-baf6-de93a6a9d18b-istio-token" pod "013b67e2-aae9-4380-baf6-de93a6a9d18b" (UID: "013b67e2-aae9-4380-baf6-de93a6a9d18b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:27:15.415593 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:15.408939    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/013b67e2-aae9-4380-baf6-de93a6a9d18b-istio-podinfo" pod "013b67e2-aae9-4380-baf6-de93a6a9d18b" (UID: "013b67e2-aae9-4380-baf6-de93a6a9d18b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 03:27:15.426853 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:15.426803    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/013b67e2-aae9-4380-baf6-de93a6a9d18b-istio-envoy" pod "013b67e2-aae9-4380-baf6-de93a6a9d18b" (UID: "013b67e2-aae9-4380-baf6-de93a6a9d18b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:27:15.433367 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:15.428142    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/013b67e2-aae9-4380-baf6-de93a6a9d18b-kube-api-access-p8sgw" pod "013b67e2-aae9-4380-baf6-de93a6a9d18b" (UID: "013b67e2-aae9-4380-baf6-de93a6a9d18b"). InnerVolumeSpecName "kube-api-access-p8sgw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:27:15.492923 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:15.492865    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8sgw\" (UniqueName: \"kubernetes.io/projected/013b67e2-aae9-4380-baf6-de93a6a9d18b-kube-api-access-p8sgw\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:27:15.492923 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:15.492905    4496 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/013b67e2-aae9-4380-baf6-de93a6a9d18b-credential-socket\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:27:15.492923 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:15.492922    4496 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/013b67e2-aae9-4380-baf6-de93a6a9d18b-istio-data\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:27:15.493383 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:15.492936    4496 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/013b67e2-aae9-4380-baf6-de93a6a9d18b-istio-podinfo\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:27:15.493383 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:15.492951    4496 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/013b67e2-aae9-4380-baf6-de93a6a9d18b-istiod-ca-cert\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:27:15.493383 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:15.492964    4496 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/013b67e2-aae9-4380-baf6-de93a6a9d18b-istio-token\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:27:15.493383 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:15.492978    4496 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/013b67e2-aae9-4380-baf6-de93a6a9d18b-workload-certs\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:27:15.493383 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:15.492992    4496 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/013b67e2-aae9-4380-baf6-de93a6a9d18b-istio-envoy\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:27:15.493383 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:15.493005    4496 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/013b67e2-aae9-4380-baf6-de93a6a9d18b-workload-socket\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:27:15.576962 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:15.576922    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b2e2446-f387-4ea4-8787-1fa5f47e18e6" path="/var/lib/kubelet/pods/6b2e2446-f387-4ea4-8787-1fa5f47e18e6/volumes"
Nov 02 03:27:15.577621 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:15.577601    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6aa56d2-87d0-4ba8-8b82-069d2d1d2acc" path="/var/lib/kubelet/pods/b6aa56d2-87d0-4ba8-8b82-069d2d1d2acc/volumes"
Nov 02 03:27:15.656069 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:15.655996    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-018cfa-94vgj"]
Nov 02 03:27:15.659510 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:15.659473    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-018cfa-94vgj"]
Nov 02 03:27:15.802565 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:15.802533    4496 generic.go:356] "Generic (PLEG): container finished" podID="013b67e2-aae9-4380-baf6-de93a6a9d18b" containerID="a5c6a4c4b834bd5d7091bedb7a209bfcd4b79b2694f89912363f29f1c05ee425" exitCode=0
Nov 02 03:27:15.802745 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:15.802584    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-t4dvl"
Nov 02 03:27:15.802745 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:15.802623    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-t4dvl" event={"ID":"013b67e2-aae9-4380-baf6-de93a6a9d18b","Type":"ContainerDied","Data":"a5c6a4c4b834bd5d7091bedb7a209bfcd4b79b2694f89912363f29f1c05ee425"}
Nov 02 03:27:15.802745 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:15.802678    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-t4dvl" event={"ID":"013b67e2-aae9-4380-baf6-de93a6a9d18b","Type":"ContainerDied","Data":"029092a660811fdfea4fcacc4ada00088314a01b5cf26e800be64cb1cb13c9bc"}
Nov 02 03:27:15.802745 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:15.802705    4496 scope.go:122] "RemoveContainer" containerID="a5c6a4c4b834bd5d7091bedb7a209bfcd4b79b2694f89912363f29f1c05ee425"
Nov 02 03:27:15.804253 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:15.804230    4496 generic.go:356] "Generic (PLEG): container finished" podID="90252c4b-9031-4994-86e1-9b7ae966cd7e" containerID="2340ff8ddfae8d2f582513548f272f8bbebb3fda69a164f7541e705420fac92e" exitCode=0
Nov 02 03:27:15.804363 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:15.804283    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-fn4nf"
Nov 02 03:27:15.804363 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:15.804314    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-fn4nf" event={"ID":"90252c4b-9031-4994-86e1-9b7ae966cd7e","Type":"ContainerDied","Data":"2340ff8ddfae8d2f582513548f272f8bbebb3fda69a164f7541e705420fac92e"}
Nov 02 03:27:15.804363 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:15.804357    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-fn4nf" event={"ID":"90252c4b-9031-4994-86e1-9b7ae966cd7e","Type":"ContainerDied","Data":"4d3d93136db023f3bf8fa611856c922f37533a1412b5deb894be4dd320080781"}
Nov 02 03:27:15.806251 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:15.806222    4496 generic.go:356] "Generic (PLEG): container finished" podID="e11f3625-7e4e-4124-a232-8cf67769fdfc" containerID="6af658c35919aae30a6a2946ea71e2f7fde491043366d64674a0c1636ab0a37e" exitCode=0
Nov 02 03:27:15.806376 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:15.806270    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-ff2fd"
Nov 02 03:27:15.806376 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:15.806299    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-ff2fd" event={"ID":"e11f3625-7e4e-4124-a232-8cf67769fdfc","Type":"ContainerDied","Data":"6af658c35919aae30a6a2946ea71e2f7fde491043366d64674a0c1636ab0a37e"}
Nov 02 03:27:15.806376 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:15.806343    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-ff2fd" event={"ID":"e11f3625-7e4e-4124-a232-8cf67769fdfc","Type":"ContainerDied","Data":"df84c563f28582af4cbc461f84626fb2b1adc27f01b87e9161c20ade8499c41a"}
Nov 02 03:27:15.814764 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:15.814749    4496 scope.go:122] "RemoveContainer" containerID="a5c6a4c4b834bd5d7091bedb7a209bfcd4b79b2694f89912363f29f1c05ee425"
Nov 02 03:27:15.815025 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:27:15.814995    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5c6a4c4b834bd5d7091bedb7a209bfcd4b79b2694f89912363f29f1c05ee425\": container with ID starting with a5c6a4c4b834bd5d7091bedb7a209bfcd4b79b2694f89912363f29f1c05ee425 not found: ID does not exist" containerID="a5c6a4c4b834bd5d7091bedb7a209bfcd4b79b2694f89912363f29f1c05ee425"
Nov 02 03:27:15.815081 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:15.815037    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5c6a4c4b834bd5d7091bedb7a209bfcd4b79b2694f89912363f29f1c05ee425"} err="failed to get container status \"a5c6a4c4b834bd5d7091bedb7a209bfcd4b79b2694f89912363f29f1c05ee425\": rpc error: code = NotFound desc = could not find container \"a5c6a4c4b834bd5d7091bedb7a209bfcd4b79b2694f89912363f29f1c05ee425\": container with ID starting with a5c6a4c4b834bd5d7091bedb7a209bfcd4b79b2694f89912363f29f1c05ee425 not found: ID does not exist"
Nov 02 03:27:15.815081 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:15.815058    4496 scope.go:122] "RemoveContainer" containerID="2340ff8ddfae8d2f582513548f272f8bbebb3fda69a164f7541e705420fac92e"
Nov 02 03:27:15.824297 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:15.824279    4496 scope.go:122] "RemoveContainer" containerID="2340ff8ddfae8d2f582513548f272f8bbebb3fda69a164f7541e705420fac92e"
Nov 02 03:27:15.824540 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:27:15.824524    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2340ff8ddfae8d2f582513548f272f8bbebb3fda69a164f7541e705420fac92e\": container with ID starting with 2340ff8ddfae8d2f582513548f272f8bbebb3fda69a164f7541e705420fac92e not found: ID does not exist" containerID="2340ff8ddfae8d2f582513548f272f8bbebb3fda69a164f7541e705420fac92e"
Nov 02 03:27:15.824593 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:15.824546    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2340ff8ddfae8d2f582513548f272f8bbebb3fda69a164f7541e705420fac92e"} err="failed to get container status \"2340ff8ddfae8d2f582513548f272f8bbebb3fda69a164f7541e705420fac92e\": rpc error: code = NotFound desc = could not find container \"2340ff8ddfae8d2f582513548f272f8bbebb3fda69a164f7541e705420fac92e\": container with ID starting with 2340ff8ddfae8d2f582513548f272f8bbebb3fda69a164f7541e705420fac92e not found: ID does not exist"
Nov 02 03:27:15.824593 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:15.824560    4496 scope.go:122] "RemoveContainer" containerID="6af658c35919aae30a6a2946ea71e2f7fde491043366d64674a0c1636ab0a37e"
Nov 02 03:27:15.833776 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:15.833758    4496 scope.go:122] "RemoveContainer" containerID="6af658c35919aae30a6a2946ea71e2f7fde491043366d64674a0c1636ab0a37e"
Nov 02 03:27:15.834073 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:27:15.834051    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6af658c35919aae30a6a2946ea71e2f7fde491043366d64674a0c1636ab0a37e\": container with ID starting with 6af658c35919aae30a6a2946ea71e2f7fde491043366d64674a0c1636ab0a37e not found: ID does not exist" containerID="6af658c35919aae30a6a2946ea71e2f7fde491043366d64674a0c1636ab0a37e"
Nov 02 03:27:15.834167 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:15.834076    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6af658c35919aae30a6a2946ea71e2f7fde491043366d64674a0c1636ab0a37e"} err="failed to get container status \"6af658c35919aae30a6a2946ea71e2f7fde491043366d64674a0c1636ab0a37e\": rpc error: code = NotFound desc = could not find container \"6af658c35919aae30a6a2946ea71e2f7fde491043366d64674a0c1636ab0a37e\": container with ID starting with 6af658c35919aae30a6a2946ea71e2f7fde491043366d64674a0c1636ab0a37e not found: ID does not exist"
Nov 02 03:27:15.975437 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:15.975407    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-t4dvl"]
Nov 02 03:27:15.978986 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:15.978957    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-t4dvl"]
Nov 02 03:27:16.091948 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:16.091917    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-ff2fd"]
Nov 02 03:27:16.095341 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:16.095319    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-ff2fd"]
Nov 02 03:27:16.210750 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:16.210712    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-fn4nf"]
Nov 02 03:27:16.218459 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:16.218426    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-fn4nf"]
Nov 02 03:27:17.575748 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:17.575712    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="013b67e2-aae9-4380-baf6-de93a6a9d18b" path="/var/lib/kubelet/pods/013b67e2-aae9-4380-baf6-de93a6a9d18b/volumes"
Nov 02 03:27:17.576434 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:17.576412    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90252c4b-9031-4994-86e1-9b7ae966cd7e" path="/var/lib/kubelet/pods/90252c4b-9031-4994-86e1-9b7ae966cd7e/volumes"
Nov 02 03:27:17.576887 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:17.576867    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d73da420-2bd5-41a2-b24b-e8601948fbd0" path="/var/lib/kubelet/pods/d73da420-2bd5-41a2-b24b-e8601948fbd0/volumes"
Nov 02 03:27:17.577679 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:17.577658    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e11f3625-7e4e-4124-a232-8cf67769fdfc" path="/var/lib/kubelet/pods/e11f3625-7e4e-4124-a232-8cf67769fdfc/volumes"
Nov 02 03:27:21.672000 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:21.671954    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c170e9-g26pq"]
Nov 02 03:27:21.672643 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:21.672623    4496 cpu_manager.go:414] "containerMap: removing container" podUID="013b67e2-aae9-4380-baf6-de93a6a9d18b" containerName="istio-proxy"
Nov 02 03:27:21.672643 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:21.672642    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="013b67e2-aae9-4380-baf6-de93a6a9d18b" containerName="istio-proxy"
Nov 02 03:27:21.672769 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:21.672664    4496 cpu_manager.go:414] "containerMap: removing container" podUID="020a5526-3ffd-46fb-8015-63d0e7657d37" containerName="payload-processing"
Nov 02 03:27:21.672769 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:21.672669    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="020a5526-3ffd-46fb-8015-63d0e7657d37" containerName="payload-processing"
Nov 02 03:27:21.672769 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:21.672681    4496 cpu_manager.go:414] "containerMap: removing container" podUID="3051f935-d87a-422e-9073-909b2c2768ac" containerName="istio-proxy"
Nov 02 03:27:21.672769 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:21.672686    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3051f935-d87a-422e-9073-909b2c2768ac" containerName="istio-proxy"
Nov 02 03:27:21.672769 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:21.672696    4496 cpu_manager.go:414] "containerMap: removing container" podUID="6b2e2446-f387-4ea4-8787-1fa5f47e18e6" containerName="istio-proxy"
Nov 02 03:27:21.672769 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:21.672700    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b2e2446-f387-4ea4-8787-1fa5f47e18e6" containerName="istio-proxy"
Nov 02 03:27:21.672769 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:21.672716    4496 cpu_manager.go:414] "containerMap: removing container" podUID="90252c4b-9031-4994-86e1-9b7ae966cd7e" containerName="istio-proxy"
Nov 02 03:27:21.672769 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:21.672721    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90252c4b-9031-4994-86e1-9b7ae966cd7e" containerName="istio-proxy"
Nov 02 03:27:21.672769 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:21.672729    4496 cpu_manager.go:414] "containerMap: removing container" podUID="a62ebf0f-f2fd-404d-aa26-6f57836432dd" containerName="revoke-keys"
Nov 02 03:27:21.672769 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:21.672733    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a62ebf0f-f2fd-404d-aa26-6f57836432dd" containerName="revoke-keys"
Nov 02 03:27:21.672769 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:21.672747    4496 cpu_manager.go:414] "containerMap: removing container" podUID="e11f3625-7e4e-4124-a232-8cf67769fdfc" containerName="istio-proxy"
Nov 02 03:27:21.672769 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:21.672752    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e11f3625-7e4e-4124-a232-8cf67769fdfc" containerName="istio-proxy"
Nov 02 03:27:21.672769 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:21.672760    4496 cpu_manager.go:414] "containerMap: removing container" podUID="938e8e17-80ba-4b93-9f96-29e68f73f927" containerName="istio-proxy"
Nov 02 03:27:21.672769 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:21.672767    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="938e8e17-80ba-4b93-9f96-29e68f73f927" containerName="istio-proxy"
Nov 02 03:27:21.673515 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:21.672782    4496 cpu_manager.go:414] "containerMap: removing container" podUID="0031b60a-f888-4c85-8999-07bddc61a513" containerName="istio-proxy"
Nov 02 03:27:21.673515 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:21.672787    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0031b60a-f888-4c85-8999-07bddc61a513" containerName="istio-proxy"
Nov 02 03:27:21.673515 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:21.672794    4496 cpu_manager.go:414] "containerMap: removing container" podUID="b8658350-dded-408d-9904-40b3403119c9" containerName="payload-pre-processing"
Nov 02 03:27:21.673515 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:21.672799    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8658350-dded-408d-9904-40b3403119c9" containerName="payload-pre-processing"
Nov 02 03:27:21.673515 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:21.672890    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="e11f3625-7e4e-4124-a232-8cf67769fdfc" containerName="istio-proxy"
Nov 02 03:27:21.673515 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:21.672897    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="938e8e17-80ba-4b93-9f96-29e68f73f927" containerName="istio-proxy"
Nov 02 03:27:21.673515 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:21.672903    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="0031b60a-f888-4c85-8999-07bddc61a513" containerName="istio-proxy"
Nov 02 03:27:21.673515 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:21.672910    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8658350-dded-408d-9904-40b3403119c9" containerName="payload-pre-processing"
Nov 02 03:27:21.673515 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:21.672918    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="013b67e2-aae9-4380-baf6-de93a6a9d18b" containerName="istio-proxy"
Nov 02 03:27:21.673515 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:21.672925    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="020a5526-3ffd-46fb-8015-63d0e7657d37" containerName="payload-processing"
Nov 02 03:27:21.673515 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:21.672931    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="3051f935-d87a-422e-9073-909b2c2768ac" containerName="istio-proxy"
Nov 02 03:27:21.673515 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:21.672936    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b2e2446-f387-4ea4-8787-1fa5f47e18e6" containerName="istio-proxy"
Nov 02 03:27:21.673515 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:21.672943    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="90252c4b-9031-4994-86e1-9b7ae966cd7e" containerName="istio-proxy"
Nov 02 03:27:21.673515 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:21.672948    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="a62ebf0f-f2fd-404d-aa26-6f57836432dd" containerName="revoke-keys"
Nov 02 03:27:21.678798 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:21.678776    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c170e9-g26pq"
Nov 02 03:27:21.736613 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:21.736575    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c170e9-g26pq"]
Nov 02 03:27:21.782275 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:21.782245    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 03:27:21.782489 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:21.782350    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 03:27:21.782489 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:21.782419    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 03:27:21.782489 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:21.782438    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9z5bk\""
Nov 02 03:27:21.802670 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:21.802635    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d5ed27d-1678-4b1e-bea9-969adb5902af-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c170e9-g26pq\" (UID: \"1d5ed27d-1678-4b1e-bea9-969adb5902af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c170e9-g26pq"
Nov 02 03:27:21.802804 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:21.802699    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d5ed27d-1678-4b1e-bea9-969adb5902af-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c170e9-g26pq\" (UID: \"1d5ed27d-1678-4b1e-bea9-969adb5902af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c170e9-g26pq"
Nov 02 03:27:21.802804 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:21.802765    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wngr\" (UniqueName: \"kubernetes.io/projected/1d5ed27d-1678-4b1e-bea9-969adb5902af-kube-api-access-8wngr\") pod \"maas-api-revoke-keys-e2e-api-b-c170e9-g26pq\" (UID: \"1d5ed27d-1678-4b1e-bea9-969adb5902af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c170e9-g26pq"
Nov 02 03:27:21.904093 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:21.904058    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wngr\" (UniqueName: \"kubernetes.io/projected/1d5ed27d-1678-4b1e-bea9-969adb5902af-kube-api-access-8wngr\") pod \"maas-api-revoke-keys-e2e-api-b-c170e9-g26pq\" (UID: \"1d5ed27d-1678-4b1e-bea9-969adb5902af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c170e9-g26pq"
Nov 02 03:27:21.904288 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:21.904241    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d5ed27d-1678-4b1e-bea9-969adb5902af-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c170e9-g26pq\" (UID: \"1d5ed27d-1678-4b1e-bea9-969adb5902af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c170e9-g26pq"
Nov 02 03:27:21.904288 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:21.904277    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d5ed27d-1678-4b1e-bea9-969adb5902af-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c170e9-g26pq\" (UID: \"1d5ed27d-1678-4b1e-bea9-969adb5902af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c170e9-g26pq"
Nov 02 03:27:21.904865 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:21.904840    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d5ed27d-1678-4b1e-bea9-969adb5902af-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c170e9-g26pq\" (UID: \"1d5ed27d-1678-4b1e-bea9-969adb5902af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c170e9-g26pq"
Nov 02 03:27:21.906634 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:21.906612    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d5ed27d-1678-4b1e-bea9-969adb5902af-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c170e9-g26pq\" (UID: \"1d5ed27d-1678-4b1e-bea9-969adb5902af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c170e9-g26pq"
Nov 02 03:27:21.962417 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:21.962337    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wngr\" (UniqueName: \"kubernetes.io/projected/1d5ed27d-1678-4b1e-bea9-969adb5902af-kube-api-access-8wngr\") pod \"maas-api-revoke-keys-e2e-api-b-c170e9-g26pq\" (UID: \"1d5ed27d-1678-4b1e-bea9-969adb5902af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c170e9-g26pq"
Nov 02 03:27:21.994855 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:21.994824    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c170e9-g26pq"
Nov 02 03:27:22.379607 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:22.379570    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c170e9-g26pq"]
Nov 02 03:27:22.383734 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:27:22.383695    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d5ed27d_1678_4b1e_bea9_969adb5902af.slice/crio-6776128a2aa87e3c0753ba9cf8e4c8f53c0fcec09682f92275b4770be5ff2661 WatchSource:0}: Error finding container 6776128a2aa87e3c0753ba9cf8e4c8f53c0fcec09682f92275b4770be5ff2661: Status 404 returned error can't find the container with id 6776128a2aa87e3c0753ba9cf8e4c8f53c0fcec09682f92275b4770be5ff2661
Nov 02 03:27:22.883191 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:22.883150    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c170e9-g26pq" event={"ID":"1d5ed27d-1678-4b1e-bea9-969adb5902af","Type":"ContainerStarted","Data":"0e3609de0d6b09ed48163dde2fcb03fdb6bcc2d7bf1a3debeb6a4dcfabb00caa"}
Nov 02 03:27:22.883574 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:22.883216    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c170e9-g26pq" event={"ID":"1d5ed27d-1678-4b1e-bea9-969adb5902af","Type":"ContainerStarted","Data":"6776128a2aa87e3c0753ba9cf8e4c8f53c0fcec09682f92275b4770be5ff2661"}
Nov 02 03:27:23.003200 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:23.003147    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c170e9-g26pq" podStartSLOduration=2.003132925 podStartE2EDuration="2.003132925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:27:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:27:22.999585444 +0000 UTC m=+2383.578192847" watchObservedRunningTime="2026-11-02 03:27:23.003132925 +0000 UTC m=+2383.581740342"
Nov 02 03:27:23.893090 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:23.893055    4496 generic.go:356] "Generic (PLEG): container finished" podID="1d5ed27d-1678-4b1e-bea9-969adb5902af" containerID="0e3609de0d6b09ed48163dde2fcb03fdb6bcc2d7bf1a3debeb6a4dcfabb00caa" exitCode=0
Nov 02 03:27:23.893491 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:23.893180    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c170e9-g26pq" event={"ID":"1d5ed27d-1678-4b1e-bea9-969adb5902af","Type":"ContainerDied","Data":"0e3609de0d6b09ed48163dde2fcb03fdb6bcc2d7bf1a3debeb6a4dcfabb00caa"}
Nov 02 03:27:25.064506 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:25.064480    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c170e9-g26pq"
Nov 02 03:27:25.153853 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:25.153775    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d5ed27d-1678-4b1e-bea9-969adb5902af-kube-api-access-8wngr\" (UniqueName: \"kubernetes.io/projected/1d5ed27d-1678-4b1e-bea9-969adb5902af-kube-api-access-8wngr\") pod \"1d5ed27d-1678-4b1e-bea9-969adb5902af\" (UID: \"1d5ed27d-1678-4b1e-bea9-969adb5902af\") "
Nov 02 03:27:25.154002 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:25.153941    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d5ed27d-1678-4b1e-bea9-969adb5902af-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d5ed27d-1678-4b1e-bea9-969adb5902af-maas-api-xks-ca\") pod \"1d5ed27d-1678-4b1e-bea9-969adb5902af\" (UID: \"1d5ed27d-1678-4b1e-bea9-969adb5902af\") "
Nov 02 03:27:25.154002 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:25.153990    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d5ed27d-1678-4b1e-bea9-969adb5902af-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d5ed27d-1678-4b1e-bea9-969adb5902af-maas-api-openshift-ca\") pod \"1d5ed27d-1678-4b1e-bea9-969adb5902af\" (UID: \"1d5ed27d-1678-4b1e-bea9-969adb5902af\") "
Nov 02 03:27:25.154385 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:25.154357    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d5ed27d-1678-4b1e-bea9-969adb5902af-maas-api-openshift-ca" pod "1d5ed27d-1678-4b1e-bea9-969adb5902af" (UID: "1d5ed27d-1678-4b1e-bea9-969adb5902af"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:27:25.156099 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:25.156073    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d5ed27d-1678-4b1e-bea9-969adb5902af-maas-api-xks-ca" pod "1d5ed27d-1678-4b1e-bea9-969adb5902af" (UID: "1d5ed27d-1678-4b1e-bea9-969adb5902af"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 03:27:25.156195 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:25.156161    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d5ed27d-1678-4b1e-bea9-969adb5902af-kube-api-access-8wngr" pod "1d5ed27d-1678-4b1e-bea9-969adb5902af" (UID: "1d5ed27d-1678-4b1e-bea9-969adb5902af"). InnerVolumeSpecName "kube-api-access-8wngr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:27:25.256048 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:25.255983    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d5ed27d-1678-4b1e-bea9-969adb5902af-maas-api-xks-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:27:25.256048 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:25.256048    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d5ed27d-1678-4b1e-bea9-969adb5902af-maas-api-openshift-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:27:25.256239 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:25.256068    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wngr\" (UniqueName: \"kubernetes.io/projected/1d5ed27d-1678-4b1e-bea9-969adb5902af-kube-api-access-8wngr\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:27:25.912079 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:25.911979    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c170e9-g26pq"
Nov 02 03:27:25.912079 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:25.911980    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c170e9-g26pq" event={"ID":"1d5ed27d-1678-4b1e-bea9-969adb5902af","Type":"ContainerDied","Data":"6776128a2aa87e3c0753ba9cf8e4c8f53c0fcec09682f92275b4770be5ff2661"}
Nov 02 03:27:25.912250 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:25.912086    4496 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6776128a2aa87e3c0753ba9cf8e4c8f53c0fcec09682f92275b4770be5ff2661"
Nov 02 03:27:29.693714 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:29.693680    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4bf04623-hfwcl"]
Nov 02 03:27:29.694299 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:29.694284    4496 cpu_manager.go:414] "containerMap: removing container" podUID="1d5ed27d-1678-4b1e-bea9-969adb5902af" containerName="revoke-keys"
Nov 02 03:27:29.694345 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:29.694300    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d5ed27d-1678-4b1e-bea9-969adb5902af" containerName="revoke-keys"
Nov 02 03:27:29.694406 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:29.694397    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d5ed27d-1678-4b1e-bea9-969adb5902af" containerName="revoke-keys"
Nov 02 03:27:29.697977 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:29.697961    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bf04623-hfwcl"
Nov 02 03:27:29.754930 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:29.754903    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4bf04623-hfwcl"]
Nov 02 03:27:29.800406 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:29.800381    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 03:27:29.800548 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:29.800494    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 03:27:29.800548 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:29.800498    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 03:27:29.800634 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:29.800545    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9z5bk\""
Nov 02 03:27:29.811103 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:29.811083    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96ab300a-48eb-4c1e-9c97-db8e6cec62c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4bf04623-hfwcl\" (UID: \"96ab300a-48eb-4c1e-9c97-db8e6cec62c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bf04623-hfwcl"
Nov 02 03:27:29.811164 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:29.811118    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96ab300a-48eb-4c1e-9c97-db8e6cec62c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4bf04623-hfwcl\" (UID: \"96ab300a-48eb-4c1e-9c97-db8e6cec62c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bf04623-hfwcl"
Nov 02 03:27:29.811267 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:29.811250    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwzkf\" (UniqueName: \"kubernetes.io/projected/96ab300a-48eb-4c1e-9c97-db8e6cec62c0-kube-api-access-zwzkf\") pod \"maas-api-revoke-sub-4bf04623-hfwcl\" (UID: \"96ab300a-48eb-4c1e-9c97-db8e6cec62c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bf04623-hfwcl"
Nov 02 03:27:29.912453 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:29.912416    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96ab300a-48eb-4c1e-9c97-db8e6cec62c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4bf04623-hfwcl\" (UID: \"96ab300a-48eb-4c1e-9c97-db8e6cec62c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bf04623-hfwcl"
Nov 02 03:27:29.912646 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:29.912479    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96ab300a-48eb-4c1e-9c97-db8e6cec62c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4bf04623-hfwcl\" (UID: \"96ab300a-48eb-4c1e-9c97-db8e6cec62c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bf04623-hfwcl"
Nov 02 03:27:29.912646 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:29.912601    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwzkf\" (UniqueName: \"kubernetes.io/projected/96ab300a-48eb-4c1e-9c97-db8e6cec62c0-kube-api-access-zwzkf\") pod \"maas-api-revoke-sub-4bf04623-hfwcl\" (UID: \"96ab300a-48eb-4c1e-9c97-db8e6cec62c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bf04623-hfwcl"
Nov 02 03:27:29.913095 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:29.913074    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96ab300a-48eb-4c1e-9c97-db8e6cec62c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4bf04623-hfwcl\" (UID: \"96ab300a-48eb-4c1e-9c97-db8e6cec62c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bf04623-hfwcl"
Nov 02 03:27:29.914917 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:29.914892    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96ab300a-48eb-4c1e-9c97-db8e6cec62c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4bf04623-hfwcl\" (UID: \"96ab300a-48eb-4c1e-9c97-db8e6cec62c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bf04623-hfwcl"
Nov 02 03:27:29.970289 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:29.970226    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwzkf\" (UniqueName: \"kubernetes.io/projected/96ab300a-48eb-4c1e-9c97-db8e6cec62c0-kube-api-access-zwzkf\") pod \"maas-api-revoke-sub-4bf04623-hfwcl\" (UID: \"96ab300a-48eb-4c1e-9c97-db8e6cec62c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bf04623-hfwcl"
Nov 02 03:27:30.012399 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:30.012370    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bf04623-hfwcl"
Nov 02 03:27:30.392324 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:30.392294    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4bf04623-hfwcl"]
Nov 02 03:27:30.392801 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:27:30.392776    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96ab300a_48eb_4c1e_9c97_db8e6cec62c0.slice/crio-aa83e73b68337690719f53b014227bb44d4392f4ad9e5d30b68995d561ee1969 WatchSource:0}: Error finding container aa83e73b68337690719f53b014227bb44d4392f4ad9e5d30b68995d561ee1969: Status 404 returned error can't find the container with id aa83e73b68337690719f53b014227bb44d4392f4ad9e5d30b68995d561ee1969
Nov 02 03:27:30.952167 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:30.952080    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bf04623-hfwcl" event={"ID":"96ab300a-48eb-4c1e-9c97-db8e6cec62c0","Type":"ContainerStarted","Data":"4af1dd372de685837c767002e08d69eea3521d167d4f7f2768d7a62ea6e121f0"}
Nov 02 03:27:30.952167 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:30.952152    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bf04623-hfwcl" event={"ID":"96ab300a-48eb-4c1e-9c97-db8e6cec62c0","Type":"ContainerStarted","Data":"aa83e73b68337690719f53b014227bb44d4392f4ad9e5d30b68995d561ee1969"}
Nov 02 03:27:31.067889 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:31.067831    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bf04623-hfwcl" podStartSLOduration=2.067816484 podStartE2EDuration="2.067816484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:27:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:27:31.066876706 +0000 UTC m=+2391.645484109" watchObservedRunningTime="2026-11-02 03:27:31.067816484 +0000 UTC m=+2391.646423864"
Nov 02 03:27:31.961462 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:31.961428    4496 generic.go:356] "Generic (PLEG): container finished" podID="96ab300a-48eb-4c1e-9c97-db8e6cec62c0" containerID="4af1dd372de685837c767002e08d69eea3521d167d4f7f2768d7a62ea6e121f0" exitCode=0
Nov 02 03:27:31.961970 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:31.961509    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bf04623-hfwcl" event={"ID":"96ab300a-48eb-4c1e-9c97-db8e6cec62c0","Type":"ContainerDied","Data":"4af1dd372de685837c767002e08d69eea3521d167d4f7f2768d7a62ea6e121f0"}
Nov 02 03:27:33.146900 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:33.146869    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c170e9-67fcf76c9-f4kkt"]
Nov 02 03:27:33.157137 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:33.157113    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bf04623-hfwcl"
Nov 02 03:27:33.255691 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:33.255655    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96ab300a-48eb-4c1e-9c97-db8e6cec62c0-kube-api-access-zwzkf\" (UniqueName: \"kubernetes.io/projected/96ab300a-48eb-4c1e-9c97-db8e6cec62c0-kube-api-access-zwzkf\") pod \"96ab300a-48eb-4c1e-9c97-db8e6cec62c0\" (UID: \"96ab300a-48eb-4c1e-9c97-db8e6cec62c0\") "
Nov 02 03:27:33.256202 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:33.256174    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96ab300a-48eb-4c1e-9c97-db8e6cec62c0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96ab300a-48eb-4c1e-9c97-db8e6cec62c0-maas-api-xks-ca\") pod \"96ab300a-48eb-4c1e-9c97-db8e6cec62c0\" (UID: \"96ab300a-48eb-4c1e-9c97-db8e6cec62c0\") "
Nov 02 03:27:33.256484 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:33.256459    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96ab300a-48eb-4c1e-9c97-db8e6cec62c0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96ab300a-48eb-4c1e-9c97-db8e6cec62c0-maas-api-openshift-ca\") pod \"96ab300a-48eb-4c1e-9c97-db8e6cec62c0\" (UID: \"96ab300a-48eb-4c1e-9c97-db8e6cec62c0\") "
Nov 02 03:27:33.256832 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:33.256808    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96ab300a-48eb-4c1e-9c97-db8e6cec62c0-maas-api-openshift-ca" pod "96ab300a-48eb-4c1e-9c97-db8e6cec62c0" (UID: "96ab300a-48eb-4c1e-9c97-db8e6cec62c0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:27:33.257076 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:33.257010    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96ab300a-48eb-4c1e-9c97-db8e6cec62c0-maas-api-openshift-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:27:33.258271 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:33.258246    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96ab300a-48eb-4c1e-9c97-db8e6cec62c0-kube-api-access-zwzkf" pod "96ab300a-48eb-4c1e-9c97-db8e6cec62c0" (UID: "96ab300a-48eb-4c1e-9c97-db8e6cec62c0"). InnerVolumeSpecName "kube-api-access-zwzkf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:27:33.258377 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:33.258351    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96ab300a-48eb-4c1e-9c97-db8e6cec62c0-maas-api-xks-ca" pod "96ab300a-48eb-4c1e-9c97-db8e6cec62c0" (UID: "96ab300a-48eb-4c1e-9c97-db8e6cec62c0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 03:27:33.360089 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:33.359887    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96ab300a-48eb-4c1e-9c97-db8e6cec62c0-maas-api-xks-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:27:33.360089 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:33.359929    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwzkf\" (UniqueName: \"kubernetes.io/projected/96ab300a-48eb-4c1e-9c97-db8e6cec62c0-kube-api-access-zwzkf\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:27:33.528079 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:27:33.527981    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 03:27:33.983614 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:33.983578    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bf04623-hfwcl"
Nov 02 03:27:33.983796 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:33.983614    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bf04623-hfwcl" event={"ID":"96ab300a-48eb-4c1e-9c97-db8e6cec62c0","Type":"ContainerDied","Data":"aa83e73b68337690719f53b014227bb44d4392f4ad9e5d30b68995d561ee1969"}
Nov 02 03:27:33.983796 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:33.983666    4496 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa83e73b68337690719f53b014227bb44d4392f4ad9e5d30b68995d561ee1969"
Nov 02 03:27:38.147781 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:38.147739    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-c170e9-67fcf76c9-f4kkt" podUID="eaa7c285-859b-4a5c-9f04-cc38b3164f2e" containerName="payload-processing" containerID="cri-o://6c8ed99d61ccd8c64e8649ce66d757394a6c159fc0a91d0b252ec12a18d7bfa1" gracePeriod=25
Nov 02 03:27:38.421850 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:38.421826    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-c170e9-67fcf76c9-f4kkt"
Nov 02 03:27:38.545766 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:38.545729    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eaa7c285-859b-4a5c-9f04-cc38b3164f2e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eaa7c285-859b-4a5c-9f04-cc38b3164f2e-plugins-config-volume\") pod \"eaa7c285-859b-4a5c-9f04-cc38b3164f2e\" (UID: \"eaa7c285-859b-4a5c-9f04-cc38b3164f2e\") "
Nov 02 03:27:38.545936 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:38.545858    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eaa7c285-859b-4a5c-9f04-cc38b3164f2e-kube-api-access-lhr4z\" (UniqueName: \"kubernetes.io/projected/eaa7c285-859b-4a5c-9f04-cc38b3164f2e-kube-api-access-lhr4z\") pod \"eaa7c285-859b-4a5c-9f04-cc38b3164f2e\" (UID: \"eaa7c285-859b-4a5c-9f04-cc38b3164f2e\") "
Nov 02 03:27:38.546786 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:38.546485    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eaa7c285-859b-4a5c-9f04-cc38b3164f2e-plugins-config-volume" pod "eaa7c285-859b-4a5c-9f04-cc38b3164f2e" (UID: "eaa7c285-859b-4a5c-9f04-cc38b3164f2e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:27:38.550079 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:38.548546    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eaa7c285-859b-4a5c-9f04-cc38b3164f2e-kube-api-access-lhr4z" pod "eaa7c285-859b-4a5c-9f04-cc38b3164f2e" (UID: "eaa7c285-859b-4a5c-9f04-cc38b3164f2e"). InnerVolumeSpecName "kube-api-access-lhr4z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:27:38.557605 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:38.557549    4496 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eaa7c285-859b-4a5c-9f04-cc38b3164f2e-plugins-config-volume\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:27:38.557605 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:38.557581    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhr4z\" (UniqueName: \"kubernetes.io/projected/eaa7c285-859b-4a5c-9f04-cc38b3164f2e-kube-api-access-lhr4z\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:27:39.034865 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:39.034827    4496 generic.go:356] "Generic (PLEG): container finished" podID="eaa7c285-859b-4a5c-9f04-cc38b3164f2e" containerID="6c8ed99d61ccd8c64e8649ce66d757394a6c159fc0a91d0b252ec12a18d7bfa1" exitCode=0
Nov 02 03:27:39.035212 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:39.034904    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-c170e9-67fcf76c9-f4kkt"
Nov 02 03:27:39.035336 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:39.034911    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-c170e9-67fcf76c9-f4kkt" event={"ID":"eaa7c285-859b-4a5c-9f04-cc38b3164f2e","Type":"ContainerDied","Data":"6c8ed99d61ccd8c64e8649ce66d757394a6c159fc0a91d0b252ec12a18d7bfa1"}
Nov 02 03:27:39.035389 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:39.035366    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-c170e9-67fcf76c9-f4kkt" event={"ID":"eaa7c285-859b-4a5c-9f04-cc38b3164f2e","Type":"ContainerDied","Data":"dbcda1d1bd69c317a59a82cdf7f81c4ecfa9079864275e87aab2b8fe8765a3ae"}
Nov 02 03:27:39.035444 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:39.035414    4496 scope.go:122] "RemoveContainer" containerID="6c8ed99d61ccd8c64e8649ce66d757394a6c159fc0a91d0b252ec12a18d7bfa1"
Nov 02 03:27:39.048225 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:39.048200    4496 scope.go:122] "RemoveContainer" containerID="6c8ed99d61ccd8c64e8649ce66d757394a6c159fc0a91d0b252ec12a18d7bfa1"
Nov 02 03:27:39.048562 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:27:39.048537    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c8ed99d61ccd8c64e8649ce66d757394a6c159fc0a91d0b252ec12a18d7bfa1\": container with ID starting with 6c8ed99d61ccd8c64e8649ce66d757394a6c159fc0a91d0b252ec12a18d7bfa1 not found: ID does not exist" containerID="6c8ed99d61ccd8c64e8649ce66d757394a6c159fc0a91d0b252ec12a18d7bfa1"
Nov 02 03:27:39.048672 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:39.048569    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c8ed99d61ccd8c64e8649ce66d757394a6c159fc0a91d0b252ec12a18d7bfa1"} err="failed to get container status \"6c8ed99d61ccd8c64e8649ce66d757394a6c159fc0a91d0b252ec12a18d7bfa1\": rpc error: code = NotFound desc = could not find container \"6c8ed99d61ccd8c64e8649ce66d757394a6c159fc0a91d0b252ec12a18d7bfa1\": container with ID starting with 6c8ed99d61ccd8c64e8649ce66d757394a6c159fc0a91d0b252ec12a18d7bfa1 not found: ID does not exist"
Nov 02 03:27:39.205388 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:39.205349    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c170e9-67fcf76c9-f4kkt"]
Nov 02 03:27:39.255645 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:39.255600    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c170e9-67fcf76c9-f4kkt"]
Nov 02 03:27:39.577622 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:39.577591    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eaa7c285-859b-4a5c-9f04-cc38b3164f2e" path="/var/lib/kubelet/pods/eaa7c285-859b-4a5c-9f04-cc38b3164f2e/volumes"
Nov 02 03:27:40.092089 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:40.092055    4496 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hww8q_5c797b08-a314-44ed-8026-8a0ce1f6c36f/ovn-acl-logging/0.log"
Nov 02 03:27:40.092646 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:40.092620    4496 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hww8q_5c797b08-a314-44ed-8026-8a0ce1f6c36f/ovn-acl-logging/0.log"
Nov 02 03:27:41.037080 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:41.037044    4496 scope.go:122] "RemoveContainer" containerID="f6efcf46d08e8aa21bfd3e6a3aa4e7e581de9bc152aac5619021018109eed5a4"
Nov 02 03:27:46.934717 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:46.934679    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a27da5-wsjdx"]
Nov 02 03:27:46.985290 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:46.985249    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a27da5-wsjdx"]
Nov 02 03:27:47.577872 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:47.577834    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a62ebf0f-f2fd-404d-aa26-6f57836432dd" path="/var/lib/kubelet/pods/a62ebf0f-f2fd-404d-aa26-6f57836432dd/volumes"
Nov 02 03:27:57.149877 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:57.149843    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6qhq2"]
Nov 02 03:27:57.150544 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:57.150502    4496 cpu_manager.go:414] "containerMap: removing container" podUID="eaa7c285-859b-4a5c-9f04-cc38b3164f2e" containerName="payload-processing"
Nov 02 03:27:57.150544 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:57.150522    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eaa7c285-859b-4a5c-9f04-cc38b3164f2e" containerName="payload-processing"
Nov 02 03:27:57.150544 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:57.150532    4496 cpu_manager.go:414] "containerMap: removing container" podUID="96ab300a-48eb-4c1e-9c97-db8e6cec62c0" containerName="revoke-keys"
Nov 02 03:27:57.150544 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:57.150537    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96ab300a-48eb-4c1e-9c97-db8e6cec62c0" containerName="revoke-keys"
Nov 02 03:27:57.150702 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:57.150634    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="eaa7c285-859b-4a5c-9f04-cc38b3164f2e" containerName="payload-processing"
Nov 02 03:27:57.150702 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:57.150642    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="96ab300a-48eb-4c1e-9c97-db8e6cec62c0" containerName="revoke-keys"
Nov 02 03:27:57.156280 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:57.156257    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6qhq2"
Nov 02 03:27:57.295707 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:57.295669    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/175952d1-62ed-4ea6-85d0-37d1a651f769-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-6qhq2\" (UID: \"175952d1-62ed-4ea6-85d0-37d1a651f769\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6qhq2"
Nov 02 03:27:57.295876 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:57.295810    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/175952d1-62ed-4ea6-85d0-37d1a651f769-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-6qhq2\" (UID: \"175952d1-62ed-4ea6-85d0-37d1a651f769\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6qhq2"
Nov 02 03:27:57.295876 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:57.295856    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/175952d1-62ed-4ea6-85d0-37d1a651f769-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-6qhq2\" (UID: \"175952d1-62ed-4ea6-85d0-37d1a651f769\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6qhq2"
Nov 02 03:27:57.296050 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:57.295876    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/175952d1-62ed-4ea6-85d0-37d1a651f769-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-6qhq2\" (UID: \"175952d1-62ed-4ea6-85d0-37d1a651f769\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6qhq2"
Nov 02 03:27:57.296151 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:57.296115    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/175952d1-62ed-4ea6-85d0-37d1a651f769-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-6qhq2\" (UID: \"175952d1-62ed-4ea6-85d0-37d1a651f769\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6qhq2"
Nov 02 03:27:57.296226 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:57.296194    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/175952d1-62ed-4ea6-85d0-37d1a651f769-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-6qhq2\" (UID: \"175952d1-62ed-4ea6-85d0-37d1a651f769\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6qhq2"
Nov 02 03:27:57.296310 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:57.296291    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/175952d1-62ed-4ea6-85d0-37d1a651f769-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-6qhq2\" (UID: \"175952d1-62ed-4ea6-85d0-37d1a651f769\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6qhq2"
Nov 02 03:27:57.296377 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:57.296366    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkccw\" (UniqueName: \"kubernetes.io/projected/175952d1-62ed-4ea6-85d0-37d1a651f769-kube-api-access-vkccw\") pod \"maas-default-gateway-openshift-default-58847f9fc5-6qhq2\" (UID: \"175952d1-62ed-4ea6-85d0-37d1a651f769\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6qhq2"
Nov 02 03:27:57.296452 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:57.296441    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/175952d1-62ed-4ea6-85d0-37d1a651f769-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-6qhq2\" (UID: \"175952d1-62ed-4ea6-85d0-37d1a651f769\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6qhq2"
Nov 02 03:27:57.356752 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:57.356712    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6qhq2"]
Nov 02 03:27:57.405534 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:57.405434    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/175952d1-62ed-4ea6-85d0-37d1a651f769-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-6qhq2\" (UID: \"175952d1-62ed-4ea6-85d0-37d1a651f769\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6qhq2"
Nov 02 03:27:57.405534 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:57.405484    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/175952d1-62ed-4ea6-85d0-37d1a651f769-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-6qhq2\" (UID: \"175952d1-62ed-4ea6-85d0-37d1a651f769\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6qhq2"
Nov 02 03:27:57.405868 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:57.405564    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/175952d1-62ed-4ea6-85d0-37d1a651f769-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-6qhq2\" (UID: \"175952d1-62ed-4ea6-85d0-37d1a651f769\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6qhq2"
Nov 02 03:27:57.405868 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:57.405614    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/175952d1-62ed-4ea6-85d0-37d1a651f769-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-6qhq2\" (UID: \"175952d1-62ed-4ea6-85d0-37d1a651f769\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6qhq2"
Nov 02 03:27:57.405868 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:57.405678    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/175952d1-62ed-4ea6-85d0-37d1a651f769-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-6qhq2\" (UID: \"175952d1-62ed-4ea6-85d0-37d1a651f769\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6qhq2"
Nov 02 03:27:57.405868 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:57.405776    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkccw\" (UniqueName: \"kubernetes.io/projected/175952d1-62ed-4ea6-85d0-37d1a651f769-kube-api-access-vkccw\") pod \"maas-default-gateway-openshift-default-58847f9fc5-6qhq2\" (UID: \"175952d1-62ed-4ea6-85d0-37d1a651f769\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6qhq2"
Nov 02 03:27:57.405868 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:57.405856    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/175952d1-62ed-4ea6-85d0-37d1a651f769-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-6qhq2\" (UID: \"175952d1-62ed-4ea6-85d0-37d1a651f769\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6qhq2"
Nov 02 03:27:57.406222 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:57.405890    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/175952d1-62ed-4ea6-85d0-37d1a651f769-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-6qhq2\" (UID: \"175952d1-62ed-4ea6-85d0-37d1a651f769\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6qhq2"
Nov 02 03:27:57.406222 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:57.405906    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/175952d1-62ed-4ea6-85d0-37d1a651f769-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-6qhq2\" (UID: \"175952d1-62ed-4ea6-85d0-37d1a651f769\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6qhq2"
Nov 02 03:27:57.406569 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:57.406537    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/175952d1-62ed-4ea6-85d0-37d1a651f769-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-6qhq2\" (UID: \"175952d1-62ed-4ea6-85d0-37d1a651f769\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6qhq2"
Nov 02 03:27:57.406832 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:57.406719    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/175952d1-62ed-4ea6-85d0-37d1a651f769-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-6qhq2\" (UID: \"175952d1-62ed-4ea6-85d0-37d1a651f769\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6qhq2"
Nov 02 03:27:57.407530 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:57.407490    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/175952d1-62ed-4ea6-85d0-37d1a651f769-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-6qhq2\" (UID: \"175952d1-62ed-4ea6-85d0-37d1a651f769\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6qhq2"
Nov 02 03:27:57.407862 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:57.407842    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/175952d1-62ed-4ea6-85d0-37d1a651f769-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-6qhq2\" (UID: \"175952d1-62ed-4ea6-85d0-37d1a651f769\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6qhq2"
Nov 02 03:27:57.407959 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:57.407844    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/175952d1-62ed-4ea6-85d0-37d1a651f769-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-6qhq2\" (UID: \"175952d1-62ed-4ea6-85d0-37d1a651f769\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6qhq2"
Nov 02 03:27:57.409509 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:57.409453    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/175952d1-62ed-4ea6-85d0-37d1a651f769-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-6qhq2\" (UID: \"175952d1-62ed-4ea6-85d0-37d1a651f769\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6qhq2"
Nov 02 03:27:57.410894 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:57.410868    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/175952d1-62ed-4ea6-85d0-37d1a651f769-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-6qhq2\" (UID: \"175952d1-62ed-4ea6-85d0-37d1a651f769\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6qhq2"
Nov 02 03:27:57.464144 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:57.464112    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/175952d1-62ed-4ea6-85d0-37d1a651f769-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-6qhq2\" (UID: \"175952d1-62ed-4ea6-85d0-37d1a651f769\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6qhq2"
Nov 02 03:27:57.464402 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:57.464382    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkccw\" (UniqueName: \"kubernetes.io/projected/175952d1-62ed-4ea6-85d0-37d1a651f769-kube-api-access-vkccw\") pod \"maas-default-gateway-openshift-default-58847f9fc5-6qhq2\" (UID: \"175952d1-62ed-4ea6-85d0-37d1a651f769\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6qhq2"
Nov 02 03:27:57.470404 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:57.470376    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6qhq2"
Nov 02 03:27:57.902973 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:57.902940    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6qhq2"]
Nov 02 03:27:57.904642 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:27:57.904611    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod175952d1_62ed_4ea6_85d0_37d1a651f769.slice/crio-3370b34b28d613dfe4ec647b3cbeff989dff57991ae0057b26bffbec6dff5dff WatchSource:0}: Error finding container 3370b34b28d613dfe4ec647b3cbeff989dff57991ae0057b26bffbec6dff5dff: Status 404 returned error can't find the container with id 3370b34b28d613dfe4ec647b3cbeff989dff57991ae0057b26bffbec6dff5dff
Nov 02 03:27:57.907604 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:57.907568    4496 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 03:27:57.907695 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:57.907638    4496 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 03:27:57.907695 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:57.907672    4496 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 03:27:58.228740 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:58.228709    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6qhq2" event={"ID":"175952d1-62ed-4ea6-85d0-37d1a651f769","Type":"ContainerStarted","Data":"d732676fda6c0553e54381592d74a6114f6870ce3d6374678e041e1fc3cae7b1"}
Nov 02 03:27:58.229201 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:58.228761    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6qhq2" event={"ID":"175952d1-62ed-4ea6-85d0-37d1a651f769","Type":"ContainerStarted","Data":"3370b34b28d613dfe4ec647b3cbeff989dff57991ae0057b26bffbec6dff5dff"}
Nov 02 03:27:58.349844 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:58.349783    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6qhq2" podStartSLOduration=1.349766263 podStartE2EDuration="1.349766263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:27:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:27:58.346156741 +0000 UTC m=+2418.924764123" watchObservedRunningTime="2026-11-02 03:27:58.349766263 +0000 UTC m=+2418.928373644"
Nov 02 03:27:58.474040 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:58.473987    4496 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6qhq2"
Nov 02 03:27:58.476039 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:58.475130    4496 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-6qhq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body=
Nov 02 03:27:58.476039 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:58.475185    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6qhq2" podUID="175952d1-62ed-4ea6-85d0-37d1a651f769" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused"
Nov 02 03:27:59.471624 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:59.471589    4496 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-6qhq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body=
Nov 02 03:27:59.472044 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:27:59.471643    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6qhq2" podUID="175952d1-62ed-4ea6-85d0-37d1a651f769" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused"
Nov 02 03:28:00.471522 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:28:00.471487    4496 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-6qhq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body=
Nov 02 03:28:00.471700 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:28:00.471541    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6qhq2" podUID="175952d1-62ed-4ea6-85d0-37d1a651f769" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused"
Nov 02 03:28:01.471169 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:28:01.471132    4496 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-6qhq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body=
Nov 02 03:28:01.471353 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:28:01.471181    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6qhq2" podUID="175952d1-62ed-4ea6-85d0-37d1a651f769" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused"
Nov 02 03:28:02.471003 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:28:02.470969    4496 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-6qhq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body=
Nov 02 03:28:02.471404 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:28:02.471036    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6qhq2" podUID="175952d1-62ed-4ea6-85d0-37d1a651f769" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused"
Nov 02 03:28:03.471501 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:28:03.471467    4496 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-6qhq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body=
Nov 02 03:28:03.471889 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:28:03.471530    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6qhq2" podUID="175952d1-62ed-4ea6-85d0-37d1a651f769" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused"
Nov 02 03:28:03.603037 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:28:03.602202    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 03:28:04.471350 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:28:04.471315    4496 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-6qhq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body=
Nov 02 03:28:04.471547 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:28:04.471391    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6qhq2" podUID="175952d1-62ed-4ea6-85d0-37d1a651f769" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused"
Nov 02 03:28:05.471495 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:28:05.471453    4496 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-6qhq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body=
Nov 02 03:28:05.471672 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:28:05.471522    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6qhq2" podUID="175952d1-62ed-4ea6-85d0-37d1a651f769" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused"
Nov 02 03:28:06.476876 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:28:06.476840    4496 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6qhq2"
Nov 02 03:28:06.477519 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:28:06.477495    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6qhq2"
Nov 02 03:28:06.478319 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:28:06.478295    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6qhq2"
Nov 02 03:28:33.527003 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:28:33.526957    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 03:28:39.832111 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:28:39.832035    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c170e9-g26pq"]
Nov 02 03:28:39.880660 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:28:39.880634    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c170e9-g26pq"]
Nov 02 03:28:41.210827 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:28:41.210796    4496 scope.go:122] "RemoveContainer" containerID="d5afad9ae36dd33ee75c1a364c7cc5f01dc7bca87bea0e0047bac619b1c6bc40"
Nov 02 03:28:41.574741 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:28:41.574704    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d5ed27d-1678-4b1e-bea9-969adb5902af" path="/var/lib/kubelet/pods/1d5ed27d-1678-4b1e-bea9-969adb5902af/volumes"
Nov 02 03:28:45.779540 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:28:45.779500    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-kj8qb"]
Nov 02 03:28:45.780059 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:28:45.779780    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-kj8qb" podUID="a41eaed2-344b-405c-86cb-631055ace710" containerName="istio-proxy" containerID="cri-o://e8f062428768b2fd4e15c6b5823e3b230b07f39789d001da76df900c686c6d21" gracePeriod=30
Nov 02 03:28:49.591087 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:28:49.591055    4496 patch_prober.go:29] interesting pod/e2e-worker-w4-398828-openshift-default-5647988965-kj8qb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 03:28:49.591463 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:28:49.591120    4496 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-kj8qb" podUID="a41eaed2-344b-405c-86cb-631055ace710" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 03:28:51.051984 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:28:51.051918    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-kj8qb"
Nov 02 03:28:51.115531 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:28:51.115498    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a41eaed2-344b-405c-86cb-631055ace710-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a41eaed2-344b-405c-86cb-631055ace710-workload-certs\") pod \"a41eaed2-344b-405c-86cb-631055ace710\" (UID: \"a41eaed2-344b-405c-86cb-631055ace710\") "
Nov 02 03:28:51.115531 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:28:51.115535    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a41eaed2-344b-405c-86cb-631055ace710-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a41eaed2-344b-405c-86cb-631055ace710-istio-podinfo\") pod \"a41eaed2-344b-405c-86cb-631055ace710\" (UID: \"a41eaed2-344b-405c-86cb-631055ace710\") "
Nov 02 03:28:51.115975 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:28:51.115602    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a41eaed2-344b-405c-86cb-631055ace710-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a41eaed2-344b-405c-86cb-631055ace710-istiod-ca-cert\") pod \"a41eaed2-344b-405c-86cb-631055ace710\" (UID: \"a41eaed2-344b-405c-86cb-631055ace710\") "
Nov 02 03:28:51.115975 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:28:51.115665    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a41eaed2-344b-405c-86cb-631055ace710-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a41eaed2-344b-405c-86cb-631055ace710-workload-socket\") pod \"a41eaed2-344b-405c-86cb-631055ace710\" (UID: \"a41eaed2-344b-405c-86cb-631055ace710\") "
Nov 02 03:28:51.115975 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:28:51.115740    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a41eaed2-344b-405c-86cb-631055ace710-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a41eaed2-344b-405c-86cb-631055ace710-istio-data\") pod \"a41eaed2-344b-405c-86cb-631055ace710\" (UID: \"a41eaed2-344b-405c-86cb-631055ace710\") "
Nov 02 03:28:51.115975 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:28:51.115771    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a41eaed2-344b-405c-86cb-631055ace710-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a41eaed2-344b-405c-86cb-631055ace710-credential-socket\") pod \"a41eaed2-344b-405c-86cb-631055ace710\" (UID: \"a41eaed2-344b-405c-86cb-631055ace710\") "
Nov 02 03:28:51.115975 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:28:51.115845    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a41eaed2-344b-405c-86cb-631055ace710-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a41eaed2-344b-405c-86cb-631055ace710-istio-envoy\") pod \"a41eaed2-344b-405c-86cb-631055ace710\" (UID: \"a41eaed2-344b-405c-86cb-631055ace710\") "
Nov 02 03:28:51.115975 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:28:51.115862    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a41eaed2-344b-405c-86cb-631055ace710-workload-certs" pod "a41eaed2-344b-405c-86cb-631055ace710" (UID: "a41eaed2-344b-405c-86cb-631055ace710"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:28:51.115975 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:28:51.115889    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a41eaed2-344b-405c-86cb-631055ace710-kube-api-access-cfhgf\" (UniqueName: \"kubernetes.io/projected/a41eaed2-344b-405c-86cb-631055ace710-kube-api-access-cfhgf\") pod \"a41eaed2-344b-405c-86cb-631055ace710\" (UID: \"a41eaed2-344b-405c-86cb-631055ace710\") "
Nov 02 03:28:51.115975 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:28:51.115927    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a41eaed2-344b-405c-86cb-631055ace710-istio-token\" (UniqueName: \"kubernetes.io/projected/a41eaed2-344b-405c-86cb-631055ace710-istio-token\") pod \"a41eaed2-344b-405c-86cb-631055ace710\" (UID: \"a41eaed2-344b-405c-86cb-631055ace710\") "
Nov 02 03:28:51.116423 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:28:51.116075    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a41eaed2-344b-405c-86cb-631055ace710-workload-socket" pod "a41eaed2-344b-405c-86cb-631055ace710" (UID: "a41eaed2-344b-405c-86cb-631055ace710"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:28:51.116423 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:28:51.116151    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a41eaed2-344b-405c-86cb-631055ace710-istio-data" pod "a41eaed2-344b-405c-86cb-631055ace710" (UID: "a41eaed2-344b-405c-86cb-631055ace710"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:28:51.116423 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:28:51.116182    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a41eaed2-344b-405c-86cb-631055ace710-credential-socket" pod "a41eaed2-344b-405c-86cb-631055ace710" (UID: "a41eaed2-344b-405c-86cb-631055ace710"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:28:51.116423 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:28:51.116242    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a41eaed2-344b-405c-86cb-631055ace710-istiod-ca-cert" pod "a41eaed2-344b-405c-86cb-631055ace710" (UID: "a41eaed2-344b-405c-86cb-631055ace710"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:28:51.116626 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:28:51.116566    4496 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a41eaed2-344b-405c-86cb-631055ace710-workload-certs\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:28:51.116626 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:28:51.116588    4496 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a41eaed2-344b-405c-86cb-631055ace710-istiod-ca-cert\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:28:51.116626 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:28:51.116604    4496 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a41eaed2-344b-405c-86cb-631055ace710-workload-socket\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:28:51.116626 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:28:51.116618    4496 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a41eaed2-344b-405c-86cb-631055ace710-istio-data\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:28:51.116829 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:28:51.116633    4496 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a41eaed2-344b-405c-86cb-631055ace710-credential-socket\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:28:51.118506 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:28:51.118480    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a41eaed2-344b-405c-86cb-631055ace710-istio-envoy" pod "a41eaed2-344b-405c-86cb-631055ace710" (UID: "a41eaed2-344b-405c-86cb-631055ace710"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:28:51.118696 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:28:51.118666    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a41eaed2-344b-405c-86cb-631055ace710-kube-api-access-cfhgf" pod "a41eaed2-344b-405c-86cb-631055ace710" (UID: "a41eaed2-344b-405c-86cb-631055ace710"). InnerVolumeSpecName "kube-api-access-cfhgf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:28:51.118761 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:28:51.118694    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a41eaed2-344b-405c-86cb-631055ace710-istio-podinfo" pod "a41eaed2-344b-405c-86cb-631055ace710" (UID: "a41eaed2-344b-405c-86cb-631055ace710"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 03:28:51.118977 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:28:51.118959    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a41eaed2-344b-405c-86cb-631055ace710-istio-token" pod "a41eaed2-344b-405c-86cb-631055ace710" (UID: "a41eaed2-344b-405c-86cb-631055ace710"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:28:51.218405 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:28:51.218368    4496 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a41eaed2-344b-405c-86cb-631055ace710-istio-envoy\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:28:51.218405 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:28:51.218401    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfhgf\" (UniqueName: \"kubernetes.io/projected/a41eaed2-344b-405c-86cb-631055ace710-kube-api-access-cfhgf\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:28:51.218405 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:28:51.218412    4496 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a41eaed2-344b-405c-86cb-631055ace710-istio-token\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:28:51.218638 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:28:51.218421    4496 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a41eaed2-344b-405c-86cb-631055ace710-istio-podinfo\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:28:51.845202 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:28:51.845110    4496 generic.go:356] "Generic (PLEG): container finished" podID="a41eaed2-344b-405c-86cb-631055ace710" containerID="e8f062428768b2fd4e15c6b5823e3b230b07f39789d001da76df900c686c6d21" exitCode=0
Nov 02 03:28:51.845202 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:28:51.845174    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-kj8qb"
Nov 02 03:28:51.845408 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:28:51.845212    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-kj8qb" event={"ID":"a41eaed2-344b-405c-86cb-631055ace710","Type":"ContainerDied","Data":"e8f062428768b2fd4e15c6b5823e3b230b07f39789d001da76df900c686c6d21"}
Nov 02 03:28:51.845408 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:28:51.845265    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-kj8qb" event={"ID":"a41eaed2-344b-405c-86cb-631055ace710","Type":"ContainerDied","Data":"82339e375f1e036d70b6bc66adf53c4818cc2f14b3a432840647fd65c068d4fc"}
Nov 02 03:28:51.845408 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:28:51.845299    4496 scope.go:122] "RemoveContainer" containerID="e8f062428768b2fd4e15c6b5823e3b230b07f39789d001da76df900c686c6d21"
Nov 02 03:28:51.858598 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:28:51.858577    4496 scope.go:122] "RemoveContainer" containerID="e8f062428768b2fd4e15c6b5823e3b230b07f39789d001da76df900c686c6d21"
Nov 02 03:28:51.858878 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:28:51.858860    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8f062428768b2fd4e15c6b5823e3b230b07f39789d001da76df900c686c6d21\": container with ID starting with e8f062428768b2fd4e15c6b5823e3b230b07f39789d001da76df900c686c6d21 not found: ID does not exist" containerID="e8f062428768b2fd4e15c6b5823e3b230b07f39789d001da76df900c686c6d21"
Nov 02 03:28:51.858944 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:28:51.858888    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8f062428768b2fd4e15c6b5823e3b230b07f39789d001da76df900c686c6d21"} err="failed to get container status \"e8f062428768b2fd4e15c6b5823e3b230b07f39789d001da76df900c686c6d21\": rpc error: code = NotFound desc = could not find container \"e8f062428768b2fd4e15c6b5823e3b230b07f39789d001da76df900c686c6d21\": container with ID starting with e8f062428768b2fd4e15c6b5823e3b230b07f39789d001da76df900c686c6d21 not found: ID does not exist"
Nov 02 03:28:52.066159 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:28:52.066126    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-kj8qb"]
Nov 02 03:28:52.114717 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:28:52.114631    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-kj8qb"]
Nov 02 03:28:53.576710 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:28:53.576672    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a41eaed2-344b-405c-86cb-631055ace710" path="/var/lib/kubelet/pods/a41eaed2-344b-405c-86cb-631055ace710/volumes"
Nov 02 03:29:03.528338 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:29:03.528300    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 03:29:25.695114 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:25.695078    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-s9dfr"]
Nov 02 03:29:25.695622 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:25.695285    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-s9dfr" podUID="9e667364-fba5-4584-9251-773ae8e1f2e7" containerName="istio-proxy" containerID="cri-o://31e1aecc835a3ae846a1bb2868bd48d05de57994a7f4b3a5de746e497bc23141" gracePeriod=30
Nov 02 03:29:27.999097 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:27.999055    4496 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-s9dfr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 03:29:27.999500 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:27.999144    4496 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-s9dfr" podUID="9e667364-fba5-4584-9251-773ae8e1f2e7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 03:29:30.971267 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:30.971199    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-s9dfr"
Nov 02 03:29:31.058633 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:31.058595    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e667364-fba5-4584-9251-773ae8e1f2e7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e667364-fba5-4584-9251-773ae8e1f2e7-workload-certs\") pod \"9e667364-fba5-4584-9251-773ae8e1f2e7\" (UID: \"9e667364-fba5-4584-9251-773ae8e1f2e7\") "
Nov 02 03:29:31.058787 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:31.058678    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e667364-fba5-4584-9251-773ae8e1f2e7-istio-token\" (UniqueName: \"kubernetes.io/projected/9e667364-fba5-4584-9251-773ae8e1f2e7-istio-token\") pod \"9e667364-fba5-4584-9251-773ae8e1f2e7\" (UID: \"9e667364-fba5-4584-9251-773ae8e1f2e7\") "
Nov 02 03:29:31.058787 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:31.058766    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e667364-fba5-4584-9251-773ae8e1f2e7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e667364-fba5-4584-9251-773ae8e1f2e7-credential-socket\") pod \"9e667364-fba5-4584-9251-773ae8e1f2e7\" (UID: \"9e667364-fba5-4584-9251-773ae8e1f2e7\") "
Nov 02 03:29:31.058880 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:31.058827    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9e667364-fba5-4584-9251-773ae8e1f2e7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e667364-fba5-4584-9251-773ae8e1f2e7-istio-podinfo\") pod \"9e667364-fba5-4584-9251-773ae8e1f2e7\" (UID: \"9e667364-fba5-4584-9251-773ae8e1f2e7\") "
Nov 02 03:29:31.058930 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:31.058883    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e667364-fba5-4584-9251-773ae8e1f2e7-workload-certs" pod "9e667364-fba5-4584-9251-773ae8e1f2e7" (UID: "9e667364-fba5-4584-9251-773ae8e1f2e7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:29:31.058930 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:31.058902    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e667364-fba5-4584-9251-773ae8e1f2e7-kube-api-access-cmh7s\" (UniqueName: \"kubernetes.io/projected/9e667364-fba5-4584-9251-773ae8e1f2e7-kube-api-access-cmh7s\") pod \"9e667364-fba5-4584-9251-773ae8e1f2e7\" (UID: \"9e667364-fba5-4584-9251-773ae8e1f2e7\") "
Nov 02 03:29:31.059077 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:31.059053    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e667364-fba5-4584-9251-773ae8e1f2e7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e667364-fba5-4584-9251-773ae8e1f2e7-workload-socket\") pod \"9e667364-fba5-4584-9251-773ae8e1f2e7\" (UID: \"9e667364-fba5-4584-9251-773ae8e1f2e7\") "
Nov 02 03:29:31.059152 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:31.059079    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e667364-fba5-4584-9251-773ae8e1f2e7-credential-socket" pod "9e667364-fba5-4584-9251-773ae8e1f2e7" (UID: "9e667364-fba5-4584-9251-773ae8e1f2e7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:29:31.059152 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:31.059136    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e667364-fba5-4584-9251-773ae8e1f2e7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e667364-fba5-4584-9251-773ae8e1f2e7-istiod-ca-cert\") pod \"9e667364-fba5-4584-9251-773ae8e1f2e7\" (UID: \"9e667364-fba5-4584-9251-773ae8e1f2e7\") "
Nov 02 03:29:31.059270 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:31.059177    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e667364-fba5-4584-9251-773ae8e1f2e7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e667364-fba5-4584-9251-773ae8e1f2e7-istio-data\") pod \"9e667364-fba5-4584-9251-773ae8e1f2e7\" (UID: \"9e667364-fba5-4584-9251-773ae8e1f2e7\") "
Nov 02 03:29:31.059270 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:31.059239    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e667364-fba5-4584-9251-773ae8e1f2e7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e667364-fba5-4584-9251-773ae8e1f2e7-istio-envoy\") pod \"9e667364-fba5-4584-9251-773ae8e1f2e7\" (UID: \"9e667364-fba5-4584-9251-773ae8e1f2e7\") "
Nov 02 03:29:31.059506 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:31.059485    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e667364-fba5-4584-9251-773ae8e1f2e7-istio-data" pod "9e667364-fba5-4584-9251-773ae8e1f2e7" (UID: "9e667364-fba5-4584-9251-773ae8e1f2e7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:29:31.059587 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:31.059508    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e667364-fba5-4584-9251-773ae8e1f2e7-istiod-ca-cert" pod "9e667364-fba5-4584-9251-773ae8e1f2e7" (UID: "9e667364-fba5-4584-9251-773ae8e1f2e7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:29:31.059587 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:31.059563    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e667364-fba5-4584-9251-773ae8e1f2e7-workload-socket" pod "9e667364-fba5-4584-9251-773ae8e1f2e7" (UID: "9e667364-fba5-4584-9251-773ae8e1f2e7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:29:31.059865 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:31.059820    4496 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e667364-fba5-4584-9251-773ae8e1f2e7-credential-socket\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:29:31.059865 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:31.059849    4496 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e667364-fba5-4584-9251-773ae8e1f2e7-workload-socket\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:29:31.059865 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:31.059865    4496 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e667364-fba5-4584-9251-773ae8e1f2e7-istiod-ca-cert\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:29:31.060095 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:31.059880    4496 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e667364-fba5-4584-9251-773ae8e1f2e7-istio-data\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:29:31.060095 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:31.059893    4496 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e667364-fba5-4584-9251-773ae8e1f2e7-workload-certs\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:29:31.061292 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:31.061263    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9e667364-fba5-4584-9251-773ae8e1f2e7-istio-podinfo" pod "9e667364-fba5-4584-9251-773ae8e1f2e7" (UID: "9e667364-fba5-4584-9251-773ae8e1f2e7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 03:29:31.061578 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:31.061558    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e667364-fba5-4584-9251-773ae8e1f2e7-kube-api-access-cmh7s" pod "9e667364-fba5-4584-9251-773ae8e1f2e7" (UID: "9e667364-fba5-4584-9251-773ae8e1f2e7"). InnerVolumeSpecName "kube-api-access-cmh7s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:29:31.061865 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:31.061841    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e667364-fba5-4584-9251-773ae8e1f2e7-istio-token" pod "9e667364-fba5-4584-9251-773ae8e1f2e7" (UID: "9e667364-fba5-4584-9251-773ae8e1f2e7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:29:31.062056 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:31.062037    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e667364-fba5-4584-9251-773ae8e1f2e7-istio-envoy" pod "9e667364-fba5-4584-9251-773ae8e1f2e7" (UID: "9e667364-fba5-4584-9251-773ae8e1f2e7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:29:31.159866 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:31.159818    4496 generic.go:356] "Generic (PLEG): container finished" podID="9e667364-fba5-4584-9251-773ae8e1f2e7" containerID="31e1aecc835a3ae846a1bb2868bd48d05de57994a7f4b3a5de746e497bc23141" exitCode=0
Nov 02 03:29:31.160211 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:31.159857    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-s9dfr"
Nov 02 03:29:31.160470 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:31.159877    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-s9dfr" event={"ID":"9e667364-fba5-4584-9251-773ae8e1f2e7","Type":"ContainerDied","Data":"31e1aecc835a3ae846a1bb2868bd48d05de57994a7f4b3a5de746e497bc23141"}
Nov 02 03:29:31.160642 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:31.160621    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-s9dfr" event={"ID":"9e667364-fba5-4584-9251-773ae8e1f2e7","Type":"ContainerDied","Data":"c78caab70c6503ba801cc1c8f4529294fdd326aff887a7cbcbc4139b38dd5f28"}
Nov 02 03:29:31.161092 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:31.160992    4496 scope.go:122] "RemoveContainer" containerID="31e1aecc835a3ae846a1bb2868bd48d05de57994a7f4b3a5de746e497bc23141"
Nov 02 03:29:31.161716 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:31.161625    4496 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e667364-fba5-4584-9251-773ae8e1f2e7-istio-token\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:29:31.161716 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:31.161653    4496 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e667364-fba5-4584-9251-773ae8e1f2e7-istio-podinfo\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:29:31.161716 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:31.161671    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmh7s\" (UniqueName: \"kubernetes.io/projected/9e667364-fba5-4584-9251-773ae8e1f2e7-kube-api-access-cmh7s\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:29:31.161716 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:31.161689    4496 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e667364-fba5-4584-9251-773ae8e1f2e7-istio-envoy\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:29:31.175963 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:31.175915    4496 scope.go:122] "RemoveContainer" containerID="31e1aecc835a3ae846a1bb2868bd48d05de57994a7f4b3a5de746e497bc23141"
Nov 02 03:29:31.176261 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:29:31.176236    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31e1aecc835a3ae846a1bb2868bd48d05de57994a7f4b3a5de746e497bc23141\": container with ID starting with 31e1aecc835a3ae846a1bb2868bd48d05de57994a7f4b3a5de746e497bc23141 not found: ID does not exist" containerID="31e1aecc835a3ae846a1bb2868bd48d05de57994a7f4b3a5de746e497bc23141"
Nov 02 03:29:31.176345 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:31.176275    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31e1aecc835a3ae846a1bb2868bd48d05de57994a7f4b3a5de746e497bc23141"} err="failed to get container status \"31e1aecc835a3ae846a1bb2868bd48d05de57994a7f4b3a5de746e497bc23141\": rpc error: code = NotFound desc = could not find container \"31e1aecc835a3ae846a1bb2868bd48d05de57994a7f4b3a5de746e497bc23141\": container with ID starting with 31e1aecc835a3ae846a1bb2868bd48d05de57994a7f4b3a5de746e497bc23141 not found: ID does not exist"
Nov 02 03:29:31.332519 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:31.332476    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-s9dfr"]
Nov 02 03:29:31.336230 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:31.336201    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-s9dfr"]
Nov 02 03:29:31.575334 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:31.575297    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e667364-fba5-4584-9251-773ae8e1f2e7" path="/var/lib/kubelet/pods/9e667364-fba5-4584-9251-773ae8e1f2e7/volumes"
Nov 02 03:29:33.529728 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:29:33.529687    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 03:29:36.455770 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:36.455730    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-d94cc3-774b67d9f-knmz4"]
Nov 02 03:29:36.457287 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:36.457257    4496 cpu_manager.go:414] "containerMap: removing container" podUID="9e667364-fba5-4584-9251-773ae8e1f2e7" containerName="istio-proxy"
Nov 02 03:29:36.457447 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:36.457434    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e667364-fba5-4584-9251-773ae8e1f2e7" containerName="istio-proxy"
Nov 02 03:29:36.457595 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:36.457571    4496 cpu_manager.go:414] "containerMap: removing container" podUID="a41eaed2-344b-405c-86cb-631055ace710" containerName="istio-proxy"
Nov 02 03:29:36.457736 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:36.457716    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a41eaed2-344b-405c-86cb-631055ace710" containerName="istio-proxy"
Nov 02 03:29:36.457997 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:36.457985    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e667364-fba5-4584-9251-773ae8e1f2e7" containerName="istio-proxy"
Nov 02 03:29:36.458138 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:36.458127    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="a41eaed2-344b-405c-86cb-631055ace710" containerName="istio-proxy"
Nov 02 03:29:36.464080 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:36.464059    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d94cc3-774b67d9f-knmz4"
Nov 02 03:29:36.487914 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:36.486299    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-d94cc3-774b67d9f-knmz4"]
Nov 02 03:29:36.525470 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:36.525429    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5215d837-30bb-43d6-81fa-6d50df5e45bf-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-d94cc3-774b67d9f-knmz4\" (UID: \"5215d837-30bb-43d6-81fa-6d50df5e45bf\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d94cc3-774b67d9f-knmz4"
Nov 02 03:29:36.525644 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:36.525499    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gxs2\" (UniqueName: \"kubernetes.io/projected/5215d837-30bb-43d6-81fa-6d50df5e45bf-kube-api-access-7gxs2\") pod \"payload-processing-e2e-shared-b-w2-d94cc3-774b67d9f-knmz4\" (UID: \"5215d837-30bb-43d6-81fa-6d50df5e45bf\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d94cc3-774b67d9f-knmz4"
Nov 02 03:29:36.527821 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:36.527795    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-d94cc3\""
Nov 02 03:29:36.527943 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:36.527829    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-d94cc3-dockercfg-bj6jz\""
Nov 02 03:29:36.627230 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:36.627178    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5215d837-30bb-43d6-81fa-6d50df5e45bf-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-d94cc3-774b67d9f-knmz4\" (UID: \"5215d837-30bb-43d6-81fa-6d50df5e45bf\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d94cc3-774b67d9f-knmz4"
Nov 02 03:29:36.627387 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:36.627280    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gxs2\" (UniqueName: \"kubernetes.io/projected/5215d837-30bb-43d6-81fa-6d50df5e45bf-kube-api-access-7gxs2\") pod \"payload-processing-e2e-shared-b-w2-d94cc3-774b67d9f-knmz4\" (UID: \"5215d837-30bb-43d6-81fa-6d50df5e45bf\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d94cc3-774b67d9f-knmz4"
Nov 02 03:29:36.627882 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:36.627858    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5215d837-30bb-43d6-81fa-6d50df5e45bf-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-d94cc3-774b67d9f-knmz4\" (UID: \"5215d837-30bb-43d6-81fa-6d50df5e45bf\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d94cc3-774b67d9f-knmz4"
Nov 02 03:29:36.684537 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:36.684512    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gxs2\" (UniqueName: \"kubernetes.io/projected/5215d837-30bb-43d6-81fa-6d50df5e45bf-kube-api-access-7gxs2\") pod \"payload-processing-e2e-shared-b-w2-d94cc3-774b67d9f-knmz4\" (UID: \"5215d837-30bb-43d6-81fa-6d50df5e45bf\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d94cc3-774b67d9f-knmz4"
Nov 02 03:29:36.793643 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:36.793612    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d94cc3-774b67d9f-knmz4"
Nov 02 03:29:37.170344 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:37.170311    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-d94cc3-774b67d9f-knmz4"]
Nov 02 03:29:37.170497 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:29:37.170307    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5215d837_30bb_43d6_81fa_6d50df5e45bf.slice/crio-e81883696cdba08bd1cec8e37933aa24e9e65b0973c325afaab3c70f4c600c9e WatchSource:0}: Error finding container e81883696cdba08bd1cec8e37933aa24e9e65b0973c325afaab3c70f4c600c9e: Status 404 returned error can't find the container with id e81883696cdba08bd1cec8e37933aa24e9e65b0973c325afaab3c70f4c600c9e
Nov 02 03:29:37.210881 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:37.210853    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d94cc3-774b67d9f-knmz4" event={"ID":"5215d837-30bb-43d6-81fa-6d50df5e45bf","Type":"ContainerStarted","Data":"e81883696cdba08bd1cec8e37933aa24e9e65b0973c325afaab3c70f4c600c9e"}
Nov 02 03:29:37.389271 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:37.389233    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-jx2rx"]
Nov 02 03:29:37.389461 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:37.389422    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-jx2rx" podUID="2ecf68d6-0a44-436c-938f-e75696a37e6a" containerName="istio-proxy" containerID="cri-o://f9985833faa6c86d92e567ccfa70c3e7dc7bc99671ec8e3f78c5b4530250ad7a" gracePeriod=30
Nov 02 03:29:37.410248 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:37.410215    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-qtwvf"]
Nov 02 03:29:37.410479 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:37.410449    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-qtwvf" podUID="4975cf07-dbfe-4cd1-b857-37773a64e47c" containerName="istio-proxy" containerID="cri-o://a0db0477225506bce774eec4f294e517234bb9c687d857d6eef8d902f5831c18" gracePeriod=30
Nov 02 03:29:38.097077 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:38.097037    4496 patch_prober.go:29] interesting pod/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-jx2rx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 03:29:38.097547 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:38.097101    4496 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-jx2rx" podUID="2ecf68d6-0a44-436c-938f-e75696a37e6a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 03:29:38.219861 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:38.219825    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d94cc3-774b67d9f-knmz4" event={"ID":"5215d837-30bb-43d6-81fa-6d50df5e45bf","Type":"ContainerStarted","Data":"4d9282d0eced70db7ca57c22282db6123d057532dd2e1f6dcc3b4781fc299163"}
Nov 02 03:29:38.220095 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:38.219894    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d94cc3-774b67d9f-knmz4"
Nov 02 03:29:39.604458 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:39.604423    4496 patch_prober.go:29] interesting pod/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-qtwvf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 03:29:39.604827 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:39.604479    4496 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-qtwvf" podUID="4975cf07-dbfe-4cd1-b857-37773a64e47c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 03:29:41.238206 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:41.238173    4496 scope.go:122] "RemoveContainer" containerID="74b4a9412faabcb34bcab873f489ed3563572c463ce0042cc498624c6caecde8"
Nov 02 03:29:42.687499 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:42.687471    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-jx2rx"
Nov 02 03:29:42.726165 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:42.725315    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-qtwvf"
Nov 02 03:29:42.827561 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:42.827528    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ecf68d6-0a44-436c-938f-e75696a37e6a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ecf68d6-0a44-436c-938f-e75696a37e6a-istio-envoy\") pod \"2ecf68d6-0a44-436c-938f-e75696a37e6a\" (UID: \"2ecf68d6-0a44-436c-938f-e75696a37e6a\") "
Nov 02 03:29:42.827561 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:42.827574    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2ecf68d6-0a44-436c-938f-e75696a37e6a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ecf68d6-0a44-436c-938f-e75696a37e6a-istio-podinfo\") pod \"2ecf68d6-0a44-436c-938f-e75696a37e6a\" (UID: \"2ecf68d6-0a44-436c-938f-e75696a37e6a\") "
Nov 02 03:29:42.827822 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:42.827614    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ecf68d6-0a44-436c-938f-e75696a37e6a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ecf68d6-0a44-436c-938f-e75696a37e6a-istiod-ca-cert\") pod \"2ecf68d6-0a44-436c-938f-e75696a37e6a\" (UID: \"2ecf68d6-0a44-436c-938f-e75696a37e6a\") "
Nov 02 03:29:42.827822 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:42.827634    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ecf68d6-0a44-436c-938f-e75696a37e6a-istio-token\" (UniqueName: \"kubernetes.io/projected/2ecf68d6-0a44-436c-938f-e75696a37e6a-istio-token\") pod \"2ecf68d6-0a44-436c-938f-e75696a37e6a\" (UID: \"2ecf68d6-0a44-436c-938f-e75696a37e6a\") "
Nov 02 03:29:42.827822 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:42.827681    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4975cf07-dbfe-4cd1-b857-37773a64e47c-kube-api-access-vk5hh\" (UniqueName: \"kubernetes.io/projected/4975cf07-dbfe-4cd1-b857-37773a64e47c-kube-api-access-vk5hh\") pod \"4975cf07-dbfe-4cd1-b857-37773a64e47c\" (UID: \"4975cf07-dbfe-4cd1-b857-37773a64e47c\") "
Nov 02 03:29:42.827822 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:42.827720    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ecf68d6-0a44-436c-938f-e75696a37e6a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ecf68d6-0a44-436c-938f-e75696a37e6a-credential-socket\") pod \"2ecf68d6-0a44-436c-938f-e75696a37e6a\" (UID: \"2ecf68d6-0a44-436c-938f-e75696a37e6a\") "
Nov 02 03:29:42.827822 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:42.827750    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ecf68d6-0a44-436c-938f-e75696a37e6a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ecf68d6-0a44-436c-938f-e75696a37e6a-istio-data\") pod \"2ecf68d6-0a44-436c-938f-e75696a37e6a\" (UID: \"2ecf68d6-0a44-436c-938f-e75696a37e6a\") "
Nov 02 03:29:42.828136 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:42.827847    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ecf68d6-0a44-436c-938f-e75696a37e6a-kube-api-access-9h4pn\" (UniqueName: \"kubernetes.io/projected/2ecf68d6-0a44-436c-938f-e75696a37e6a-kube-api-access-9h4pn\") pod \"2ecf68d6-0a44-436c-938f-e75696a37e6a\" (UID: \"2ecf68d6-0a44-436c-938f-e75696a37e6a\") "
Nov 02 03:29:42.828136 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:42.827876    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4975cf07-dbfe-4cd1-b857-37773a64e47c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4975cf07-dbfe-4cd1-b857-37773a64e47c-istio-podinfo\") pod \"4975cf07-dbfe-4cd1-b857-37773a64e47c\" (UID: \"4975cf07-dbfe-4cd1-b857-37773a64e47c\") "
Nov 02 03:29:42.828136 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:42.827914    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4975cf07-dbfe-4cd1-b857-37773a64e47c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4975cf07-dbfe-4cd1-b857-37773a64e47c-istio-envoy\") pod \"4975cf07-dbfe-4cd1-b857-37773a64e47c\" (UID: \"4975cf07-dbfe-4cd1-b857-37773a64e47c\") "
Nov 02 03:29:42.828136 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:42.827976    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4975cf07-dbfe-4cd1-b857-37773a64e47c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4975cf07-dbfe-4cd1-b857-37773a64e47c-workload-socket\") pod \"4975cf07-dbfe-4cd1-b857-37773a64e47c\" (UID: \"4975cf07-dbfe-4cd1-b857-37773a64e47c\") "
Nov 02 03:29:42.828136 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:42.828041    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ecf68d6-0a44-436c-938f-e75696a37e6a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ecf68d6-0a44-436c-938f-e75696a37e6a-workload-socket\") pod \"2ecf68d6-0a44-436c-938f-e75696a37e6a\" (UID: \"2ecf68d6-0a44-436c-938f-e75696a37e6a\") "
Nov 02 03:29:42.828136 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:42.828069    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ecf68d6-0a44-436c-938f-e75696a37e6a-istiod-ca-cert" pod "2ecf68d6-0a44-436c-938f-e75696a37e6a" (UID: "2ecf68d6-0a44-436c-938f-e75696a37e6a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:29:42.828136 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:42.828088    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4975cf07-dbfe-4cd1-b857-37773a64e47c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4975cf07-dbfe-4cd1-b857-37773a64e47c-workload-certs\") pod \"4975cf07-dbfe-4cd1-b857-37773a64e47c\" (UID: \"4975cf07-dbfe-4cd1-b857-37773a64e47c\") "
Nov 02 03:29:42.828136 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:42.828070    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ecf68d6-0a44-436c-938f-e75696a37e6a-credential-socket" pod "2ecf68d6-0a44-436c-938f-e75696a37e6a" (UID: "2ecf68d6-0a44-436c-938f-e75696a37e6a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:29:42.828558 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:42.828142    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ecf68d6-0a44-436c-938f-e75696a37e6a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ecf68d6-0a44-436c-938f-e75696a37e6a-workload-certs\") pod \"2ecf68d6-0a44-436c-938f-e75696a37e6a\" (UID: \"2ecf68d6-0a44-436c-938f-e75696a37e6a\") "
Nov 02 03:29:42.828558 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:42.828173    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4975cf07-dbfe-4cd1-b857-37773a64e47c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4975cf07-dbfe-4cd1-b857-37773a64e47c-istiod-ca-cert\") pod \"4975cf07-dbfe-4cd1-b857-37773a64e47c\" (UID: \"4975cf07-dbfe-4cd1-b857-37773a64e47c\") "
Nov 02 03:29:42.828558 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:42.828221    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4975cf07-dbfe-4cd1-b857-37773a64e47c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4975cf07-dbfe-4cd1-b857-37773a64e47c-credential-socket\") pod \"4975cf07-dbfe-4cd1-b857-37773a64e47c\" (UID: \"4975cf07-dbfe-4cd1-b857-37773a64e47c\") "
Nov 02 03:29:42.828558 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:42.828276    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4975cf07-dbfe-4cd1-b857-37773a64e47c-workload-socket" pod "4975cf07-dbfe-4cd1-b857-37773a64e47c" (UID: "4975cf07-dbfe-4cd1-b857-37773a64e47c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:29:42.828558 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:42.828330    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4975cf07-dbfe-4cd1-b857-37773a64e47c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4975cf07-dbfe-4cd1-b857-37773a64e47c-istio-data\") pod \"4975cf07-dbfe-4cd1-b857-37773a64e47c\" (UID: \"4975cf07-dbfe-4cd1-b857-37773a64e47c\") "
Nov 02 03:29:42.828558 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:42.828393    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4975cf07-dbfe-4cd1-b857-37773a64e47c-istio-token\" (UniqueName: \"kubernetes.io/projected/4975cf07-dbfe-4cd1-b857-37773a64e47c-istio-token\") pod \"4975cf07-dbfe-4cd1-b857-37773a64e47c\" (UID: \"4975cf07-dbfe-4cd1-b857-37773a64e47c\") "
Nov 02 03:29:42.828558 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:42.828520    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ecf68d6-0a44-436c-938f-e75696a37e6a-istio-data" pod "2ecf68d6-0a44-436c-938f-e75696a37e6a" (UID: "2ecf68d6-0a44-436c-938f-e75696a37e6a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:29:42.828949 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:42.828686    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ecf68d6-0a44-436c-938f-e75696a37e6a-workload-certs" pod "2ecf68d6-0a44-436c-938f-e75696a37e6a" (UID: "2ecf68d6-0a44-436c-938f-e75696a37e6a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:29:42.829193 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:42.829168    4496 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ecf68d6-0a44-436c-938f-e75696a37e6a-credential-socket\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:29:42.829289 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:42.829197    4496 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ecf68d6-0a44-436c-938f-e75696a37e6a-istio-data\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:29:42.829289 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:42.829250    4496 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4975cf07-dbfe-4cd1-b857-37773a64e47c-workload-socket\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:29:42.829289 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:42.829265    4496 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ecf68d6-0a44-436c-938f-e75696a37e6a-workload-certs\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:29:42.829289 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:42.829280    4496 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ecf68d6-0a44-436c-938f-e75696a37e6a-istiod-ca-cert\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:29:42.830653 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:42.830625    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ecf68d6-0a44-436c-938f-e75696a37e6a-workload-socket" pod "2ecf68d6-0a44-436c-938f-e75696a37e6a" (UID: "2ecf68d6-0a44-436c-938f-e75696a37e6a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:29:42.831424 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:42.830861    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4975cf07-dbfe-4cd1-b857-37773a64e47c-workload-certs" pod "4975cf07-dbfe-4cd1-b857-37773a64e47c" (UID: "4975cf07-dbfe-4cd1-b857-37773a64e47c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:29:42.831424 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:42.831119    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4975cf07-dbfe-4cd1-b857-37773a64e47c-credential-socket" pod "4975cf07-dbfe-4cd1-b857-37773a64e47c" (UID: "4975cf07-dbfe-4cd1-b857-37773a64e47c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:29:42.831424 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:42.831256    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4975cf07-dbfe-4cd1-b857-37773a64e47c-istio-podinfo" pod "4975cf07-dbfe-4cd1-b857-37773a64e47c" (UID: "4975cf07-dbfe-4cd1-b857-37773a64e47c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 03:29:42.831424 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:42.831315    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4975cf07-dbfe-4cd1-b857-37773a64e47c-istio-data" pod "4975cf07-dbfe-4cd1-b857-37773a64e47c" (UID: "4975cf07-dbfe-4cd1-b857-37773a64e47c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:29:42.831424 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:42.831368    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4975cf07-dbfe-4cd1-b857-37773a64e47c-istiod-ca-cert" pod "4975cf07-dbfe-4cd1-b857-37773a64e47c" (UID: "4975cf07-dbfe-4cd1-b857-37773a64e47c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:29:42.831424 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:42.831387    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4975cf07-dbfe-4cd1-b857-37773a64e47c-istio-token" pod "4975cf07-dbfe-4cd1-b857-37773a64e47c" (UID: "4975cf07-dbfe-4cd1-b857-37773a64e47c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:29:42.831720 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:42.831619    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2ecf68d6-0a44-436c-938f-e75696a37e6a-istio-podinfo" pod "2ecf68d6-0a44-436c-938f-e75696a37e6a" (UID: "2ecf68d6-0a44-436c-938f-e75696a37e6a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 03:29:42.831772 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:42.831728    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ecf68d6-0a44-436c-938f-e75696a37e6a-istio-envoy" pod "2ecf68d6-0a44-436c-938f-e75696a37e6a" (UID: "2ecf68d6-0a44-436c-938f-e75696a37e6a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:29:42.832243 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:42.832221    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4975cf07-dbfe-4cd1-b857-37773a64e47c-istio-envoy" pod "4975cf07-dbfe-4cd1-b857-37773a64e47c" (UID: "4975cf07-dbfe-4cd1-b857-37773a64e47c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:29:42.832379 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:42.832353    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ecf68d6-0a44-436c-938f-e75696a37e6a-istio-token" pod "2ecf68d6-0a44-436c-938f-e75696a37e6a" (UID: "2ecf68d6-0a44-436c-938f-e75696a37e6a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:29:42.833228 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:42.833207    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ecf68d6-0a44-436c-938f-e75696a37e6a-kube-api-access-9h4pn" pod "2ecf68d6-0a44-436c-938f-e75696a37e6a" (UID: "2ecf68d6-0a44-436c-938f-e75696a37e6a"). InnerVolumeSpecName "kube-api-access-9h4pn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:29:42.833479 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:42.833448    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4975cf07-dbfe-4cd1-b857-37773a64e47c-kube-api-access-vk5hh" pod "4975cf07-dbfe-4cd1-b857-37773a64e47c" (UID: "4975cf07-dbfe-4cd1-b857-37773a64e47c"). InnerVolumeSpecName "kube-api-access-vk5hh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:29:42.858552 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:42.858501    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d94cc3-774b67d9f-knmz4" podStartSLOduration=6.858486137 podStartE2EDuration="6.858486137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:29:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:29:38.353667696 +0000 UTC m=+2518.932275076" watchObservedRunningTime="2026-11-02 03:29:42.858486137 +0000 UTC m=+2523.437093522"
Nov 02 03:29:42.930799 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:42.930729    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vk5hh\" (UniqueName: \"kubernetes.io/projected/4975cf07-dbfe-4cd1-b857-37773a64e47c-kube-api-access-vk5hh\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:29:42.930799 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:42.930754    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9h4pn\" (UniqueName: \"kubernetes.io/projected/2ecf68d6-0a44-436c-938f-e75696a37e6a-kube-api-access-9h4pn\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:29:42.930799 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:42.930765    4496 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4975cf07-dbfe-4cd1-b857-37773a64e47c-istio-podinfo\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:29:42.930799 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:42.930774    4496 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4975cf07-dbfe-4cd1-b857-37773a64e47c-istio-envoy\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:29:42.930799 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:42.930783    4496 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ecf68d6-0a44-436c-938f-e75696a37e6a-workload-socket\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:29:42.930799 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:42.930791    4496 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4975cf07-dbfe-4cd1-b857-37773a64e47c-workload-certs\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:29:42.930799 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:42.930799    4496 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4975cf07-dbfe-4cd1-b857-37773a64e47c-istiod-ca-cert\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:29:42.931210 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:42.930807    4496 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4975cf07-dbfe-4cd1-b857-37773a64e47c-credential-socket\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:29:42.931210 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:42.930815    4496 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4975cf07-dbfe-4cd1-b857-37773a64e47c-istio-data\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:29:42.931210 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:42.930822    4496 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4975cf07-dbfe-4cd1-b857-37773a64e47c-istio-token\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:29:42.931210 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:42.930832    4496 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ecf68d6-0a44-436c-938f-e75696a37e6a-istio-envoy\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:29:42.931210 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:42.930840    4496 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ecf68d6-0a44-436c-938f-e75696a37e6a-istio-podinfo\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:29:42.931210 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:42.930847    4496 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ecf68d6-0a44-436c-938f-e75696a37e6a-istio-token\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:29:43.273035 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:43.271134    4496 generic.go:356] "Generic (PLEG): container finished" podID="4975cf07-dbfe-4cd1-b857-37773a64e47c" containerID="a0db0477225506bce774eec4f294e517234bb9c687d857d6eef8d902f5831c18" exitCode=0
Nov 02 03:29:43.273035 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:43.271200    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-qtwvf"
Nov 02 03:29:43.273035 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:43.271231    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-qtwvf" event={"ID":"4975cf07-dbfe-4cd1-b857-37773a64e47c","Type":"ContainerDied","Data":"a0db0477225506bce774eec4f294e517234bb9c687d857d6eef8d902f5831c18"}
Nov 02 03:29:43.273035 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:43.271299    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-qtwvf" event={"ID":"4975cf07-dbfe-4cd1-b857-37773a64e47c","Type":"ContainerDied","Data":"1f65c19dba51cfd3e9beb31c7f5053d57695508a4e7552e7c8c2afd937947b8b"}
Nov 02 03:29:43.273035 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:43.271330    4496 scope.go:122] "RemoveContainer" containerID="a0db0477225506bce774eec4f294e517234bb9c687d857d6eef8d902f5831c18"
Nov 02 03:29:43.276295 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:43.276268    4496 generic.go:356] "Generic (PLEG): container finished" podID="2ecf68d6-0a44-436c-938f-e75696a37e6a" containerID="f9985833faa6c86d92e567ccfa70c3e7dc7bc99671ec8e3f78c5b4530250ad7a" exitCode=0
Nov 02 03:29:43.276505 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:43.276488    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-jx2rx" event={"ID":"2ecf68d6-0a44-436c-938f-e75696a37e6a","Type":"ContainerDied","Data":"f9985833faa6c86d92e567ccfa70c3e7dc7bc99671ec8e3f78c5b4530250ad7a"}
Nov 02 03:29:43.276573 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:43.276537    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-jx2rx" event={"ID":"2ecf68d6-0a44-436c-938f-e75696a37e6a","Type":"ContainerDied","Data":"43af5d8c6176e55c48b8923b427d27b0d3624b208d684a20a981cbde2f75f0f9"}
Nov 02 03:29:43.276634 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:43.276601    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-jx2rx"
Nov 02 03:29:43.294627 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:43.294602    4496 scope.go:122] "RemoveContainer" containerID="a0db0477225506bce774eec4f294e517234bb9c687d857d6eef8d902f5831c18"
Nov 02 03:29:43.295059 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:29:43.295029    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0db0477225506bce774eec4f294e517234bb9c687d857d6eef8d902f5831c18\": container with ID starting with a0db0477225506bce774eec4f294e517234bb9c687d857d6eef8d902f5831c18 not found: ID does not exist" containerID="a0db0477225506bce774eec4f294e517234bb9c687d857d6eef8d902f5831c18"
Nov 02 03:29:43.295157 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:43.295070    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0db0477225506bce774eec4f294e517234bb9c687d857d6eef8d902f5831c18"} err="failed to get container status \"a0db0477225506bce774eec4f294e517234bb9c687d857d6eef8d902f5831c18\": rpc error: code = NotFound desc = could not find container \"a0db0477225506bce774eec4f294e517234bb9c687d857d6eef8d902f5831c18\": container with ID starting with a0db0477225506bce774eec4f294e517234bb9c687d857d6eef8d902f5831c18 not found: ID does not exist"
Nov 02 03:29:43.295157 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:43.295096    4496 scope.go:122] "RemoveContainer" containerID="f9985833faa6c86d92e567ccfa70c3e7dc7bc99671ec8e3f78c5b4530250ad7a"
Nov 02 03:29:43.312533 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:43.312510    4496 scope.go:122] "RemoveContainer" containerID="f9985833faa6c86d92e567ccfa70c3e7dc7bc99671ec8e3f78c5b4530250ad7a"
Nov 02 03:29:43.313363 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:29:43.313325    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9985833faa6c86d92e567ccfa70c3e7dc7bc99671ec8e3f78c5b4530250ad7a\": container with ID starting with f9985833faa6c86d92e567ccfa70c3e7dc7bc99671ec8e3f78c5b4530250ad7a not found: ID does not exist" containerID="f9985833faa6c86d92e567ccfa70c3e7dc7bc99671ec8e3f78c5b4530250ad7a"
Nov 02 03:29:43.313480 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:43.313364    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9985833faa6c86d92e567ccfa70c3e7dc7bc99671ec8e3f78c5b4530250ad7a"} err="failed to get container status \"f9985833faa6c86d92e567ccfa70c3e7dc7bc99671ec8e3f78c5b4530250ad7a\": rpc error: code = NotFound desc = could not find container \"f9985833faa6c86d92e567ccfa70c3e7dc7bc99671ec8e3f78c5b4530250ad7a\": container with ID starting with f9985833faa6c86d92e567ccfa70c3e7dc7bc99671ec8e3f78c5b4530250ad7a not found: ID does not exist"
Nov 02 03:29:43.444169 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:43.442876    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-jx2rx"]
Nov 02 03:29:43.488567 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:43.488538    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-jx2rx"]
Nov 02 03:29:43.553613 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:43.553529    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-qtwvf"]
Nov 02 03:29:43.558311 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:43.558279    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-qtwvf"]
Nov 02 03:29:43.574961 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:43.574933    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ecf68d6-0a44-436c-938f-e75696a37e6a" path="/var/lib/kubelet/pods/2ecf68d6-0a44-436c-938f-e75696a37e6a/volumes"
Nov 02 03:29:43.575454 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:43.575439    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4975cf07-dbfe-4cd1-b857-37773a64e47c" path="/var/lib/kubelet/pods/4975cf07-dbfe-4cd1-b857-37773a64e47c/volumes"
Nov 02 03:29:44.131579 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:44.131536    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-018375-openshift-default-7fbc48965-7mlcx"]
Nov 02 03:29:44.132191 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:44.132174    4496 cpu_manager.go:414] "containerMap: removing container" podUID="4975cf07-dbfe-4cd1-b857-37773a64e47c" containerName="istio-proxy"
Nov 02 03:29:44.132191 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:44.132191    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4975cf07-dbfe-4cd1-b857-37773a64e47c" containerName="istio-proxy"
Nov 02 03:29:44.132296 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:44.132215    4496 cpu_manager.go:414] "containerMap: removing container" podUID="2ecf68d6-0a44-436c-938f-e75696a37e6a" containerName="istio-proxy"
Nov 02 03:29:44.132296 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:44.132220    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ecf68d6-0a44-436c-938f-e75696a37e6a" containerName="istio-proxy"
Nov 02 03:29:44.132364 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:44.132324    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="4975cf07-dbfe-4cd1-b857-37773a64e47c" containerName="istio-proxy"
Nov 02 03:29:44.132364 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:44.132335    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ecf68d6-0a44-436c-938f-e75696a37e6a" containerName="istio-proxy"
Nov 02 03:29:44.137467 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:44.137447    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-018375-openshift-default-7fbc48965-7mlcx"
Nov 02 03:29:44.196610 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:44.196575    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-018375-openshift-default-dockercfg-dmj9r\""
Nov 02 03:29:44.222068 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:44.204490    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-018375-openshift-default-7fbc48965-7mlcx"]
Nov 02 03:29:44.254323 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:44.254281    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecfffbc9-4809-49a3-aed3-35b771fcb388-istio-envoy\") pod \"e2e-ipp-b-018375-openshift-default-7fbc48965-7mlcx\" (UID: \"ecfffbc9-4809-49a3-aed3-35b771fcb388\") " pod="openshift-ingress/e2e-ipp-b-018375-openshift-default-7fbc48965-7mlcx"
Nov 02 03:29:44.254518 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:44.254363    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecfffbc9-4809-49a3-aed3-35b771fcb388-workload-socket\") pod \"e2e-ipp-b-018375-openshift-default-7fbc48965-7mlcx\" (UID: \"ecfffbc9-4809-49a3-aed3-35b771fcb388\") " pod="openshift-ingress/e2e-ipp-b-018375-openshift-default-7fbc48965-7mlcx"
Nov 02 03:29:44.254518 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:44.254394    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecfffbc9-4809-49a3-aed3-35b771fcb388-istio-podinfo\") pod \"e2e-ipp-b-018375-openshift-default-7fbc48965-7mlcx\" (UID: \"ecfffbc9-4809-49a3-aed3-35b771fcb388\") " pod="openshift-ingress/e2e-ipp-b-018375-openshift-default-7fbc48965-7mlcx"
Nov 02 03:29:44.254518 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:44.254425    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sjp2\" (UniqueName: \"kubernetes.io/projected/ecfffbc9-4809-49a3-aed3-35b771fcb388-kube-api-access-2sjp2\") pod \"e2e-ipp-b-018375-openshift-default-7fbc48965-7mlcx\" (UID: \"ecfffbc9-4809-49a3-aed3-35b771fcb388\") " pod="openshift-ingress/e2e-ipp-b-018375-openshift-default-7fbc48965-7mlcx"
Nov 02 03:29:44.254518 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:44.254514    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecfffbc9-4809-49a3-aed3-35b771fcb388-workload-certs\") pod \"e2e-ipp-b-018375-openshift-default-7fbc48965-7mlcx\" (UID: \"ecfffbc9-4809-49a3-aed3-35b771fcb388\") " pod="openshift-ingress/e2e-ipp-b-018375-openshift-default-7fbc48965-7mlcx"
Nov 02 03:29:44.254742 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:44.254559    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecfffbc9-4809-49a3-aed3-35b771fcb388-istio-token\") pod \"e2e-ipp-b-018375-openshift-default-7fbc48965-7mlcx\" (UID: \"ecfffbc9-4809-49a3-aed3-35b771fcb388\") " pod="openshift-ingress/e2e-ipp-b-018375-openshift-default-7fbc48965-7mlcx"
Nov 02 03:29:44.254742 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:44.254617    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecfffbc9-4809-49a3-aed3-35b771fcb388-istio-data\") pod \"e2e-ipp-b-018375-openshift-default-7fbc48965-7mlcx\" (UID: \"ecfffbc9-4809-49a3-aed3-35b771fcb388\") " pod="openshift-ingress/e2e-ipp-b-018375-openshift-default-7fbc48965-7mlcx"
Nov 02 03:29:44.254742 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:44.254650    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecfffbc9-4809-49a3-aed3-35b771fcb388-credential-socket\") pod \"e2e-ipp-b-018375-openshift-default-7fbc48965-7mlcx\" (UID: \"ecfffbc9-4809-49a3-aed3-35b771fcb388\") " pod="openshift-ingress/e2e-ipp-b-018375-openshift-default-7fbc48965-7mlcx"
Nov 02 03:29:44.254742 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:44.254711    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecfffbc9-4809-49a3-aed3-35b771fcb388-istiod-ca-cert\") pod \"e2e-ipp-b-018375-openshift-default-7fbc48965-7mlcx\" (UID: \"ecfffbc9-4809-49a3-aed3-35b771fcb388\") " pod="openshift-ingress/e2e-ipp-b-018375-openshift-default-7fbc48965-7mlcx"
Nov 02 03:29:44.355573 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:44.355537    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecfffbc9-4809-49a3-aed3-35b771fcb388-workload-certs\") pod \"e2e-ipp-b-018375-openshift-default-7fbc48965-7mlcx\" (UID: \"ecfffbc9-4809-49a3-aed3-35b771fcb388\") " pod="openshift-ingress/e2e-ipp-b-018375-openshift-default-7fbc48965-7mlcx"
Nov 02 03:29:44.355755 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:44.355591    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecfffbc9-4809-49a3-aed3-35b771fcb388-istio-token\") pod \"e2e-ipp-b-018375-openshift-default-7fbc48965-7mlcx\" (UID: \"ecfffbc9-4809-49a3-aed3-35b771fcb388\") " pod="openshift-ingress/e2e-ipp-b-018375-openshift-default-7fbc48965-7mlcx"
Nov 02 03:29:44.355755 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:44.355653    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecfffbc9-4809-49a3-aed3-35b771fcb388-istio-data\") pod \"e2e-ipp-b-018375-openshift-default-7fbc48965-7mlcx\" (UID: \"ecfffbc9-4809-49a3-aed3-35b771fcb388\") " pod="openshift-ingress/e2e-ipp-b-018375-openshift-default-7fbc48965-7mlcx"
Nov 02 03:29:44.355755 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:44.355708    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecfffbc9-4809-49a3-aed3-35b771fcb388-credential-socket\") pod \"e2e-ipp-b-018375-openshift-default-7fbc48965-7mlcx\" (UID: \"ecfffbc9-4809-49a3-aed3-35b771fcb388\") " pod="openshift-ingress/e2e-ipp-b-018375-openshift-default-7fbc48965-7mlcx"
Nov 02 03:29:44.356061 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:44.356038    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecfffbc9-4809-49a3-aed3-35b771fcb388-workload-certs\") pod \"e2e-ipp-b-018375-openshift-default-7fbc48965-7mlcx\" (UID: \"ecfffbc9-4809-49a3-aed3-35b771fcb388\") " pod="openshift-ingress/e2e-ipp-b-018375-openshift-default-7fbc48965-7mlcx"
Nov 02 03:29:44.356125 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:44.356077    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecfffbc9-4809-49a3-aed3-35b771fcb388-istio-data\") pod \"e2e-ipp-b-018375-openshift-default-7fbc48965-7mlcx\" (UID: \"ecfffbc9-4809-49a3-aed3-35b771fcb388\") " pod="openshift-ingress/e2e-ipp-b-018375-openshift-default-7fbc48965-7mlcx"
Nov 02 03:29:44.356202 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:44.356182    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecfffbc9-4809-49a3-aed3-35b771fcb388-istiod-ca-cert\") pod \"e2e-ipp-b-018375-openshift-default-7fbc48965-7mlcx\" (UID: \"ecfffbc9-4809-49a3-aed3-35b771fcb388\") " pod="openshift-ingress/e2e-ipp-b-018375-openshift-default-7fbc48965-7mlcx"
Nov 02 03:29:44.356297 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:44.356278    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecfffbc9-4809-49a3-aed3-35b771fcb388-istio-envoy\") pod \"e2e-ipp-b-018375-openshift-default-7fbc48965-7mlcx\" (UID: \"ecfffbc9-4809-49a3-aed3-35b771fcb388\") " pod="openshift-ingress/e2e-ipp-b-018375-openshift-default-7fbc48965-7mlcx"
Nov 02 03:29:44.356363 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:44.356341    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecfffbc9-4809-49a3-aed3-35b771fcb388-credential-socket\") pod \"e2e-ipp-b-018375-openshift-default-7fbc48965-7mlcx\" (UID: \"ecfffbc9-4809-49a3-aed3-35b771fcb388\") " pod="openshift-ingress/e2e-ipp-b-018375-openshift-default-7fbc48965-7mlcx"
Nov 02 03:29:44.356697 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:44.356676    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecfffbc9-4809-49a3-aed3-35b771fcb388-workload-socket\") pod \"e2e-ipp-b-018375-openshift-default-7fbc48965-7mlcx\" (UID: \"ecfffbc9-4809-49a3-aed3-35b771fcb388\") " pod="openshift-ingress/e2e-ipp-b-018375-openshift-default-7fbc48965-7mlcx"
Nov 02 03:29:44.356776 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:44.356710    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecfffbc9-4809-49a3-aed3-35b771fcb388-istio-podinfo\") pod \"e2e-ipp-b-018375-openshift-default-7fbc48965-7mlcx\" (UID: \"ecfffbc9-4809-49a3-aed3-35b771fcb388\") " pod="openshift-ingress/e2e-ipp-b-018375-openshift-default-7fbc48965-7mlcx"
Nov 02 03:29:44.356776 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:44.356742    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sjp2\" (UniqueName: \"kubernetes.io/projected/ecfffbc9-4809-49a3-aed3-35b771fcb388-kube-api-access-2sjp2\") pod \"e2e-ipp-b-018375-openshift-default-7fbc48965-7mlcx\" (UID: \"ecfffbc9-4809-49a3-aed3-35b771fcb388\") " pod="openshift-ingress/e2e-ipp-b-018375-openshift-default-7fbc48965-7mlcx"
Nov 02 03:29:44.357189 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:44.357167    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecfffbc9-4809-49a3-aed3-35b771fcb388-workload-socket\") pod \"e2e-ipp-b-018375-openshift-default-7fbc48965-7mlcx\" (UID: \"ecfffbc9-4809-49a3-aed3-35b771fcb388\") " pod="openshift-ingress/e2e-ipp-b-018375-openshift-default-7fbc48965-7mlcx"
Nov 02 03:29:44.357346 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:44.357185    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecfffbc9-4809-49a3-aed3-35b771fcb388-istiod-ca-cert\") pod \"e2e-ipp-b-018375-openshift-default-7fbc48965-7mlcx\" (UID: \"ecfffbc9-4809-49a3-aed3-35b771fcb388\") " pod="openshift-ingress/e2e-ipp-b-018375-openshift-default-7fbc48965-7mlcx"
Nov 02 03:29:44.358982 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:44.358957    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecfffbc9-4809-49a3-aed3-35b771fcb388-istio-envoy\") pod \"e2e-ipp-b-018375-openshift-default-7fbc48965-7mlcx\" (UID: \"ecfffbc9-4809-49a3-aed3-35b771fcb388\") " pod="openshift-ingress/e2e-ipp-b-018375-openshift-default-7fbc48965-7mlcx"
Nov 02 03:29:44.359771 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:44.359747    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecfffbc9-4809-49a3-aed3-35b771fcb388-istio-podinfo\") pod \"e2e-ipp-b-018375-openshift-default-7fbc48965-7mlcx\" (UID: \"ecfffbc9-4809-49a3-aed3-35b771fcb388\") " pod="openshift-ingress/e2e-ipp-b-018375-openshift-default-7fbc48965-7mlcx"
Nov 02 03:29:44.414303 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:44.414234    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecfffbc9-4809-49a3-aed3-35b771fcb388-istio-token\") pod \"e2e-ipp-b-018375-openshift-default-7fbc48965-7mlcx\" (UID: \"ecfffbc9-4809-49a3-aed3-35b771fcb388\") " pod="openshift-ingress/e2e-ipp-b-018375-openshift-default-7fbc48965-7mlcx"
Nov 02 03:29:44.414947 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:44.414922    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sjp2\" (UniqueName: \"kubernetes.io/projected/ecfffbc9-4809-49a3-aed3-35b771fcb388-kube-api-access-2sjp2\") pod \"e2e-ipp-b-018375-openshift-default-7fbc48965-7mlcx\" (UID: \"ecfffbc9-4809-49a3-aed3-35b771fcb388\") " pod="openshift-ingress/e2e-ipp-b-018375-openshift-default-7fbc48965-7mlcx"
Nov 02 03:29:44.454036 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:44.453667    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-018375-openshift-default-7fbc48965-7mlcx"
Nov 02 03:29:44.925301 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:44.924543    4496 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 03:29:44.925301 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:44.924632    4496 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 03:29:44.925301 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:44.924677    4496 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 03:29:44.925885 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:44.925858    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-018375-openshift-default-7fbc48965-7mlcx"]
Nov 02 03:29:45.328773 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:45.328736    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-018375-openshift-default-7fbc48965-7mlcx" event={"ID":"ecfffbc9-4809-49a3-aed3-35b771fcb388","Type":"ContainerStarted","Data":"04e1a6d2ce978d286dad971f200a968a4b40fbf2ffdd3299875e06eafd008708"}
Nov 02 03:29:45.329311 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:45.328802    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-018375-openshift-default-7fbc48965-7mlcx" event={"ID":"ecfffbc9-4809-49a3-aed3-35b771fcb388","Type":"ContainerStarted","Data":"c1c011a71df6103706a22e81e242eecda6f6b5709cca49085455f0b34ab03557"}
Nov 02 03:29:45.449587 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:45.449496    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-018375-openshift-default-7fbc48965-7mlcx" podStartSLOduration=1.449473374 podStartE2EDuration="1.449473374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:29:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:29:45.448529347 +0000 UTC m=+2526.027136730" watchObservedRunningTime="2026-11-02 03:29:45.449473374 +0000 UTC m=+2526.028080754"
Nov 02 03:29:45.454737 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:45.454708    4496 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-018375-openshift-default-7fbc48965-7mlcx"
Nov 02 03:29:46.460941 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:46.460908    4496 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-018375-openshift-default-7fbc48965-7mlcx"
Nov 02 03:29:47.351037 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:47.346662    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-018375-openshift-default-7fbc48965-7mlcx"
Nov 02 03:29:47.352274 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:47.352246    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-018375-openshift-default-7fbc48965-7mlcx"
Nov 02 03:29:49.231615 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:49.231584    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d94cc3-774b67d9f-knmz4"
Nov 02 03:29:54.738428 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:54.738390    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-018375-778658cdb7-cr9bl"]
Nov 02 03:29:54.742660 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:54.742643    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-018375-778658cdb7-cr9bl"
Nov 02 03:29:54.808162 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:54.806405    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-018375-778658cdb7-cr9bl"]
Nov 02 03:29:54.890446 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:54.888748    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-018375\""
Nov 02 03:29:54.890446 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:54.889436    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-018375-dockercfg-qdtns\""
Nov 02 03:29:54.910028 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:54.909979    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjkw5\" (UniqueName: \"kubernetes.io/projected/4880f85e-a7cd-43b2-8040-9ca6dbae82ae-kube-api-access-jjkw5\") pod \"payload-pre-processing-e2e-ipp-b-018375-778658cdb7-cr9bl\" (UID: \"4880f85e-a7cd-43b2-8040-9ca6dbae82ae\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-018375-778658cdb7-cr9bl"
Nov 02 03:29:54.910233 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:54.910141    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4880f85e-a7cd-43b2-8040-9ca6dbae82ae-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-018375-778658cdb7-cr9bl\" (UID: \"4880f85e-a7cd-43b2-8040-9ca6dbae82ae\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-018375-778658cdb7-cr9bl"
Nov 02 03:29:54.915779 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:54.915745    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-018375-6b46774f5f-dd49t"]
Nov 02 03:29:54.922984 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:54.922956    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-018375-6b46774f5f-dd49t"
Nov 02 03:29:54.976388 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:54.976355    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-018375-6b46774f5f-dd49t"]
Nov 02 03:29:55.011362 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:55.011281    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cssdm\" (UniqueName: \"kubernetes.io/projected/ab645ce9-6586-4caa-b502-421e3c95652d-kube-api-access-cssdm\") pod \"payload-processing-e2e-ipp-b-018375-6b46774f5f-dd49t\" (UID: \"ab645ce9-6586-4caa-b502-421e3c95652d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-018375-6b46774f5f-dd49t"
Nov 02 03:29:55.011362 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:55.011346    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab645ce9-6586-4caa-b502-421e3c95652d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-018375-6b46774f5f-dd49t\" (UID: \"ab645ce9-6586-4caa-b502-421e3c95652d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-018375-6b46774f5f-dd49t"
Nov 02 03:29:55.011592 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:55.011406    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjkw5\" (UniqueName: \"kubernetes.io/projected/4880f85e-a7cd-43b2-8040-9ca6dbae82ae-kube-api-access-jjkw5\") pod \"payload-pre-processing-e2e-ipp-b-018375-778658cdb7-cr9bl\" (UID: \"4880f85e-a7cd-43b2-8040-9ca6dbae82ae\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-018375-778658cdb7-cr9bl"
Nov 02 03:29:55.011592 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:55.011456    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4880f85e-a7cd-43b2-8040-9ca6dbae82ae-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-018375-778658cdb7-cr9bl\" (UID: \"4880f85e-a7cd-43b2-8040-9ca6dbae82ae\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-018375-778658cdb7-cr9bl"
Nov 02 03:29:55.012138 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:55.012118    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4880f85e-a7cd-43b2-8040-9ca6dbae82ae-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-018375-778658cdb7-cr9bl\" (UID: \"4880f85e-a7cd-43b2-8040-9ca6dbae82ae\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-018375-778658cdb7-cr9bl"
Nov 02 03:29:55.069516 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:55.069488    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjkw5\" (UniqueName: \"kubernetes.io/projected/4880f85e-a7cd-43b2-8040-9ca6dbae82ae-kube-api-access-jjkw5\") pod \"payload-pre-processing-e2e-ipp-b-018375-778658cdb7-cr9bl\" (UID: \"4880f85e-a7cd-43b2-8040-9ca6dbae82ae\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-018375-778658cdb7-cr9bl"
Nov 02 03:29:55.112130 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:55.112099    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cssdm\" (UniqueName: \"kubernetes.io/projected/ab645ce9-6586-4caa-b502-421e3c95652d-kube-api-access-cssdm\") pod \"payload-processing-e2e-ipp-b-018375-6b46774f5f-dd49t\" (UID: \"ab645ce9-6586-4caa-b502-421e3c95652d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-018375-6b46774f5f-dd49t"
Nov 02 03:29:55.112284 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:55.112183    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab645ce9-6586-4caa-b502-421e3c95652d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-018375-6b46774f5f-dd49t\" (UID: \"ab645ce9-6586-4caa-b502-421e3c95652d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-018375-6b46774f5f-dd49t"
Nov 02 03:29:55.112874 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:55.112854    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab645ce9-6586-4caa-b502-421e3c95652d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-018375-6b46774f5f-dd49t\" (UID: \"ab645ce9-6586-4caa-b502-421e3c95652d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-018375-6b46774f5f-dd49t"
Nov 02 03:29:55.170108 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:55.170075    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cssdm\" (UniqueName: \"kubernetes.io/projected/ab645ce9-6586-4caa-b502-421e3c95652d-kube-api-access-cssdm\") pod \"payload-processing-e2e-ipp-b-018375-6b46774f5f-dd49t\" (UID: \"ab645ce9-6586-4caa-b502-421e3c95652d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-018375-6b46774f5f-dd49t"
Nov 02 03:29:55.239330 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:55.239301    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-018375-6b46774f5f-dd49t"
Nov 02 03:29:55.358153 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:55.357814    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-018375-778658cdb7-cr9bl"
Nov 02 03:29:55.615999 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:55.615964    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-018375-6b46774f5f-dd49t"]
Nov 02 03:29:55.616329 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:29:55.616301    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab645ce9_6586_4caa_b502_421e3c95652d.slice/crio-e16387a16c4fca72fa58d479a2b51ca370df090b972830fbbaf676272aab8b7c WatchSource:0}: Error finding container e16387a16c4fca72fa58d479a2b51ca370df090b972830fbbaf676272aab8b7c: Status 404 returned error can't find the container with id e16387a16c4fca72fa58d479a2b51ca370df090b972830fbbaf676272aab8b7c
Nov 02 03:29:55.744848 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:55.744824    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-018375-778658cdb7-cr9bl"]
Nov 02 03:29:55.746062 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:29:55.746030    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4880f85e_a7cd_43b2_8040_9ca6dbae82ae.slice/crio-49d1cf7540791754be18938539a9ce9e178f7bd520a9201b204fb11e4bd72ad4 WatchSource:0}: Error finding container 49d1cf7540791754be18938539a9ce9e178f7bd520a9201b204fb11e4bd72ad4: Status 404 returned error can't find the container with id 49d1cf7540791754be18938539a9ce9e178f7bd520a9201b204fb11e4bd72ad4
Nov 02 03:29:56.458769 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:56.458725    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-018375-6b46774f5f-dd49t" event={"ID":"ab645ce9-6586-4caa-b502-421e3c95652d","Type":"ContainerStarted","Data":"41d06a0e8a7f3cb07c6754850aa0a15da30fc9e04934c1435a6f198918baea82"}
Nov 02 03:29:56.459169 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:56.459144    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-018375-6b46774f5f-dd49t"
Nov 02 03:29:56.459346 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:56.459190    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-018375-6b46774f5f-dd49t" event={"ID":"ab645ce9-6586-4caa-b502-421e3c95652d","Type":"ContainerStarted","Data":"e16387a16c4fca72fa58d479a2b51ca370df090b972830fbbaf676272aab8b7c"}
Nov 02 03:29:56.460821 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:56.460794    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-018375-778658cdb7-cr9bl" event={"ID":"4880f85e-a7cd-43b2-8040-9ca6dbae82ae","Type":"ContainerStarted","Data":"99a397351ac2f2ed9c1c14e7afa8e4ec605c019ff86f98d6700f92b6fda718cf"}
Nov 02 03:29:56.460929 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:56.460851    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-018375-778658cdb7-cr9bl"
Nov 02 03:29:56.460929 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:56.460875    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-018375-778658cdb7-cr9bl" event={"ID":"4880f85e-a7cd-43b2-8040-9ca6dbae82ae","Type":"ContainerStarted","Data":"49d1cf7540791754be18938539a9ce9e178f7bd520a9201b204fb11e4bd72ad4"}
Nov 02 03:29:56.575859 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:56.575796    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-018375-6b46774f5f-dd49t" podStartSLOduration=2.575782001 podStartE2EDuration="2.575782001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:29:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:29:56.573229023 +0000 UTC m=+2537.151836404" watchObservedRunningTime="2026-11-02 03:29:56.575782001 +0000 UTC m=+2537.154389377"
Nov 02 03:29:56.690043 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:29:56.689805    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-018375-778658cdb7-cr9bl" podStartSLOduration=2.6897855010000002 podStartE2EDuration="2.689785501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:29:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:29:56.68760925 +0000 UTC m=+2537.266216643" watchObservedRunningTime="2026-11-02 03:29:56.689785501 +0000 UTC m=+2537.268392883"
Nov 02 03:30:00.160544 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:00.160506    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4c3ea5-29893170-gdz6l"]
Nov 02 03:30:00.165789 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:00.165765    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893170-5w4cf"]
Nov 02 03:30:00.166094 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:00.166073    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4c3ea5-29893170-gdz6l"
Nov 02 03:30:00.174489 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:00.174457    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-858236-29893170-kc5hk"]
Nov 02 03:30:00.174633 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:00.174589    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893170-5w4cf"
Nov 02 03:30:00.179516 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:00.179458    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-858236-29893170-kc5hk"
Nov 02 03:30:00.210530 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:00.210496    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-398828-29893170-7g7w5"]
Nov 02 03:30:00.214695 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:00.214672    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-6a5285-29893170-qv8v4"]
Nov 02 03:30:00.214822 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:00.214794    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-398828-29893170-7g7w5"
Nov 02 03:30:00.218857 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:00.218831    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f1a692-29893170-hl7kd"]
Nov 02 03:30:00.218987 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:00.218960    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-6a5285-29893170-qv8v4"
Nov 02 03:30:00.222969 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:00.222948    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f1a692-29893170-hl7kd"
Nov 02 03:30:00.223308 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:00.223295    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-abc348-29893170-2csv6"]
Nov 02 03:30:00.227249 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:00.227233    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-abc348-29893170-2csv6"
Nov 02 03:30:00.228472 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:00.228454    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893170-srl8r"]
Nov 02 03:30:00.232391 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:00.232374    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893170-srl8r"
Nov 02 03:30:00.233664 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:00.233640    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4c3ea5-29893170-gdz6l"]
Nov 02 03:30:00.257988 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:00.257959    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 03:30:00.264160 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:00.264134    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893170-5w4cf"]
Nov 02 03:30:00.265009 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:00.264985    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9z5bk\""
Nov 02 03:30:00.265252 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:00.265234    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 03:30:00.290199 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:00.290158    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-398828-29893170-7g7w5"]
Nov 02 03:30:00.290400 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:00.290255    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-858236-29893170-kc5hk"]
Nov 02 03:30:00.295726 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:00.292080    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-abc348-29893170-2csv6"]
Nov 02 03:30:00.302570 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:00.299367    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5mlz\" (UniqueName: \"kubernetes.io/projected/f95982ce-8006-43da-bf24-f1ad6417e935-kube-api-access-w5mlz\") pod \"maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893170-5w4cf\" (UID: \"f95982ce-8006-43da-bf24-f1ad6417e935\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893170-5w4cf"
Nov 02 03:30:00.302570 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:00.299455    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xcrj\" (UniqueName: \"kubernetes.io/projected/a81713dd-5c95-42d8-ad64-2a7be6e342a9-kube-api-access-6xcrj\") pod \"maas-api-key-cleanup-e2e-worker-w3-858236-29893170-kc5hk\" (UID: \"a81713dd-5c95-42d8-ad64-2a7be6e342a9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-858236-29893170-kc5hk"
Nov 02 03:30:00.302570 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:00.299504    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6wm6\" (UniqueName: \"kubernetes.io/projected/39d67039-b717-49ee-bb04-84c03d1cc400-kube-api-access-p6wm6\") pod \"maas-api-key-cleanup-e2e-ipp-a-abc348-29893170-2csv6\" (UID: \"39d67039-b717-49ee-bb04-84c03d1cc400\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-abc348-29893170-2csv6"
Nov 02 03:30:00.302570 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:00.299537    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a81713dd-5c95-42d8-ad64-2a7be6e342a9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-858236-29893170-kc5hk\" (UID: \"a81713dd-5c95-42d8-ad64-2a7be6e342a9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-858236-29893170-kc5hk"
Nov 02 03:30:00.302570 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:00.299568    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6ljk\" (UniqueName: \"kubernetes.io/projected/4b6e12f8-0901-40ef-89f1-bbf0f5cfc6a6-kube-api-access-z6ljk\") pod \"maas-api-key-cleanup-e2e-worker-w4-398828-29893170-7g7w5\" (UID: \"4b6e12f8-0901-40ef-89f1-bbf0f5cfc6a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-398828-29893170-7g7w5"
Nov 02 03:30:00.302570 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:00.299604    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5629aca-5c6e-4681-883a-bd854c24fe0a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-4c3ea5-29893170-gdz6l\" (UID: \"d5629aca-5c6e-4681-883a-bd854c24fe0a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4c3ea5-29893170-gdz6l"
Nov 02 03:30:00.302570 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:00.299657    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nghcg\" (UniqueName: \"kubernetes.io/projected/d5629aca-5c6e-4681-883a-bd854c24fe0a-kube-api-access-nghcg\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-4c3ea5-29893170-gdz6l\" (UID: \"d5629aca-5c6e-4681-883a-bd854c24fe0a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4c3ea5-29893170-gdz6l"
Nov 02 03:30:00.302570 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:00.299776    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f35619a0-8970-44f4-93c4-9cb89601d026-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-6a5285-29893170-qv8v4\" (UID: \"f35619a0-8970-44f4-93c4-9cb89601d026\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-6a5285-29893170-qv8v4"
Nov 02 03:30:00.302570 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:00.299827    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b6e12f8-0901-40ef-89f1-bbf0f5cfc6a6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-398828-29893170-7g7w5\" (UID: \"4b6e12f8-0901-40ef-89f1-bbf0f5cfc6a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-398828-29893170-7g7w5"
Nov 02 03:30:00.302570 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:00.299865    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbxgw\" (UniqueName: \"kubernetes.io/projected/ec2999b1-4a22-466d-94e4-76e47fce9a05-kube-api-access-wbxgw\") pod \"maas-api-key-cleanup-e2e-api-b-f1a692-29893170-hl7kd\" (UID: \"ec2999b1-4a22-466d-94e4-76e47fce9a05\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f1a692-29893170-hl7kd"
Nov 02 03:30:00.302570 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:00.299959    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec2999b1-4a22-466d-94e4-76e47fce9a05-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-f1a692-29893170-hl7kd\" (UID: \"ec2999b1-4a22-466d-94e4-76e47fce9a05\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f1a692-29893170-hl7kd"
Nov 02 03:30:00.302570 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:00.300002    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f95982ce-8006-43da-bf24-f1ad6417e935-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893170-5w4cf\" (UID: \"f95982ce-8006-43da-bf24-f1ad6417e935\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893170-5w4cf"
Nov 02 03:30:00.302570 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:00.300059    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d17fc996-41cb-4fae-bb3e-25ceefc1933c-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893170-srl8r\" (UID: \"d17fc996-41cb-4fae-bb3e-25ceefc1933c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893170-srl8r"
Nov 02 03:30:00.302570 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:00.300095    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2zjd\" (UniqueName: \"kubernetes.io/projected/f35619a0-8970-44f4-93c4-9cb89601d026-kube-api-access-q2zjd\") pod \"maas-api-key-cleanup-e2e-api-a-6a5285-29893170-qv8v4\" (UID: \"f35619a0-8970-44f4-93c4-9cb89601d026\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-6a5285-29893170-qv8v4"
Nov 02 03:30:00.302570 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:00.300167    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frzz7\" (UniqueName: \"kubernetes.io/projected/d17fc996-41cb-4fae-bb3e-25ceefc1933c-kube-api-access-frzz7\") pod \"maas-api-key-cleanup-29893170-srl8r\" (UID: \"d17fc996-41cb-4fae-bb3e-25ceefc1933c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893170-srl8r"
Nov 02 03:30:00.302570 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:00.300229    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/39d67039-b717-49ee-bb04-84c03d1cc400-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-abc348-29893170-2csv6\" (UID: \"39d67039-b717-49ee-bb04-84c03d1cc400\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-abc348-29893170-2csv6"
Nov 02 03:30:00.303478 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:00.302994    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893170-srl8r"]
Nov 02 03:30:00.342496 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:00.342365    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f1a692-29893170-hl7kd"]
Nov 02 03:30:00.358265 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:00.358234    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-6a5285-29893170-qv8v4"]
Nov 02 03:30:00.401918 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:00.401888    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b6e12f8-0901-40ef-89f1-bbf0f5cfc6a6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-398828-29893170-7g7w5\" (UID: \"4b6e12f8-0901-40ef-89f1-bbf0f5cfc6a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-398828-29893170-7g7w5"
Nov 02 03:30:00.402150 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:00.401926    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbxgw\" (UniqueName: \"kubernetes.io/projected/ec2999b1-4a22-466d-94e4-76e47fce9a05-kube-api-access-wbxgw\") pod \"maas-api-key-cleanup-e2e-api-b-f1a692-29893170-hl7kd\" (UID: \"ec2999b1-4a22-466d-94e4-76e47fce9a05\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f1a692-29893170-hl7kd"
Nov 02 03:30:00.402150 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:00.401978    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec2999b1-4a22-466d-94e4-76e47fce9a05-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-f1a692-29893170-hl7kd\" (UID: \"ec2999b1-4a22-466d-94e4-76e47fce9a05\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f1a692-29893170-hl7kd"
Nov 02 03:30:00.402150 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:00.401996    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f95982ce-8006-43da-bf24-f1ad6417e935-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893170-5w4cf\" (UID: \"f95982ce-8006-43da-bf24-f1ad6417e935\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893170-5w4cf"
Nov 02 03:30:00.402150 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:00.402050    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d17fc996-41cb-4fae-bb3e-25ceefc1933c-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893170-srl8r\" (UID: \"d17fc996-41cb-4fae-bb3e-25ceefc1933c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893170-srl8r"
Nov 02 03:30:00.402150 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:00.402078    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2zjd\" (UniqueName: \"kubernetes.io/projected/f35619a0-8970-44f4-93c4-9cb89601d026-kube-api-access-q2zjd\") pod \"maas-api-key-cleanup-e2e-api-a-6a5285-29893170-qv8v4\" (UID: \"f35619a0-8970-44f4-93c4-9cb89601d026\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-6a5285-29893170-qv8v4"
Nov 02 03:30:00.402150 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:00.402140    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frzz7\" (UniqueName: \"kubernetes.io/projected/d17fc996-41cb-4fae-bb3e-25ceefc1933c-kube-api-access-frzz7\") pod \"maas-api-key-cleanup-29893170-srl8r\" (UID: \"d17fc996-41cb-4fae-bb3e-25ceefc1933c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893170-srl8r"
Nov 02 03:30:00.402511 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:00.402194    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/39d67039-b717-49ee-bb04-84c03d1cc400-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-abc348-29893170-2csv6\" (UID: \"39d67039-b717-49ee-bb04-84c03d1cc400\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-abc348-29893170-2csv6"
Nov 02 03:30:00.402511 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:00.402240    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5mlz\" (UniqueName: \"kubernetes.io/projected/f95982ce-8006-43da-bf24-f1ad6417e935-kube-api-access-w5mlz\") pod \"maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893170-5w4cf\" (UID: \"f95982ce-8006-43da-bf24-f1ad6417e935\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893170-5w4cf"
Nov 02 03:30:00.402511 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:00.402305    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xcrj\" (UniqueName: \"kubernetes.io/projected/a81713dd-5c95-42d8-ad64-2a7be6e342a9-kube-api-access-6xcrj\") pod \"maas-api-key-cleanup-e2e-worker-w3-858236-29893170-kc5hk\" (UID: \"a81713dd-5c95-42d8-ad64-2a7be6e342a9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-858236-29893170-kc5hk"
Nov 02 03:30:00.402511 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:00.402342    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6wm6\" (UniqueName: \"kubernetes.io/projected/39d67039-b717-49ee-bb04-84c03d1cc400-kube-api-access-p6wm6\") pod \"maas-api-key-cleanup-e2e-ipp-a-abc348-29893170-2csv6\" (UID: \"39d67039-b717-49ee-bb04-84c03d1cc400\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-abc348-29893170-2csv6"
Nov 02 03:30:00.402511 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:00.402369    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a81713dd-5c95-42d8-ad64-2a7be6e342a9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-858236-29893170-kc5hk\" (UID: \"a81713dd-5c95-42d8-ad64-2a7be6e342a9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-858236-29893170-kc5hk"
Nov 02 03:30:00.402511 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:00.402395    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6ljk\" (UniqueName: \"kubernetes.io/projected/4b6e12f8-0901-40ef-89f1-bbf0f5cfc6a6-kube-api-access-z6ljk\") pod \"maas-api-key-cleanup-e2e-worker-w4-398828-29893170-7g7w5\" (UID: \"4b6e12f8-0901-40ef-89f1-bbf0f5cfc6a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-398828-29893170-7g7w5"
Nov 02 03:30:00.402511 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:00.402421    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5629aca-5c6e-4681-883a-bd854c24fe0a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-4c3ea5-29893170-gdz6l\" (UID: \"d5629aca-5c6e-4681-883a-bd854c24fe0a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4c3ea5-29893170-gdz6l"
Nov 02 03:30:00.402511 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:00.402464    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nghcg\" (UniqueName: \"kubernetes.io/projected/d5629aca-5c6e-4681-883a-bd854c24fe0a-kube-api-access-nghcg\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-4c3ea5-29893170-gdz6l\" (UID: \"d5629aca-5c6e-4681-883a-bd854c24fe0a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4c3ea5-29893170-gdz6l"
Nov 02 03:30:00.402951 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:00.402593    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f35619a0-8970-44f4-93c4-9cb89601d026-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-6a5285-29893170-qv8v4\" (UID: \"f35619a0-8970-44f4-93c4-9cb89601d026\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-6a5285-29893170-qv8v4"
Nov 02 03:30:00.402951 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:00.402774    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b6e12f8-0901-40ef-89f1-bbf0f5cfc6a6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-398828-29893170-7g7w5\" (UID: \"4b6e12f8-0901-40ef-89f1-bbf0f5cfc6a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-398828-29893170-7g7w5"
Nov 02 03:30:00.402951 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:00.402774    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f95982ce-8006-43da-bf24-f1ad6417e935-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893170-5w4cf\" (UID: \"f95982ce-8006-43da-bf24-f1ad6417e935\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893170-5w4cf"
Nov 02 03:30:00.402951 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:00.402907    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec2999b1-4a22-466d-94e4-76e47fce9a05-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-f1a692-29893170-hl7kd\" (UID: \"ec2999b1-4a22-466d-94e4-76e47fce9a05\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f1a692-29893170-hl7kd"
Nov 02 03:30:00.402951 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:00.402938    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d17fc996-41cb-4fae-bb3e-25ceefc1933c-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893170-srl8r\" (UID: \"d17fc996-41cb-4fae-bb3e-25ceefc1933c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893170-srl8r"
Nov 02 03:30:00.403252 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:00.403110    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/39d67039-b717-49ee-bb04-84c03d1cc400-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-abc348-29893170-2csv6\" (UID: \"39d67039-b717-49ee-bb04-84c03d1cc400\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-abc348-29893170-2csv6"
Nov 02 03:30:00.403316 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:00.403302    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f35619a0-8970-44f4-93c4-9cb89601d026-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-6a5285-29893170-qv8v4\" (UID: \"f35619a0-8970-44f4-93c4-9cb89601d026\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-6a5285-29893170-qv8v4"
Nov 02 03:30:00.403466 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:00.403442    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5629aca-5c6e-4681-883a-bd854c24fe0a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-4c3ea5-29893170-gdz6l\" (UID: \"d5629aca-5c6e-4681-883a-bd854c24fe0a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4c3ea5-29893170-gdz6l"
Nov 02 03:30:00.403622 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:00.403600    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a81713dd-5c95-42d8-ad64-2a7be6e342a9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-858236-29893170-kc5hk\" (UID: \"a81713dd-5c95-42d8-ad64-2a7be6e342a9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-858236-29893170-kc5hk"
Nov 02 03:30:00.462777 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:00.462689    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbxgw\" (UniqueName: \"kubernetes.io/projected/ec2999b1-4a22-466d-94e4-76e47fce9a05-kube-api-access-wbxgw\") pod \"maas-api-key-cleanup-e2e-api-b-f1a692-29893170-hl7kd\" (UID: \"ec2999b1-4a22-466d-94e4-76e47fce9a05\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f1a692-29893170-hl7kd"
Nov 02 03:30:00.463537 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:00.463511    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2zjd\" (UniqueName: \"kubernetes.io/projected/f35619a0-8970-44f4-93c4-9cb89601d026-kube-api-access-q2zjd\") pod \"maas-api-key-cleanup-e2e-api-a-6a5285-29893170-qv8v4\" (UID: \"f35619a0-8970-44f4-93c4-9cb89601d026\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-6a5285-29893170-qv8v4"
Nov 02 03:30:00.465485 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:00.465450    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frzz7\" (UniqueName: \"kubernetes.io/projected/d17fc996-41cb-4fae-bb3e-25ceefc1933c-kube-api-access-frzz7\") pod \"maas-api-key-cleanup-29893170-srl8r\" (UID: \"d17fc996-41cb-4fae-bb3e-25ceefc1933c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893170-srl8r"
Nov 02 03:30:00.466656 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:00.466047    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6wm6\" (UniqueName: \"kubernetes.io/projected/39d67039-b717-49ee-bb04-84c03d1cc400-kube-api-access-p6wm6\") pod \"maas-api-key-cleanup-e2e-ipp-a-abc348-29893170-2csv6\" (UID: \"39d67039-b717-49ee-bb04-84c03d1cc400\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-abc348-29893170-2csv6"
Nov 02 03:30:00.466656 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:00.466363    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6ljk\" (UniqueName: \"kubernetes.io/projected/4b6e12f8-0901-40ef-89f1-bbf0f5cfc6a6-kube-api-access-z6ljk\") pod \"maas-api-key-cleanup-e2e-worker-w4-398828-29893170-7g7w5\" (UID: \"4b6e12f8-0901-40ef-89f1-bbf0f5cfc6a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-398828-29893170-7g7w5"
Nov 02 03:30:00.466656 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:00.466611    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5mlz\" (UniqueName: \"kubernetes.io/projected/f95982ce-8006-43da-bf24-f1ad6417e935-kube-api-access-w5mlz\") pod \"maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893170-5w4cf\" (UID: \"f95982ce-8006-43da-bf24-f1ad6417e935\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893170-5w4cf"
Nov 02 03:30:00.466894 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:00.466867    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xcrj\" (UniqueName: \"kubernetes.io/projected/a81713dd-5c95-42d8-ad64-2a7be6e342a9-kube-api-access-6xcrj\") pod \"maas-api-key-cleanup-e2e-worker-w3-858236-29893170-kc5hk\" (UID: \"a81713dd-5c95-42d8-ad64-2a7be6e342a9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-858236-29893170-kc5hk"
Nov 02 03:30:00.467054 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:00.466991    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nghcg\" (UniqueName: \"kubernetes.io/projected/d5629aca-5c6e-4681-883a-bd854c24fe0a-kube-api-access-nghcg\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-4c3ea5-29893170-gdz6l\" (UID: \"d5629aca-5c6e-4681-883a-bd854c24fe0a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4c3ea5-29893170-gdz6l"
Nov 02 03:30:00.485535 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:00.485508    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4c3ea5-29893170-gdz6l"
Nov 02 03:30:00.493398 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:00.493375    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893170-5w4cf"
Nov 02 03:30:00.500581 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:00.500558    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-858236-29893170-kc5hk"
Nov 02 03:30:00.534748 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:00.534517    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-398828-29893170-7g7w5"
Nov 02 03:30:00.546252 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:00.546217    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-6a5285-29893170-qv8v4"
Nov 02 03:30:00.562426 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:00.559590    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f1a692-29893170-hl7kd"
Nov 02 03:30:00.605791 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:00.605361    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-abc348-29893170-2csv6"
Nov 02 03:30:00.621190 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:00.620683    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893170-srl8r"
Nov 02 03:30:00.891152 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:00.891121    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4c3ea5-29893170-gdz6l"]
Nov 02 03:30:00.891947 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:30:00.891915    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5629aca_5c6e_4681_883a_bd854c24fe0a.slice/crio-601222f7263335fbcfa772462e5e6ff4532d74f91d225a446b2144d7bfe07d0d WatchSource:0}: Error finding container 601222f7263335fbcfa772462e5e6ff4532d74f91d225a446b2144d7bfe07d0d: Status 404 returned error can't find the container with id 601222f7263335fbcfa772462e5e6ff4532d74f91d225a446b2144d7bfe07d0d
Nov 02 03:30:01.052056 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:30:01.050777    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf35619a0_8970_44f4_93c4_9cb89601d026.slice/crio-20a7108af3a2e0417f2f45e3fb70eba5cf5760b8d8ed95bce0e805adc5556dce WatchSource:0}: Error finding container 20a7108af3a2e0417f2f45e3fb70eba5cf5760b8d8ed95bce0e805adc5556dce: Status 404 returned error can't find the container with id 20a7108af3a2e0417f2f45e3fb70eba5cf5760b8d8ed95bce0e805adc5556dce
Nov 02 03:30:01.052056 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:01.050813    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-6a5285-29893170-qv8v4"]
Nov 02 03:30:01.054369 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:30:01.054303    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b6e12f8_0901_40ef_89f1_bbf0f5cfc6a6.slice/crio-45fe6789fc79e9f32bfc68898449aff30830d846de85e693b3b48927e1b8e661 WatchSource:0}: Error finding container 45fe6789fc79e9f32bfc68898449aff30830d846de85e693b3b48927e1b8e661: Status 404 returned error can't find the container with id 45fe6789fc79e9f32bfc68898449aff30830d846de85e693b3b48927e1b8e661
Nov 02 03:30:01.054932 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:01.054912    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-398828-29893170-7g7w5"]
Nov 02 03:30:01.101121 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:01.101086    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-abc348-29893170-2csv6"]
Nov 02 03:30:01.101473 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:30:01.101451    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39d67039_b717_49ee_bb04_84c03d1cc400.slice/crio-5a600d9a4b3c6c96f01134743eea83a499d01bd555ddc1cedb9f67d9e8b55838 WatchSource:0}: Error finding container 5a600d9a4b3c6c96f01134743eea83a499d01bd555ddc1cedb9f67d9e8b55838: Status 404 returned error can't find the container with id 5a600d9a4b3c6c96f01134743eea83a499d01bd555ddc1cedb9f67d9e8b55838
Nov 02 03:30:01.104729 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:30:01.104703    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec2999b1_4a22_466d_94e4_76e47fce9a05.slice/crio-6dd173e2ebac817d087e2d8bd5f6dfe07b2a96017d547bb6b6a95560fa583478 WatchSource:0}: Error finding container 6dd173e2ebac817d087e2d8bd5f6dfe07b2a96017d547bb6b6a95560fa583478: Status 404 returned error can't find the container with id 6dd173e2ebac817d087e2d8bd5f6dfe07b2a96017d547bb6b6a95560fa583478
Nov 02 03:30:01.105934 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:30:01.105905    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf95982ce_8006_43da_bf24_f1ad6417e935.slice/crio-67b0bbfface4e479bab7fe7f7eacfc6446f9e778d3326de4fbf543ef4a15b139 WatchSource:0}: Error finding container 67b0bbfface4e479bab7fe7f7eacfc6446f9e778d3326de4fbf543ef4a15b139: Status 404 returned error can't find the container with id 67b0bbfface4e479bab7fe7f7eacfc6446f9e778d3326de4fbf543ef4a15b139
Nov 02 03:30:01.106206 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:01.106188    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893170-5w4cf"]
Nov 02 03:30:01.107385 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:30:01.107364    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd17fc996_41cb_4fae_bb3e_25ceefc1933c.slice/crio-817eecdddc738671c94156d3004849728c9046382c9304023fa0134d7ab1f370 WatchSource:0}: Error finding container 817eecdddc738671c94156d3004849728c9046382c9304023fa0134d7ab1f370: Status 404 returned error can't find the container with id 817eecdddc738671c94156d3004849728c9046382c9304023fa0134d7ab1f370
Nov 02 03:30:01.110705 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:01.110688    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f1a692-29893170-hl7kd"]
Nov 02 03:30:01.115709 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:01.115687    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893170-srl8r"]
Nov 02 03:30:01.159357 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:30:01.159320    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda81713dd_5c95_42d8_ad64_2a7be6e342a9.slice/crio-8db6836aa73cfb02138218442290de0daa5d4fa8508e74b490db831c42ec4f10 WatchSource:0}: Error finding container 8db6836aa73cfb02138218442290de0daa5d4fa8508e74b490db831c42ec4f10: Status 404 returned error can't find the container with id 8db6836aa73cfb02138218442290de0daa5d4fa8508e74b490db831c42ec4f10
Nov 02 03:30:01.161913 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:01.161884    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-858236-29893170-kc5hk"]
Nov 02 03:30:01.504568 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:01.504532    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f1a692-29893170-hl7kd" event={"ID":"ec2999b1-4a22-466d-94e4-76e47fce9a05","Type":"ContainerStarted","Data":"6dd173e2ebac817d087e2d8bd5f6dfe07b2a96017d547bb6b6a95560fa583478"}
Nov 02 03:30:01.505871 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:01.505846    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-398828-29893170-7g7w5" event={"ID":"4b6e12f8-0901-40ef-89f1-bbf0f5cfc6a6","Type":"ContainerStarted","Data":"45fe6789fc79e9f32bfc68898449aff30830d846de85e693b3b48927e1b8e661"}
Nov 02 03:30:01.507295 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:01.507270    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-858236-29893170-kc5hk" event={"ID":"a81713dd-5c95-42d8-ad64-2a7be6e342a9","Type":"ContainerStarted","Data":"8db6836aa73cfb02138218442290de0daa5d4fa8508e74b490db831c42ec4f10"}
Nov 02 03:30:01.508807 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:01.508785    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4c3ea5-29893170-gdz6l" event={"ID":"d5629aca-5c6e-4681-883a-bd854c24fe0a","Type":"ContainerStarted","Data":"3aa41acea44cbee232d1c1eee6ffd071fe9b28bb294ed8b1542cf2b28e0b5bb7"}
Nov 02 03:30:01.508876 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:01.508825    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4c3ea5-29893170-gdz6l" event={"ID":"d5629aca-5c6e-4681-883a-bd854c24fe0a","Type":"ContainerStarted","Data":"601222f7263335fbcfa772462e5e6ff4532d74f91d225a446b2144d7bfe07d0d"}
Nov 02 03:30:01.510137 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:01.510116    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-abc348-29893170-2csv6" event={"ID":"39d67039-b717-49ee-bb04-84c03d1cc400","Type":"ContainerStarted","Data":"5a600d9a4b3c6c96f01134743eea83a499d01bd555ddc1cedb9f67d9e8b55838"}
Nov 02 03:30:01.511421 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:01.511398    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893170-srl8r" event={"ID":"d17fc996-41cb-4fae-bb3e-25ceefc1933c","Type":"ContainerStarted","Data":"817eecdddc738671c94156d3004849728c9046382c9304023fa0134d7ab1f370"}
Nov 02 03:30:01.512583 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:01.512545    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-6a5285-29893170-qv8v4" event={"ID":"f35619a0-8970-44f4-93c4-9cb89601d026","Type":"ContainerStarted","Data":"20a7108af3a2e0417f2f45e3fb70eba5cf5760b8d8ed95bce0e805adc5556dce"}
Nov 02 03:30:01.513827 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:01.513809    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893170-5w4cf" event={"ID":"f95982ce-8006-43da-bf24-f1ad6417e935","Type":"ContainerStarted","Data":"67b0bbfface4e479bab7fe7f7eacfc6446f9e778d3326de4fbf543ef4a15b139"}
Nov 02 03:30:01.625508 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:01.625400    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4c3ea5-29893170-gdz6l" podStartSLOduration=1.6253861870000001 podStartE2EDuration="1.625386187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:30:01.622347364 +0000 UTC m=+2542.200954726" watchObservedRunningTime="2026-11-02 03:30:01.625386187 +0000 UTC m=+2542.203993566"
Nov 02 03:30:02.522944 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:02.522909    4496 generic.go:356] "Generic (PLEG): container finished" podID="4b6e12f8-0901-40ef-89f1-bbf0f5cfc6a6" containerID="33e55a76b9655642d6ddedbc40ec5acbaee8c0ed5806659809599322b044e564" exitCode=60
Nov 02 03:30:02.523384 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:02.522996    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-398828-29893170-7g7w5" event={"ID":"4b6e12f8-0901-40ef-89f1-bbf0f5cfc6a6","Type":"ContainerDied","Data":"33e55a76b9655642d6ddedbc40ec5acbaee8c0ed5806659809599322b044e564"}
Nov 02 03:30:02.523384 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:02.523345    4496 scope.go:122] "RemoveContainer" containerID="33e55a76b9655642d6ddedbc40ec5acbaee8c0ed5806659809599322b044e564"
Nov 02 03:30:02.525123 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:02.525058    4496 generic.go:356] "Generic (PLEG): container finished" podID="a81713dd-5c95-42d8-ad64-2a7be6e342a9" containerID="9a085f1f38ee9f1c8f4f1bbac0eb3dcedd76858b9753829dfb6d3d514d39845a" exitCode=60
Nov 02 03:30:02.525171 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:02.525155    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-858236-29893170-kc5hk" event={"ID":"a81713dd-5c95-42d8-ad64-2a7be6e342a9","Type":"ContainerDied","Data":"9a085f1f38ee9f1c8f4f1bbac0eb3dcedd76858b9753829dfb6d3d514d39845a"}
Nov 02 03:30:02.525393 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:02.525375    4496 scope.go:122] "RemoveContainer" containerID="9a085f1f38ee9f1c8f4f1bbac0eb3dcedd76858b9753829dfb6d3d514d39845a"
Nov 02 03:30:02.527104 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:02.527058    4496 generic.go:356] "Generic (PLEG): container finished" podID="d5629aca-5c6e-4681-883a-bd854c24fe0a" containerID="3aa41acea44cbee232d1c1eee6ffd071fe9b28bb294ed8b1542cf2b28e0b5bb7" exitCode=60
Nov 02 03:30:02.527184 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:02.527162    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4c3ea5-29893170-gdz6l" event={"ID":"d5629aca-5c6e-4681-883a-bd854c24fe0a","Type":"ContainerDied","Data":"3aa41acea44cbee232d1c1eee6ffd071fe9b28bb294ed8b1542cf2b28e0b5bb7"}
Nov 02 03:30:02.527383 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:02.527365    4496 scope.go:122] "RemoveContainer" containerID="3aa41acea44cbee232d1c1eee6ffd071fe9b28bb294ed8b1542cf2b28e0b5bb7"
Nov 02 03:30:02.528990 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:02.528973    4496 generic.go:356] "Generic (PLEG): container finished" podID="39d67039-b717-49ee-bb04-84c03d1cc400" containerID="6633671bc199655eff60fb1c50a54103dc87d906a1ff6f3b702e3b4ffcb9ef49" exitCode=60
Nov 02 03:30:02.529086 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:02.529057    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-abc348-29893170-2csv6" event={"ID":"39d67039-b717-49ee-bb04-84c03d1cc400","Type":"ContainerDied","Data":"6633671bc199655eff60fb1c50a54103dc87d906a1ff6f3b702e3b4ffcb9ef49"}
Nov 02 03:30:02.529297 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:02.529281    4496 scope.go:122] "RemoveContainer" containerID="6633671bc199655eff60fb1c50a54103dc87d906a1ff6f3b702e3b4ffcb9ef49"
Nov 02 03:30:02.530748 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:02.530729    4496 generic.go:356] "Generic (PLEG): container finished" podID="d17fc996-41cb-4fae-bb3e-25ceefc1933c" containerID="a02a4fb3019d6fc9cd094168497f94a093627c861181c0cd9a2766fe783de7a5" exitCode=60
Nov 02 03:30:02.530850 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:02.530811    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893170-srl8r" event={"ID":"d17fc996-41cb-4fae-bb3e-25ceefc1933c","Type":"ContainerDied","Data":"a02a4fb3019d6fc9cd094168497f94a093627c861181c0cd9a2766fe783de7a5"}
Nov 02 03:30:02.530984 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:02.530970    4496 scope.go:122] "RemoveContainer" containerID="a02a4fb3019d6fc9cd094168497f94a093627c861181c0cd9a2766fe783de7a5"
Nov 02 03:30:02.533706 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:02.533682    4496 generic.go:356] "Generic (PLEG): container finished" podID="f35619a0-8970-44f4-93c4-9cb89601d026" containerID="6b1b20505a2102200e87a442e2e8854703e8b100fe8bc4e558b303fddb0578c9" exitCode=60
Nov 02 03:30:02.533814 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:02.533748    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-6a5285-29893170-qv8v4" event={"ID":"f35619a0-8970-44f4-93c4-9cb89601d026","Type":"ContainerDied","Data":"6b1b20505a2102200e87a442e2e8854703e8b100fe8bc4e558b303fddb0578c9"}
Nov 02 03:30:02.534004 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:02.533957    4496 scope.go:122] "RemoveContainer" containerID="6b1b20505a2102200e87a442e2e8854703e8b100fe8bc4e558b303fddb0578c9"
Nov 02 03:30:02.535684 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:02.535658    4496 generic.go:356] "Generic (PLEG): container finished" podID="f95982ce-8006-43da-bf24-f1ad6417e935" containerID="6a44f5ac8e98ed09a8495017a357f66965ebae6b8b2e58c082ffbaca4ddef3ba" exitCode=60
Nov 02 03:30:02.535817 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:02.535802    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893170-5w4cf" event={"ID":"f95982ce-8006-43da-bf24-f1ad6417e935","Type":"ContainerDied","Data":"6a44f5ac8e98ed09a8495017a357f66965ebae6b8b2e58c082ffbaca4ddef3ba"}
Nov 02 03:30:02.535944 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:02.535917    4496 scope.go:122] "RemoveContainer" containerID="6a44f5ac8e98ed09a8495017a357f66965ebae6b8b2e58c082ffbaca4ddef3ba"
Nov 02 03:30:02.544189 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:02.543927    4496 generic.go:356] "Generic (PLEG): container finished" podID="ec2999b1-4a22-466d-94e4-76e47fce9a05" containerID="5ebe594b4b4173802c93b0b636fab7ff7c8582e97985cd5857b7fbe3a08cfbbb" exitCode=60
Nov 02 03:30:02.544189 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:02.543974    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f1a692-29893170-hl7kd" event={"ID":"ec2999b1-4a22-466d-94e4-76e47fce9a05","Type":"ContainerDied","Data":"5ebe594b4b4173802c93b0b636fab7ff7c8582e97985cd5857b7fbe3a08cfbbb"}
Nov 02 03:30:02.552865 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:02.545882    4496 scope.go:122] "RemoveContainer" containerID="5ebe594b4b4173802c93b0b636fab7ff7c8582e97985cd5857b7fbe3a08cfbbb"
Nov 02 03:30:03.526470 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:30:03.526438    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 03:30:03.555462 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:03.555424    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-398828-29893170-7g7w5" event={"ID":"4b6e12f8-0901-40ef-89f1-bbf0f5cfc6a6","Type":"ContainerStarted","Data":"a9ff68a645c395653a1cb992f3484b285d768461f5444fa4c186a510339ad1bb"}
Nov 02 03:30:03.557659 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:03.557624    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-858236-29893170-kc5hk" event={"ID":"a81713dd-5c95-42d8-ad64-2a7be6e342a9","Type":"ContainerStarted","Data":"1f55040b64e95994b7c57f0d417842340d0a2f5c78cf2d999d541fe5586f1393"}
Nov 02 03:30:03.559654 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:03.559629    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4c3ea5-29893170-gdz6l" event={"ID":"d5629aca-5c6e-4681-883a-bd854c24fe0a","Type":"ContainerStarted","Data":"fa464ed118c852e26b5b338af37085993aeecd8cc50744afda305646446aaa8b"}
Nov 02 03:30:03.561557 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:03.561530    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-abc348-29893170-2csv6" event={"ID":"39d67039-b717-49ee-bb04-84c03d1cc400","Type":"ContainerStarted","Data":"5755055c2721ceea70485f1e381580a935476c2812fad03712076486aa0497ce"}
Nov 02 03:30:03.563714 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:03.563687    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893170-srl8r" event={"ID":"d17fc996-41cb-4fae-bb3e-25ceefc1933c","Type":"ContainerStarted","Data":"8547a22d128f3223a3cf8cae6a2a2046230e6a1f6020a7bab515aafaf35a87f2"}
Nov 02 03:30:03.574254 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:03.574230    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-6a5285-29893170-qv8v4" event={"ID":"f35619a0-8970-44f4-93c4-9cb89601d026","Type":"ContainerStarted","Data":"e032e7d7d294afed70abd75acbc6fecbd5e33dffdc23ab4f05c11f22c9c1560f"}
Nov 02 03:30:03.574346 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:03.574274    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893170-5w4cf" event={"ID":"f95982ce-8006-43da-bf24-f1ad6417e935","Type":"ContainerStarted","Data":"cc8729883a5681e8ac1487c6a1aa01252ec321e459491bde18fa6a6046d7bfc8"}
Nov 02 03:30:03.574346 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:03.574294    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f1a692-29893170-hl7kd" event={"ID":"ec2999b1-4a22-466d-94e4-76e47fce9a05","Type":"ContainerStarted","Data":"9266666ae4877211b3bdc66a7d5a79eb5f58f31942eeaf3c306cf1cbdab5eeb8"}
Nov 02 03:30:03.673028 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:03.672975    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-398828-29893170-7g7w5" podStartSLOduration=3.672961456 podStartE2EDuration="3.672961456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:30:03.669485778 +0000 UTC m=+2544.248093154" watchObservedRunningTime="2026-11-02 03:30:03.672961456 +0000 UTC m=+2544.251568836"
Nov 02 03:30:03.787163 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:03.787080    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-abc348-29893170-2csv6" podStartSLOduration=3.787066487 podStartE2EDuration="3.787066487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:30:03.783580009 +0000 UTC m=+2544.362187372" watchObservedRunningTime="2026-11-02 03:30:03.787066487 +0000 UTC m=+2544.365673867"
Nov 02 03:30:03.898036 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:03.897963    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f1a692-29893170-hl7kd" podStartSLOduration=3.897949474 podStartE2EDuration="3.897949474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:30:03.897184042 +0000 UTC m=+2544.475791422" watchObservedRunningTime="2026-11-02 03:30:03.897949474 +0000 UTC m=+2544.476556853"
Nov 02 03:30:04.012770 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:04.012717    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-6a5285-29893170-qv8v4" podStartSLOduration=4.012704257 podStartE2EDuration="4.012704257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:30:04.011565632 +0000 UTC m=+2544.590173013" watchObservedRunningTime="2026-11-02 03:30:04.012704257 +0000 UTC m=+2544.591311637"
Nov 02 03:30:04.129907 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:04.129856    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893170-5w4cf" podStartSLOduration=4.129840406 podStartE2EDuration="4.129840406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:30:04.125347353 +0000 UTC m=+2544.703954738" watchObservedRunningTime="2026-11-02 03:30:04.129840406 +0000 UTC m=+2544.708447785"
Nov 02 03:30:04.239207 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:04.239154    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893170-srl8r" podStartSLOduration=4.239141664 podStartE2EDuration="4.239141664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:30:04.23861921 +0000 UTC m=+2544.817226590" watchObservedRunningTime="2026-11-02 03:30:04.239141664 +0000 UTC m=+2544.817749043"
Nov 02 03:30:04.469683 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:04.469584    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-858236-29893170-kc5hk" podStartSLOduration=4.469571071 podStartE2EDuration="4.469571071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:30:04.467576392 +0000 UTC m=+2545.046183771" watchObservedRunningTime="2026-11-02 03:30:04.469571071 +0000 UTC m=+2545.048178454"
Nov 02 03:30:04.579963 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:04.579930    4496 generic.go:356] "Generic (PLEG): container finished" podID="a81713dd-5c95-42d8-ad64-2a7be6e342a9" containerID="1f55040b64e95994b7c57f0d417842340d0a2f5c78cf2d999d541fe5586f1393" exitCode=60
Nov 02 03:30:04.580432 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:04.580011    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-858236-29893170-kc5hk" event={"ID":"a81713dd-5c95-42d8-ad64-2a7be6e342a9","Type":"ContainerDied","Data":"1f55040b64e95994b7c57f0d417842340d0a2f5c78cf2d999d541fe5586f1393"}
Nov 02 03:30:04.580432 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:04.580090    4496 scope.go:122] "RemoveContainer" containerID="9a085f1f38ee9f1c8f4f1bbac0eb3dcedd76858b9753829dfb6d3d514d39845a"
Nov 02 03:30:04.580432 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:04.580255    4496 scope.go:122] "RemoveContainer" containerID="1f55040b64e95994b7c57f0d417842340d0a2f5c78cf2d999d541fe5586f1393"
Nov 02 03:30:04.580630 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:30:04.580532    4496 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-858236-29893170-kc5hk_odh-ai-gateway-infra(a81713dd-5c95-42d8-ad64-2a7be6e342a9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-858236-29893170-kc5hk" podUID="a81713dd-5c95-42d8-ad64-2a7be6e342a9"
Nov 02 03:30:04.582501 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:04.582471    4496 generic.go:356] "Generic (PLEG): container finished" podID="d5629aca-5c6e-4681-883a-bd854c24fe0a" containerID="fa464ed118c852e26b5b338af37085993aeecd8cc50744afda305646446aaa8b" exitCode=60
Nov 02 03:30:04.582640 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:04.582503    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4c3ea5-29893170-gdz6l" event={"ID":"d5629aca-5c6e-4681-883a-bd854c24fe0a","Type":"ContainerDied","Data":"fa464ed118c852e26b5b338af37085993aeecd8cc50744afda305646446aaa8b"}
Nov 02 03:30:04.582868 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:04.582846    4496 scope.go:122] "RemoveContainer" containerID="fa464ed118c852e26b5b338af37085993aeecd8cc50744afda305646446aaa8b"
Nov 02 03:30:04.583267 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:30:04.583237    4496 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-4c3ea5-29893170-gdz6l_odh-ai-gateway-infra(d5629aca-5c6e-4681-883a-bd854c24fe0a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4c3ea5-29893170-gdz6l" podUID="d5629aca-5c6e-4681-883a-bd854c24fe0a"
Nov 02 03:30:04.584612 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:04.584590    4496 generic.go:356] "Generic (PLEG): container finished" podID="39d67039-b717-49ee-bb04-84c03d1cc400" containerID="5755055c2721ceea70485f1e381580a935476c2812fad03712076486aa0497ce" exitCode=60
Nov 02 03:30:04.584708 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:04.584617    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-abc348-29893170-2csv6" event={"ID":"39d67039-b717-49ee-bb04-84c03d1cc400","Type":"ContainerDied","Data":"5755055c2721ceea70485f1e381580a935476c2812fad03712076486aa0497ce"}
Nov 02 03:30:04.584911 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:04.584895    4496 scope.go:122] "RemoveContainer" containerID="5755055c2721ceea70485f1e381580a935476c2812fad03712076486aa0497ce"
Nov 02 03:30:04.585248 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:30:04.585219    4496 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-abc348-29893170-2csv6_odh-ai-gateway-infra(39d67039-b717-49ee-bb04-84c03d1cc400)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-abc348-29893170-2csv6" podUID="39d67039-b717-49ee-bb04-84c03d1cc400"
Nov 02 03:30:04.586614 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:04.586594    4496 generic.go:356] "Generic (PLEG): container finished" podID="d17fc996-41cb-4fae-bb3e-25ceefc1933c" containerID="8547a22d128f3223a3cf8cae6a2a2046230e6a1f6020a7bab515aafaf35a87f2" exitCode=60
Nov 02 03:30:04.586715 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:04.586681    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893170-srl8r" event={"ID":"d17fc996-41cb-4fae-bb3e-25ceefc1933c","Type":"ContainerDied","Data":"8547a22d128f3223a3cf8cae6a2a2046230e6a1f6020a7bab515aafaf35a87f2"}
Nov 02 03:30:04.586858 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:04.586845    4496 scope.go:122] "RemoveContainer" containerID="8547a22d128f3223a3cf8cae6a2a2046230e6a1f6020a7bab515aafaf35a87f2"
Nov 02 03:30:04.587121 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:30:04.587101    4496 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893170-srl8r_odh-ai-gateway-infra(d17fc996-41cb-4fae-bb3e-25ceefc1933c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893170-srl8r" podUID="d17fc996-41cb-4fae-bb3e-25ceefc1933c"
Nov 02 03:30:04.588649 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:04.588629    4496 generic.go:356] "Generic (PLEG): container finished" podID="f35619a0-8970-44f4-93c4-9cb89601d026" containerID="e032e7d7d294afed70abd75acbc6fecbd5e33dffdc23ab4f05c11f22c9c1560f" exitCode=60
Nov 02 03:30:04.588735 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:04.588666    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-6a5285-29893170-qv8v4" event={"ID":"f35619a0-8970-44f4-93c4-9cb89601d026","Type":"ContainerDied","Data":"e032e7d7d294afed70abd75acbc6fecbd5e33dffdc23ab4f05c11f22c9c1560f"}
Nov 02 03:30:04.588936 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:04.588917    4496 scope.go:122] "RemoveContainer" containerID="e032e7d7d294afed70abd75acbc6fecbd5e33dffdc23ab4f05c11f22c9c1560f"
Nov 02 03:30:04.589171 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:30:04.589148    4496 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-6a5285-29893170-qv8v4_odh-ai-gateway-infra(f35619a0-8970-44f4-93c4-9cb89601d026)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-6a5285-29893170-qv8v4" podUID="f35619a0-8970-44f4-93c4-9cb89601d026"
Nov 02 03:30:04.590683 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:04.590663    4496 generic.go:356] "Generic (PLEG): container finished" podID="f95982ce-8006-43da-bf24-f1ad6417e935" containerID="cc8729883a5681e8ac1487c6a1aa01252ec321e459491bde18fa6a6046d7bfc8" exitCode=60
Nov 02 03:30:04.590773 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:04.590686    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893170-5w4cf" event={"ID":"f95982ce-8006-43da-bf24-f1ad6417e935","Type":"ContainerDied","Data":"cc8729883a5681e8ac1487c6a1aa01252ec321e459491bde18fa6a6046d7bfc8"}
Nov 02 03:30:04.590916 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:04.590901    4496 scope.go:122] "RemoveContainer" containerID="cc8729883a5681e8ac1487c6a1aa01252ec321e459491bde18fa6a6046d7bfc8"
Nov 02 03:30:04.591142 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:30:04.591123    4496 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893170-5w4cf_odh-ai-gateway-infra(f95982ce-8006-43da-bf24-f1ad6417e935)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893170-5w4cf" podUID="f95982ce-8006-43da-bf24-f1ad6417e935"
Nov 02 03:30:04.592711 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:04.592687    4496 generic.go:356] "Generic (PLEG): container finished" podID="ec2999b1-4a22-466d-94e4-76e47fce9a05" containerID="9266666ae4877211b3bdc66a7d5a79eb5f58f31942eeaf3c306cf1cbdab5eeb8" exitCode=60
Nov 02 03:30:04.592809 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:04.592768    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f1a692-29893170-hl7kd" event={"ID":"ec2999b1-4a22-466d-94e4-76e47fce9a05","Type":"ContainerDied","Data":"9266666ae4877211b3bdc66a7d5a79eb5f58f31942eeaf3c306cf1cbdab5eeb8"}
Nov 02 03:30:04.592980 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:04.592964    4496 scope.go:122] "RemoveContainer" containerID="9266666ae4877211b3bdc66a7d5a79eb5f58f31942eeaf3c306cf1cbdab5eeb8"
Nov 02 03:30:04.593204 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:30:04.593185    4496 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-f1a692-29893170-hl7kd_odh-ai-gateway-infra(ec2999b1-4a22-466d-94e4-76e47fce9a05)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f1a692-29893170-hl7kd" podUID="ec2999b1-4a22-466d-94e4-76e47fce9a05"
Nov 02 03:30:04.594711 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:04.594687    4496 generic.go:356] "Generic (PLEG): container finished" podID="4b6e12f8-0901-40ef-89f1-bbf0f5cfc6a6" containerID="a9ff68a645c395653a1cb992f3484b285d768461f5444fa4c186a510339ad1bb" exitCode=60
Nov 02 03:30:04.594806 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:04.594717    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-398828-29893170-7g7w5" event={"ID":"4b6e12f8-0901-40ef-89f1-bbf0f5cfc6a6","Type":"ContainerDied","Data":"a9ff68a645c395653a1cb992f3484b285d768461f5444fa4c186a510339ad1bb"}
Nov 02 03:30:04.594921 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:04.594906    4496 scope.go:122] "RemoveContainer" containerID="a9ff68a645c395653a1cb992f3484b285d768461f5444fa4c186a510339ad1bb"
Nov 02 03:30:04.595115 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:30:04.595095    4496 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-398828-29893170-7g7w5_odh-ai-gateway-infra(4b6e12f8-0901-40ef-89f1-bbf0f5cfc6a6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-398828-29893170-7g7w5" podUID="4b6e12f8-0901-40ef-89f1-bbf0f5cfc6a6"
Nov 02 03:30:04.595672 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:04.595657    4496 scope.go:122] "RemoveContainer" containerID="3aa41acea44cbee232d1c1eee6ffd071fe9b28bb294ed8b1542cf2b28e0b5bb7"
Nov 02 03:30:04.608563 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:04.608537    4496 scope.go:122] "RemoveContainer" containerID="6633671bc199655eff60fb1c50a54103dc87d906a1ff6f3b702e3b4ffcb9ef49"
Nov 02 03:30:04.621339 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:04.621319    4496 scope.go:122] "RemoveContainer" containerID="a02a4fb3019d6fc9cd094168497f94a093627c861181c0cd9a2766fe783de7a5"
Nov 02 03:30:04.634598 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:04.634584    4496 scope.go:122] "RemoveContainer" containerID="6b1b20505a2102200e87a442e2e8854703e8b100fe8bc4e558b303fddb0578c9"
Nov 02 03:30:04.647324 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:04.647303    4496 scope.go:122] "RemoveContainer" containerID="6a44f5ac8e98ed09a8495017a357f66965ebae6b8b2e58c082ffbaca4ddef3ba"
Nov 02 03:30:04.659749 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:04.659728    4496 scope.go:122] "RemoveContainer" containerID="5ebe594b4b4173802c93b0b636fab7ff7c8582e97985cd5857b7fbe3a08cfbbb"
Nov 02 03:30:04.672243 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:04.672227    4496 scope.go:122] "RemoveContainer" containerID="33e55a76b9655642d6ddedbc40ec5acbaee8c0ed5806659809599322b044e564"
Nov 02 03:30:05.603185 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:05.603159    4496 scope.go:122] "RemoveContainer" containerID="a9ff68a645c395653a1cb992f3484b285d768461f5444fa4c186a510339ad1bb"
Nov 02 03:30:05.603617 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:30:05.603419    4496 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-398828-29893170-7g7w5_odh-ai-gateway-infra(4b6e12f8-0901-40ef-89f1-bbf0f5cfc6a6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-398828-29893170-7g7w5" podUID="4b6e12f8-0901-40ef-89f1-bbf0f5cfc6a6"
Nov 02 03:30:05.604756 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:05.604734    4496 scope.go:122] "RemoveContainer" containerID="1f55040b64e95994b7c57f0d417842340d0a2f5c78cf2d999d541fe5586f1393"
Nov 02 03:30:05.604986 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:30:05.604961    4496 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-858236-29893170-kc5hk_odh-ai-gateway-infra(a81713dd-5c95-42d8-ad64-2a7be6e342a9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-858236-29893170-kc5hk" podUID="a81713dd-5c95-42d8-ad64-2a7be6e342a9"
Nov 02 03:30:05.606549 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:05.606525    4496 scope.go:122] "RemoveContainer" containerID="fa464ed118c852e26b5b338af37085993aeecd8cc50744afda305646446aaa8b"
Nov 02 03:30:05.606795 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:30:05.606771    4496 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-4c3ea5-29893170-gdz6l_odh-ai-gateway-infra(d5629aca-5c6e-4681-883a-bd854c24fe0a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4c3ea5-29893170-gdz6l" podUID="d5629aca-5c6e-4681-883a-bd854c24fe0a"
Nov 02 03:30:05.608118 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:05.608088    4496 scope.go:122] "RemoveContainer" containerID="5755055c2721ceea70485f1e381580a935476c2812fad03712076486aa0497ce"
Nov 02 03:30:05.608311 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:30:05.608292    4496 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-abc348-29893170-2csv6_odh-ai-gateway-infra(39d67039-b717-49ee-bb04-84c03d1cc400)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-abc348-29893170-2csv6" podUID="39d67039-b717-49ee-bb04-84c03d1cc400"
Nov 02 03:30:05.609698 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:05.609680    4496 scope.go:122] "RemoveContainer" containerID="8547a22d128f3223a3cf8cae6a2a2046230e6a1f6020a7bab515aafaf35a87f2"
Nov 02 03:30:05.609890 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:30:05.609867    4496 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893170-srl8r_odh-ai-gateway-infra(d17fc996-41cb-4fae-bb3e-25ceefc1933c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893170-srl8r" podUID="d17fc996-41cb-4fae-bb3e-25ceefc1933c"
Nov 02 03:30:05.611384 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:05.611364    4496 scope.go:122] "RemoveContainer" containerID="e032e7d7d294afed70abd75acbc6fecbd5e33dffdc23ab4f05c11f22c9c1560f"
Nov 02 03:30:05.611621 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:30:05.611597    4496 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-6a5285-29893170-qv8v4_odh-ai-gateway-infra(f35619a0-8970-44f4-93c4-9cb89601d026)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-6a5285-29893170-qv8v4" podUID="f35619a0-8970-44f4-93c4-9cb89601d026"
Nov 02 03:30:05.613000 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:05.612984    4496 scope.go:122] "RemoveContainer" containerID="cc8729883a5681e8ac1487c6a1aa01252ec321e459491bde18fa6a6046d7bfc8"
Nov 02 03:30:05.613202 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:30:05.613183    4496 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893170-5w4cf_odh-ai-gateway-infra(f95982ce-8006-43da-bf24-f1ad6417e935)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893170-5w4cf" podUID="f95982ce-8006-43da-bf24-f1ad6417e935"
Nov 02 03:30:05.614574 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:05.614559    4496 scope.go:122] "RemoveContainer" containerID="9266666ae4877211b3bdc66a7d5a79eb5f58f31942eeaf3c306cf1cbdab5eeb8"
Nov 02 03:30:05.614755 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:30:05.614735    4496 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-f1a692-29893170-hl7kd_odh-ai-gateway-infra(ec2999b1-4a22-466d-94e4-76e47fce9a05)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f1a692-29893170-hl7kd" podUID="ec2999b1-4a22-466d-94e4-76e47fce9a05"
Nov 02 03:30:07.473470 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:07.473442    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-018375-6b46774f5f-dd49t"
Nov 02 03:30:07.473840 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:07.473515    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-018375-778658cdb7-cr9bl"
Nov 02 03:30:14.567256 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:14.567222    4496 scope.go:122] "RemoveContainer" containerID="fa464ed118c852e26b5b338af37085993aeecd8cc50744afda305646446aaa8b"
Nov 02 03:30:14.567725 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:14.567302    4496 scope.go:122] "RemoveContainer" containerID="5755055c2721ceea70485f1e381580a935476c2812fad03712076486aa0497ce"
Nov 02 03:30:15.611058 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:15.608800    4496 scope.go:122] "RemoveContainer" containerID="cc8729883a5681e8ac1487c6a1aa01252ec321e459491bde18fa6a6046d7bfc8"
Nov 02 03:30:15.697725 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:15.697647    4496 generic.go:356] "Generic (PLEG): container finished" podID="d5629aca-5c6e-4681-883a-bd854c24fe0a" containerID="7d5fd4a0b2b9695e39ffbf06dc5e06d6a7b35804cb03af1d401854d676e92910" exitCode=60
Nov 02 03:30:15.697725 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:15.697699    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4c3ea5-29893170-gdz6l" event={"ID":"d5629aca-5c6e-4681-883a-bd854c24fe0a","Type":"ContainerDied","Data":"7d5fd4a0b2b9695e39ffbf06dc5e06d6a7b35804cb03af1d401854d676e92910"}
Nov 02 03:30:15.697917 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:15.697790    4496 scope.go:122] "RemoveContainer" containerID="fa464ed118c852e26b5b338af37085993aeecd8cc50744afda305646446aaa8b"
Nov 02 03:30:15.698188 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:15.698166    4496 scope.go:122] "RemoveContainer" containerID="7d5fd4a0b2b9695e39ffbf06dc5e06d6a7b35804cb03af1d401854d676e92910"
Nov 02 03:30:15.698512 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:30:15.698487    4496 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-4c3ea5-29893170-gdz6l_odh-ai-gateway-infra(d5629aca-5c6e-4681-883a-bd854c24fe0a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4c3ea5-29893170-gdz6l" podUID="d5629aca-5c6e-4681-883a-bd854c24fe0a"
Nov 02 03:30:15.699991 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:15.699970    4496 generic.go:356] "Generic (PLEG): container finished" podID="39d67039-b717-49ee-bb04-84c03d1cc400" containerID="c4f48dcad24f94aacd9f9da79ca71836ee7310fc3846713a3d2903115e4b83d7" exitCode=60
Nov 02 03:30:15.700081 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:15.700061    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-abc348-29893170-2csv6" event={"ID":"39d67039-b717-49ee-bb04-84c03d1cc400","Type":"ContainerDied","Data":"c4f48dcad24f94aacd9f9da79ca71836ee7310fc3846713a3d2903115e4b83d7"}
Nov 02 03:30:15.700382 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:15.700369    4496 scope.go:122] "RemoveContainer" containerID="c4f48dcad24f94aacd9f9da79ca71836ee7310fc3846713a3d2903115e4b83d7"
Nov 02 03:30:15.700632 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:30:15.700616    4496 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-abc348-29893170-2csv6_odh-ai-gateway-infra(39d67039-b717-49ee-bb04-84c03d1cc400)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-abc348-29893170-2csv6" podUID="39d67039-b717-49ee-bb04-84c03d1cc400"
Nov 02 03:30:15.712908 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:15.712887    4496 scope.go:122] "RemoveContainer" containerID="5755055c2721ceea70485f1e381580a935476c2812fad03712076486aa0497ce"
Nov 02 03:30:16.566124 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:16.566096    4496 scope.go:122] "RemoveContainer" containerID="1f55040b64e95994b7c57f0d417842340d0a2f5c78cf2d999d541fe5586f1393"
Nov 02 03:30:16.712359 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:16.712282    4496 generic.go:356] "Generic (PLEG): container finished" podID="f95982ce-8006-43da-bf24-f1ad6417e935" containerID="74b06fa205d2b2be65fbd41b69ed3f695b6fc7e59c78aac9bc8d6e5a3b5b7326" exitCode=60
Nov 02 03:30:16.712714 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:16.712355    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893170-5w4cf" event={"ID":"f95982ce-8006-43da-bf24-f1ad6417e935","Type":"ContainerDied","Data":"74b06fa205d2b2be65fbd41b69ed3f695b6fc7e59c78aac9bc8d6e5a3b5b7326"}
Nov 02 03:30:16.712714 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:16.712416    4496 scope.go:122] "RemoveContainer" containerID="cc8729883a5681e8ac1487c6a1aa01252ec321e459491bde18fa6a6046d7bfc8"
Nov 02 03:30:16.712816 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:16.712736    4496 scope.go:122] "RemoveContainer" containerID="74b06fa205d2b2be65fbd41b69ed3f695b6fc7e59c78aac9bc8d6e5a3b5b7326"
Nov 02 03:30:16.713063 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:30:16.713009    4496 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893170-5w4cf_odh-ai-gateway-infra(f95982ce-8006-43da-bf24-f1ad6417e935)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893170-5w4cf" podUID="f95982ce-8006-43da-bf24-f1ad6417e935"
Nov 02 03:30:16.826113 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:16.826083    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4c3ea5-29893170-gdz6l"]
Nov 02 03:30:16.951054 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:16.941979    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-abc348-29893170-2csv6"]
Nov 02 03:30:16.987304 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:16.987278    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4c3ea5-29893170-gdz6l"
Nov 02 03:30:17.041054 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:17.037658    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5629aca-5c6e-4681-883a-bd854c24fe0a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5629aca-5c6e-4681-883a-bd854c24fe0a-maas-api-service-ca\") pod \"d5629aca-5c6e-4681-883a-bd854c24fe0a\" (UID: \"d5629aca-5c6e-4681-883a-bd854c24fe0a\") "
Nov 02 03:30:17.041054 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:17.037733    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5629aca-5c6e-4681-883a-bd854c24fe0a-kube-api-access-nghcg\" (UniqueName: \"kubernetes.io/projected/d5629aca-5c6e-4681-883a-bd854c24fe0a-kube-api-access-nghcg\") pod \"d5629aca-5c6e-4681-883a-bd854c24fe0a\" (UID: \"d5629aca-5c6e-4681-883a-bd854c24fe0a\") "
Nov 02 03:30:17.041054 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:17.038571    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5629aca-5c6e-4681-883a-bd854c24fe0a-maas-api-service-ca" pod "d5629aca-5c6e-4681-883a-bd854c24fe0a" (UID: "d5629aca-5c6e-4681-883a-bd854c24fe0a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:30:17.041054 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:17.040575    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5629aca-5c6e-4681-883a-bd854c24fe0a-kube-api-access-nghcg" pod "d5629aca-5c6e-4681-883a-bd854c24fe0a" (UID: "d5629aca-5c6e-4681-883a-bd854c24fe0a"). InnerVolumeSpecName "kube-api-access-nghcg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:30:17.111085 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:17.111058    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-abc348-29893170-2csv6"
Nov 02 03:30:17.139796 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:17.139755    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5629aca-5c6e-4681-883a-bd854c24fe0a-maas-api-service-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:30:17.139796 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:17.139791    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nghcg\" (UniqueName: \"kubernetes.io/projected/d5629aca-5c6e-4681-883a-bd854c24fe0a-kube-api-access-nghcg\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:30:17.241251 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:17.241211    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39d67039-b717-49ee-bb04-84c03d1cc400-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/39d67039-b717-49ee-bb04-84c03d1cc400-maas-api-service-ca\") pod \"39d67039-b717-49ee-bb04-84c03d1cc400\" (UID: \"39d67039-b717-49ee-bb04-84c03d1cc400\") "
Nov 02 03:30:17.241449 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:17.241275    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39d67039-b717-49ee-bb04-84c03d1cc400-kube-api-access-p6wm6\" (UniqueName: \"kubernetes.io/projected/39d67039-b717-49ee-bb04-84c03d1cc400-kube-api-access-p6wm6\") pod \"39d67039-b717-49ee-bb04-84c03d1cc400\" (UID: \"39d67039-b717-49ee-bb04-84c03d1cc400\") "
Nov 02 03:30:17.241563 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:17.241539    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39d67039-b717-49ee-bb04-84c03d1cc400-maas-api-service-ca" pod "39d67039-b717-49ee-bb04-84c03d1cc400" (UID: "39d67039-b717-49ee-bb04-84c03d1cc400"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:30:17.241797 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:17.241781    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/39d67039-b717-49ee-bb04-84c03d1cc400-maas-api-service-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:30:17.243360 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:17.243332    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39d67039-b717-49ee-bb04-84c03d1cc400-kube-api-access-p6wm6" pod "39d67039-b717-49ee-bb04-84c03d1cc400" (UID: "39d67039-b717-49ee-bb04-84c03d1cc400"). InnerVolumeSpecName "kube-api-access-p6wm6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:30:17.343494 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:17.343451    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6wm6\" (UniqueName: \"kubernetes.io/projected/39d67039-b717-49ee-bb04-84c03d1cc400-kube-api-access-p6wm6\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:30:17.613057 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:17.609720    4496 scope.go:122] "RemoveContainer" containerID="a9ff68a645c395653a1cb992f3484b285d768461f5444fa4c186a510339ad1bb"
Nov 02 03:30:17.619347 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:17.619313    4496 scope.go:122] "RemoveContainer" containerID="9266666ae4877211b3bdc66a7d5a79eb5f58f31942eeaf3c306cf1cbdab5eeb8"
Nov 02 03:30:17.625777 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:17.623994    4496 scope.go:122] "RemoveContainer" containerID="8547a22d128f3223a3cf8cae6a2a2046230e6a1f6020a7bab515aafaf35a87f2"
Nov 02 03:30:17.625777 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:17.624673    4496 scope.go:122] "RemoveContainer" containerID="e032e7d7d294afed70abd75acbc6fecbd5e33dffdc23ab4f05c11f22c9c1560f"
Nov 02 03:30:17.730615 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:17.730580    4496 generic.go:356] "Generic (PLEG): container finished" podID="a81713dd-5c95-42d8-ad64-2a7be6e342a9" containerID="c6840476e556398a280c2f27922b2b180355f3fa1fe50bffa6f134e10a1ec420" exitCode=60
Nov 02 03:30:17.731123 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:17.730649    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-858236-29893170-kc5hk" event={"ID":"a81713dd-5c95-42d8-ad64-2a7be6e342a9","Type":"ContainerDied","Data":"c6840476e556398a280c2f27922b2b180355f3fa1fe50bffa6f134e10a1ec420"}
Nov 02 03:30:17.731123 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:17.730719    4496 scope.go:122] "RemoveContainer" containerID="1f55040b64e95994b7c57f0d417842340d0a2f5c78cf2d999d541fe5586f1393"
Nov 02 03:30:17.731691 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:17.731661    4496 scope.go:122] "RemoveContainer" containerID="c6840476e556398a280c2f27922b2b180355f3fa1fe50bffa6f134e10a1ec420"
Nov 02 03:30:17.732095 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:30:17.732059    4496 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-858236-29893170-kc5hk_odh-ai-gateway-infra(a81713dd-5c95-42d8-ad64-2a7be6e342a9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-858236-29893170-kc5hk" podUID="a81713dd-5c95-42d8-ad64-2a7be6e342a9"
Nov 02 03:30:17.737110 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:17.736309    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4c3ea5-29893170-gdz6l" event={"ID":"d5629aca-5c6e-4681-883a-bd854c24fe0a","Type":"ContainerDied","Data":"601222f7263335fbcfa772462e5e6ff4532d74f91d225a446b2144d7bfe07d0d"}
Nov 02 03:30:17.737110 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:17.736419    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4c3ea5-29893170-gdz6l"
Nov 02 03:30:17.743089 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:17.742389    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-abc348-29893170-2csv6" event={"ID":"39d67039-b717-49ee-bb04-84c03d1cc400","Type":"ContainerDied","Data":"5a600d9a4b3c6c96f01134743eea83a499d01bd555ddc1cedb9f67d9e8b55838"}
Nov 02 03:30:17.743089 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:17.742509    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-abc348-29893170-2csv6"
Nov 02 03:30:17.754505 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:17.754436    4496 scope.go:122] "RemoveContainer" containerID="7d5fd4a0b2b9695e39ffbf06dc5e06d6a7b35804cb03af1d401854d676e92910"
Nov 02 03:30:17.771528 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:17.771503    4496 scope.go:122] "RemoveContainer" containerID="c4f48dcad24f94aacd9f9da79ca71836ee7310fc3846713a3d2903115e4b83d7"
Nov 02 03:30:17.840835 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:17.840800    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893170-5w4cf"]
Nov 02 03:30:18.028233 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:18.028193    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893170-5w4cf"
Nov 02 03:30:18.062058 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:18.061972    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f95982ce-8006-43da-bf24-f1ad6417e935-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f95982ce-8006-43da-bf24-f1ad6417e935-maas-api-service-ca\") pod \"f95982ce-8006-43da-bf24-f1ad6417e935\" (UID: \"f95982ce-8006-43da-bf24-f1ad6417e935\") "
Nov 02 03:30:18.062183 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:18.062160    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f95982ce-8006-43da-bf24-f1ad6417e935-kube-api-access-w5mlz\" (UniqueName: \"kubernetes.io/projected/f95982ce-8006-43da-bf24-f1ad6417e935-kube-api-access-w5mlz\") pod \"f95982ce-8006-43da-bf24-f1ad6417e935\" (UID: \"f95982ce-8006-43da-bf24-f1ad6417e935\") "
Nov 02 03:30:18.062361 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:18.062337    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f95982ce-8006-43da-bf24-f1ad6417e935-maas-api-service-ca" pod "f95982ce-8006-43da-bf24-f1ad6417e935" (UID: "f95982ce-8006-43da-bf24-f1ad6417e935"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:30:18.062573 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:18.062554    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f95982ce-8006-43da-bf24-f1ad6417e935-maas-api-service-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:30:18.080791 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:18.080756    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4c3ea5-29893170-gdz6l"]
Nov 02 03:30:18.084392 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:18.084362    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4c3ea5-29893170-gdz6l"]
Nov 02 03:30:18.133818 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:18.133776    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f95982ce-8006-43da-bf24-f1ad6417e935-kube-api-access-w5mlz" pod "f95982ce-8006-43da-bf24-f1ad6417e935" (UID: "f95982ce-8006-43da-bf24-f1ad6417e935"). InnerVolumeSpecName "kube-api-access-w5mlz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:30:18.163521 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:18.163480    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5mlz\" (UniqueName: \"kubernetes.io/projected/f95982ce-8006-43da-bf24-f1ad6417e935-kube-api-access-w5mlz\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:30:18.199566 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:18.199522    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-abc348-29893170-2csv6"]
Nov 02 03:30:18.205074 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:18.205039    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-abc348-29893170-2csv6"]
Nov 02 03:30:18.763251 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:18.762198    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f1a692-29893170-hl7kd" event={"ID":"ec2999b1-4a22-466d-94e4-76e47fce9a05","Type":"ContainerStarted","Data":"5f67316e88da5ff0cae3cd7f548d695046c42bb3a83a67f134ea45b8083cdbdf"}
Nov 02 03:30:18.767578 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:18.767547    4496 generic.go:356] "Generic (PLEG): container finished" podID="4b6e12f8-0901-40ef-89f1-bbf0f5cfc6a6" containerID="67daeaf0a67c1b335cc8a0c57770458d0406addd55f3c4c03f5509c75d2fc095" exitCode=60
Nov 02 03:30:18.767754 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:18.767633    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-398828-29893170-7g7w5" event={"ID":"4b6e12f8-0901-40ef-89f1-bbf0f5cfc6a6","Type":"ContainerDied","Data":"67daeaf0a67c1b335cc8a0c57770458d0406addd55f3c4c03f5509c75d2fc095"}
Nov 02 03:30:18.767754 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:18.767686    4496 scope.go:122] "RemoveContainer" containerID="a9ff68a645c395653a1cb992f3484b285d768461f5444fa4c186a510339ad1bb"
Nov 02 03:30:18.768256 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:18.768236    4496 scope.go:122] "RemoveContainer" containerID="67daeaf0a67c1b335cc8a0c57770458d0406addd55f3c4c03f5509c75d2fc095"
Nov 02 03:30:18.768792 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:30:18.768761    4496 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-398828-29893170-7g7w5_odh-ai-gateway-infra(4b6e12f8-0901-40ef-89f1-bbf0f5cfc6a6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-398828-29893170-7g7w5" podUID="4b6e12f8-0901-40ef-89f1-bbf0f5cfc6a6"
Nov 02 03:30:18.784293 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:18.784262    4496 generic.go:356] "Generic (PLEG): container finished" podID="d17fc996-41cb-4fae-bb3e-25ceefc1933c" containerID="111ae587b25803a31d73fc54b507402309f55e27ea25e4b20f2bb1b63ba5af9a" exitCode=60
Nov 02 03:30:18.784439 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:18.784398    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893170-srl8r" event={"ID":"d17fc996-41cb-4fae-bb3e-25ceefc1933c","Type":"ContainerDied","Data":"111ae587b25803a31d73fc54b507402309f55e27ea25e4b20f2bb1b63ba5af9a"}
Nov 02 03:30:18.784851 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:18.784811    4496 scope.go:122] "RemoveContainer" containerID="111ae587b25803a31d73fc54b507402309f55e27ea25e4b20f2bb1b63ba5af9a"
Nov 02 03:30:18.785392 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:30:18.785362    4496 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29893170-srl8r_odh-ai-gateway-infra(d17fc996-41cb-4fae-bb3e-25ceefc1933c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893170-srl8r" podUID="d17fc996-41cb-4fae-bb3e-25ceefc1933c"
Nov 02 03:30:18.792896 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:18.791707    4496 generic.go:356] "Generic (PLEG): container finished" podID="f35619a0-8970-44f4-93c4-9cb89601d026" containerID="1ddb10673fb9b4916cde2fd83f47f30d21f261bef441afcf501a7e07dc5de9d2" exitCode=60
Nov 02 03:30:18.792896 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:18.791878    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-6a5285-29893170-qv8v4" event={"ID":"f35619a0-8970-44f4-93c4-9cb89601d026","Type":"ContainerDied","Data":"1ddb10673fb9b4916cde2fd83f47f30d21f261bef441afcf501a7e07dc5de9d2"}
Nov 02 03:30:18.792896 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:18.792490    4496 scope.go:122] "RemoveContainer" containerID="1ddb10673fb9b4916cde2fd83f47f30d21f261bef441afcf501a7e07dc5de9d2"
Nov 02 03:30:18.792896 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:30:18.792832    4496 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-6a5285-29893170-qv8v4_odh-ai-gateway-infra(f35619a0-8970-44f4-93c4-9cb89601d026)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-6a5285-29893170-qv8v4" podUID="f35619a0-8970-44f4-93c4-9cb89601d026"
Nov 02 03:30:18.796968 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:18.796923    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893170-5w4cf"
Nov 02 03:30:18.796968 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:18.796930    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893170-5w4cf" event={"ID":"f95982ce-8006-43da-bf24-f1ad6417e935","Type":"ContainerDied","Data":"67b0bbfface4e479bab7fe7f7eacfc6446f9e778d3326de4fbf543ef4a15b139"}
Nov 02 03:30:18.812865 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:18.812839    4496 scope.go:122] "RemoveContainer" containerID="8547a22d128f3223a3cf8cae6a2a2046230e6a1f6020a7bab515aafaf35a87f2"
Nov 02 03:30:18.831251 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:18.831227    4496 scope.go:122] "RemoveContainer" containerID="e032e7d7d294afed70abd75acbc6fecbd5e33dffdc23ab4f05c11f22c9c1560f"
Nov 02 03:30:18.846036 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:18.846004    4496 scope.go:122] "RemoveContainer" containerID="74b06fa205d2b2be65fbd41b69ed3f695b6fc7e59c78aac9bc8d6e5a3b5b7326"
Nov 02 03:30:18.862995 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:18.862970    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-858236-29893170-kc5hk"]
Nov 02 03:30:19.011915 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:19.011889    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-858236-29893170-kc5hk"
Nov 02 03:30:19.075875 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:19.075831    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a81713dd-5c95-42d8-ad64-2a7be6e342a9-kube-api-access-6xcrj\" (UniqueName: \"kubernetes.io/projected/a81713dd-5c95-42d8-ad64-2a7be6e342a9-kube-api-access-6xcrj\") pod \"a81713dd-5c95-42d8-ad64-2a7be6e342a9\" (UID: \"a81713dd-5c95-42d8-ad64-2a7be6e342a9\") "
Nov 02 03:30:19.076028 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:19.075988    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a81713dd-5c95-42d8-ad64-2a7be6e342a9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a81713dd-5c95-42d8-ad64-2a7be6e342a9-maas-api-service-ca\") pod \"a81713dd-5c95-42d8-ad64-2a7be6e342a9\" (UID: \"a81713dd-5c95-42d8-ad64-2a7be6e342a9\") "
Nov 02 03:30:19.076314 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:19.076290    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a81713dd-5c95-42d8-ad64-2a7be6e342a9-maas-api-service-ca" pod "a81713dd-5c95-42d8-ad64-2a7be6e342a9" (UID: "a81713dd-5c95-42d8-ad64-2a7be6e342a9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:30:19.076637 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:19.076617    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a81713dd-5c95-42d8-ad64-2a7be6e342a9-maas-api-service-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:30:19.078055 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:19.077999    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a81713dd-5c95-42d8-ad64-2a7be6e342a9-kube-api-access-6xcrj" pod "a81713dd-5c95-42d8-ad64-2a7be6e342a9" (UID: "a81713dd-5c95-42d8-ad64-2a7be6e342a9"). InnerVolumeSpecName "kube-api-access-6xcrj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:30:19.110777 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:19.110751    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893170-5w4cf"]
Nov 02 03:30:19.114357 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:19.114335    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893170-5w4cf"]
Nov 02 03:30:19.177355 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:19.177290    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xcrj\" (UniqueName: \"kubernetes.io/projected/a81713dd-5c95-42d8-ad64-2a7be6e342a9-kube-api-access-6xcrj\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:30:19.577506 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:19.577469    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39d67039-b717-49ee-bb04-84c03d1cc400" path="/var/lib/kubelet/pods/39d67039-b717-49ee-bb04-84c03d1cc400/volumes"
Nov 02 03:30:19.577976 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:19.577960    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5629aca-5c6e-4681-883a-bd854c24fe0a" path="/var/lib/kubelet/pods/d5629aca-5c6e-4681-883a-bd854c24fe0a/volumes"
Nov 02 03:30:19.578385 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:19.578371    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f95982ce-8006-43da-bf24-f1ad6417e935" path="/var/lib/kubelet/pods/f95982ce-8006-43da-bf24-f1ad6417e935/volumes"
Nov 02 03:30:19.807338 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:19.807303    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-858236-29893170-kc5hk"
Nov 02 03:30:19.807338 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:19.807316    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-858236-29893170-kc5hk" event={"ID":"a81713dd-5c95-42d8-ad64-2a7be6e342a9","Type":"ContainerDied","Data":"8db6836aa73cfb02138218442290de0daa5d4fa8508e74b490db831c42ec4f10"}
Nov 02 03:30:19.807863 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:19.807406    4496 scope.go:122] "RemoveContainer" containerID="c6840476e556398a280c2f27922b2b180355f3fa1fe50bffa6f134e10a1ec420"
Nov 02 03:30:19.814305 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:19.814280    4496 generic.go:356] "Generic (PLEG): container finished" podID="ec2999b1-4a22-466d-94e4-76e47fce9a05" containerID="5f67316e88da5ff0cae3cd7f548d695046c42bb3a83a67f134ea45b8083cdbdf" exitCode=60
Nov 02 03:30:19.814426 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:19.814314    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f1a692-29893170-hl7kd" event={"ID":"ec2999b1-4a22-466d-94e4-76e47fce9a05","Type":"ContainerDied","Data":"5f67316e88da5ff0cae3cd7f548d695046c42bb3a83a67f134ea45b8083cdbdf"}
Nov 02 03:30:19.814688 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:19.814670    4496 scope.go:122] "RemoveContainer" containerID="5f67316e88da5ff0cae3cd7f548d695046c42bb3a83a67f134ea45b8083cdbdf"
Nov 02 03:30:19.814939 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:30:19.814918    4496 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-f1a692-29893170-hl7kd_odh-ai-gateway-infra(ec2999b1-4a22-466d-94e4-76e47fce9a05)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f1a692-29893170-hl7kd" podUID="ec2999b1-4a22-466d-94e4-76e47fce9a05"
Nov 02 03:30:19.819119 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:19.819054    4496 scope.go:122] "RemoveContainer" containerID="9266666ae4877211b3bdc66a7d5a79eb5f58f31942eeaf3c306cf1cbdab5eeb8"
Nov 02 03:30:19.892119 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:19.892086    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f1a692-29893170-hl7kd"]
Nov 02 03:30:20.008549 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:20.008513    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-6a5285-29893170-qv8v4"]
Nov 02 03:30:20.092501 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:20.092466    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-858236-29893170-kc5hk"]
Nov 02 03:30:20.106706 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:20.106673    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-858236-29893170-kc5hk"]
Nov 02 03:30:20.195935 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:20.195909    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-6a5285-29893170-qv8v4"
Nov 02 03:30:20.236836 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:20.236802    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893170-srl8r"]
Nov 02 03:30:20.290197 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:20.290158    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f35619a0-8970-44f4-93c4-9cb89601d026-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f35619a0-8970-44f4-93c4-9cb89601d026-maas-api-service-ca\") pod \"f35619a0-8970-44f4-93c4-9cb89601d026\" (UID: \"f35619a0-8970-44f4-93c4-9cb89601d026\") "
Nov 02 03:30:20.290371 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:20.290355    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f35619a0-8970-44f4-93c4-9cb89601d026-kube-api-access-q2zjd\" (UniqueName: \"kubernetes.io/projected/f35619a0-8970-44f4-93c4-9cb89601d026-kube-api-access-q2zjd\") pod \"f35619a0-8970-44f4-93c4-9cb89601d026\" (UID: \"f35619a0-8970-44f4-93c4-9cb89601d026\") "
Nov 02 03:30:20.290661 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:20.290631    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f35619a0-8970-44f4-93c4-9cb89601d026-maas-api-service-ca" pod "f35619a0-8970-44f4-93c4-9cb89601d026" (UID: "f35619a0-8970-44f4-93c4-9cb89601d026"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:30:20.291085 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:20.291046    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f35619a0-8970-44f4-93c4-9cb89601d026-maas-api-service-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:30:20.292705 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:20.292675    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f35619a0-8970-44f4-93c4-9cb89601d026-kube-api-access-q2zjd" pod "f35619a0-8970-44f4-93c4-9cb89601d026" (UID: "f35619a0-8970-44f4-93c4-9cb89601d026"). InnerVolumeSpecName "kube-api-access-q2zjd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:30:20.355359 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:20.355274    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-398828-29893170-7g7w5"]
Nov 02 03:30:20.394231 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:20.393140    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2zjd\" (UniqueName: \"kubernetes.io/projected/f35619a0-8970-44f4-93c4-9cb89601d026-kube-api-access-q2zjd\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:30:20.447712 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:20.447685    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893170-srl8r"
Nov 02 03:30:20.495449 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:20.494585    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d17fc996-41cb-4fae-bb3e-25ceefc1933c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d17fc996-41cb-4fae-bb3e-25ceefc1933c-maas-api-service-ca\") pod \"d17fc996-41cb-4fae-bb3e-25ceefc1933c\" (UID: \"d17fc996-41cb-4fae-bb3e-25ceefc1933c\") "
Nov 02 03:30:20.495449 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:20.494752    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d17fc996-41cb-4fae-bb3e-25ceefc1933c-kube-api-access-frzz7\" (UniqueName: \"kubernetes.io/projected/d17fc996-41cb-4fae-bb3e-25ceefc1933c-kube-api-access-frzz7\") pod \"d17fc996-41cb-4fae-bb3e-25ceefc1933c\" (UID: \"d17fc996-41cb-4fae-bb3e-25ceefc1933c\") "
Nov 02 03:30:20.496133 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:20.496063    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d17fc996-41cb-4fae-bb3e-25ceefc1933c-maas-api-service-ca" pod "d17fc996-41cb-4fae-bb3e-25ceefc1933c" (UID: "d17fc996-41cb-4fae-bb3e-25ceefc1933c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:30:20.497760 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:20.497730    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d17fc996-41cb-4fae-bb3e-25ceefc1933c-kube-api-access-frzz7" pod "d17fc996-41cb-4fae-bb3e-25ceefc1933c" (UID: "d17fc996-41cb-4fae-bb3e-25ceefc1933c"). InnerVolumeSpecName "kube-api-access-frzz7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:30:20.559008 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:20.558984    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-398828-29893170-7g7w5"
Nov 02 03:30:20.597703 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:20.597664    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d17fc996-41cb-4fae-bb3e-25ceefc1933c-maas-api-service-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:30:20.597703 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:20.597704    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frzz7\" (UniqueName: \"kubernetes.io/projected/d17fc996-41cb-4fae-bb3e-25ceefc1933c-kube-api-access-frzz7\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:30:20.698476 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:20.698379    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b6e12f8-0901-40ef-89f1-bbf0f5cfc6a6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b6e12f8-0901-40ef-89f1-bbf0f5cfc6a6-maas-api-service-ca\") pod \"4b6e12f8-0901-40ef-89f1-bbf0f5cfc6a6\" (UID: \"4b6e12f8-0901-40ef-89f1-bbf0f5cfc6a6\") "
Nov 02 03:30:20.698639 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:20.698519    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b6e12f8-0901-40ef-89f1-bbf0f5cfc6a6-kube-api-access-z6ljk\" (UniqueName: \"kubernetes.io/projected/4b6e12f8-0901-40ef-89f1-bbf0f5cfc6a6-kube-api-access-z6ljk\") pod \"4b6e12f8-0901-40ef-89f1-bbf0f5cfc6a6\" (UID: \"4b6e12f8-0901-40ef-89f1-bbf0f5cfc6a6\") "
Nov 02 03:30:20.698795 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:20.698769    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b6e12f8-0901-40ef-89f1-bbf0f5cfc6a6-maas-api-service-ca" pod "4b6e12f8-0901-40ef-89f1-bbf0f5cfc6a6" (UID: "4b6e12f8-0901-40ef-89f1-bbf0f5cfc6a6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:30:20.699169 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:20.699149    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b6e12f8-0901-40ef-89f1-bbf0f5cfc6a6-maas-api-service-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:30:20.700922 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:20.700896    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b6e12f8-0901-40ef-89f1-bbf0f5cfc6a6-kube-api-access-z6ljk" pod "4b6e12f8-0901-40ef-89f1-bbf0f5cfc6a6" (UID: "4b6e12f8-0901-40ef-89f1-bbf0f5cfc6a6"). InnerVolumeSpecName "kube-api-access-z6ljk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:30:20.800773 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:20.800733    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6ljk\" (UniqueName: \"kubernetes.io/projected/4b6e12f8-0901-40ef-89f1-bbf0f5cfc6a6-kube-api-access-z6ljk\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:30:20.831183 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:20.831155    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893170-srl8r"
Nov 02 03:30:20.831621 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:20.831146    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893170-srl8r" event={"ID":"d17fc996-41cb-4fae-bb3e-25ceefc1933c","Type":"ContainerDied","Data":"817eecdddc738671c94156d3004849728c9046382c9304023fa0134d7ab1f370"}
Nov 02 03:30:20.831621 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:20.831289    4496 scope.go:122] "RemoveContainer" containerID="111ae587b25803a31d73fc54b507402309f55e27ea25e4b20f2bb1b63ba5af9a"
Nov 02 03:30:20.833191 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:20.833167    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-6a5285-29893170-qv8v4"
Nov 02 03:30:20.833754 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:20.833194    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-6a5285-29893170-qv8v4" event={"ID":"f35619a0-8970-44f4-93c4-9cb89601d026","Type":"ContainerDied","Data":"20a7108af3a2e0417f2f45e3fb70eba5cf5760b8d8ed95bce0e805adc5556dce"}
Nov 02 03:30:20.838858 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:20.838827    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-398828-29893170-7g7w5" event={"ID":"4b6e12f8-0901-40ef-89f1-bbf0f5cfc6a6","Type":"ContainerDied","Data":"45fe6789fc79e9f32bfc68898449aff30830d846de85e693b3b48927e1b8e661"}
Nov 02 03:30:20.838988 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:20.838951    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-398828-29893170-7g7w5"
Nov 02 03:30:20.855286 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:20.855239    4496 scope.go:122] "RemoveContainer" containerID="1ddb10673fb9b4916cde2fd83f47f30d21f261bef441afcf501a7e07dc5de9d2"
Nov 02 03:30:20.874168 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:20.874138    4496 scope.go:122] "RemoveContainer" containerID="67daeaf0a67c1b335cc8a0c57770458d0406addd55f3c4c03f5509c75d2fc095"
Nov 02 03:30:20.990116 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:20.990087    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f1a692-29893170-hl7kd"
Nov 02 03:30:21.025837 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:21.025791    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893170-srl8r"]
Nov 02 03:30:21.096312 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:21.096272    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893170-srl8r"]
Nov 02 03:30:21.116833 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:21.116799    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec2999b1-4a22-466d-94e4-76e47fce9a05-kube-api-access-wbxgw\" (UniqueName: \"kubernetes.io/projected/ec2999b1-4a22-466d-94e4-76e47fce9a05-kube-api-access-wbxgw\") pod \"ec2999b1-4a22-466d-94e4-76e47fce9a05\" (UID: \"ec2999b1-4a22-466d-94e4-76e47fce9a05\") "
Nov 02 03:30:21.117007 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:21.116973    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec2999b1-4a22-466d-94e4-76e47fce9a05-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec2999b1-4a22-466d-94e4-76e47fce9a05-maas-api-service-ca\") pod \"ec2999b1-4a22-466d-94e4-76e47fce9a05\" (UID: \"ec2999b1-4a22-466d-94e4-76e47fce9a05\") "
Nov 02 03:30:21.117739 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:21.117711    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec2999b1-4a22-466d-94e4-76e47fce9a05-maas-api-service-ca" pod "ec2999b1-4a22-466d-94e4-76e47fce9a05" (UID: "ec2999b1-4a22-466d-94e4-76e47fce9a05"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:30:21.120411 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:21.120387    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec2999b1-4a22-466d-94e4-76e47fce9a05-kube-api-access-wbxgw" pod "ec2999b1-4a22-466d-94e4-76e47fce9a05" (UID: "ec2999b1-4a22-466d-94e4-76e47fce9a05"). InnerVolumeSpecName "kube-api-access-wbxgw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:30:21.160721 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:21.160693    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-398828-29893170-7g7w5"]
Nov 02 03:30:21.166508 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:21.166476    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-398828-29893170-7g7w5"]
Nov 02 03:30:21.218416 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:21.218385    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec2999b1-4a22-466d-94e4-76e47fce9a05-maas-api-service-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:30:21.218650 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:21.218622    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbxgw\" (UniqueName: \"kubernetes.io/projected/ec2999b1-4a22-466d-94e4-76e47fce9a05-kube-api-access-wbxgw\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:30:21.386641 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:21.386565    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-6a5285-29893170-qv8v4"]
Nov 02 03:30:21.389944 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:21.389914    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-6a5285-29893170-qv8v4"]
Nov 02 03:30:21.574803 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:21.574768    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b6e12f8-0901-40ef-89f1-bbf0f5cfc6a6" path="/var/lib/kubelet/pods/4b6e12f8-0901-40ef-89f1-bbf0f5cfc6a6/volumes"
Nov 02 03:30:21.575430 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:21.575398    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a81713dd-5c95-42d8-ad64-2a7be6e342a9" path="/var/lib/kubelet/pods/a81713dd-5c95-42d8-ad64-2a7be6e342a9/volumes"
Nov 02 03:30:21.576137 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:21.576109    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d17fc996-41cb-4fae-bb3e-25ceefc1933c" path="/var/lib/kubelet/pods/d17fc996-41cb-4fae-bb3e-25ceefc1933c/volumes"
Nov 02 03:30:21.577296 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:21.577274    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f35619a0-8970-44f4-93c4-9cb89601d026" path="/var/lib/kubelet/pods/f35619a0-8970-44f4-93c4-9cb89601d026/volumes"
Nov 02 03:30:21.858196 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:21.858162    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f1a692-29893170-hl7kd"
Nov 02 03:30:21.858631 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:21.858195    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f1a692-29893170-hl7kd" event={"ID":"ec2999b1-4a22-466d-94e4-76e47fce9a05","Type":"ContainerDied","Data":"6dd173e2ebac817d087e2d8bd5f6dfe07b2a96017d547bb6b6a95560fa583478"}
Nov 02 03:30:21.858631 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:21.858263    4496 scope.go:122] "RemoveContainer" containerID="5f67316e88da5ff0cae3cd7f548d695046c42bb3a83a67f134ea45b8083cdbdf"
Nov 02 03:30:22.031413 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:22.031364    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f1a692-29893170-hl7kd"]
Nov 02 03:30:22.036742 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:22.036707    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f1a692-29893170-hl7kd"]
Nov 02 03:30:23.577389 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:23.577293    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec2999b1-4a22-466d-94e4-76e47fce9a05" path="/var/lib/kubelet/pods/ec2999b1-4a22-466d-94e4-76e47fce9a05/volumes"
Nov 02 03:30:30.624299 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:30.624259    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9d5625-54769b969-tfkvv"]
Nov 02 03:30:30.625929 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:30.625899    4496 cpu_manager.go:414] "containerMap: removing container" podUID="f95982ce-8006-43da-bf24-f1ad6417e935" containerName="cleanup"
Nov 02 03:30:30.626113 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:30.626097    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f95982ce-8006-43da-bf24-f1ad6417e935" containerName="cleanup"
Nov 02 03:30:30.626225 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:30.626212    4496 cpu_manager.go:414] "containerMap: removing container" podUID="a81713dd-5c95-42d8-ad64-2a7be6e342a9" containerName="cleanup"
Nov 02 03:30:30.626313 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:30.626302    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a81713dd-5c95-42d8-ad64-2a7be6e342a9" containerName="cleanup"
Nov 02 03:30:30.626462 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:30.626449    4496 cpu_manager.go:414] "containerMap: removing container" podUID="d17fc996-41cb-4fae-bb3e-25ceefc1933c" containerName="cleanup"
Nov 02 03:30:30.626557 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:30.626544    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d17fc996-41cb-4fae-bb3e-25ceefc1933c" containerName="cleanup"
Nov 02 03:30:30.626642 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:30.626632    4496 cpu_manager.go:414] "containerMap: removing container" podUID="4b6e12f8-0901-40ef-89f1-bbf0f5cfc6a6" containerName="cleanup"
Nov 02 03:30:30.626720 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:30.626709    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b6e12f8-0901-40ef-89f1-bbf0f5cfc6a6" containerName="cleanup"
Nov 02 03:30:30.626817 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:30.626805    4496 cpu_manager.go:414] "containerMap: removing container" podUID="ec2999b1-4a22-466d-94e4-76e47fce9a05" containerName="cleanup"
Nov 02 03:30:30.626900 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:30.626890    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec2999b1-4a22-466d-94e4-76e47fce9a05" containerName="cleanup"
Nov 02 03:30:30.626984 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:30.626973    4496 cpu_manager.go:414] "containerMap: removing container" podUID="a81713dd-5c95-42d8-ad64-2a7be6e342a9" containerName="cleanup"
Nov 02 03:30:30.627084 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:30.627071    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a81713dd-5c95-42d8-ad64-2a7be6e342a9" containerName="cleanup"
Nov 02 03:30:30.627180 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:30.627168    4496 cpu_manager.go:414] "containerMap: removing container" podUID="39d67039-b717-49ee-bb04-84c03d1cc400" containerName="cleanup"
Nov 02 03:30:30.627262 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:30.627251    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39d67039-b717-49ee-bb04-84c03d1cc400" containerName="cleanup"
Nov 02 03:30:30.627346 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:30.627336    4496 cpu_manager.go:414] "containerMap: removing container" podUID="39d67039-b717-49ee-bb04-84c03d1cc400" containerName="cleanup"
Nov 02 03:30:30.627427 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:30.627416    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39d67039-b717-49ee-bb04-84c03d1cc400" containerName="cleanup"
Nov 02 03:30:30.627528 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:30.627517    4496 cpu_manager.go:414] "containerMap: removing container" podUID="d5629aca-5c6e-4681-883a-bd854c24fe0a" containerName="cleanup"
Nov 02 03:30:30.627605 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:30.627594    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5629aca-5c6e-4681-883a-bd854c24fe0a" containerName="cleanup"
Nov 02 03:30:30.627684 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:30.627674    4496 cpu_manager.go:414] "containerMap: removing container" podUID="d17fc996-41cb-4fae-bb3e-25ceefc1933c" containerName="cleanup"
Nov 02 03:30:30.627771 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:30.627761    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d17fc996-41cb-4fae-bb3e-25ceefc1933c" containerName="cleanup"
Nov 02 03:30:30.627860 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:30.627850    4496 cpu_manager.go:414] "containerMap: removing container" podUID="f35619a0-8970-44f4-93c4-9cb89601d026" containerName="cleanup"
Nov 02 03:30:30.627940 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:30.627930    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f35619a0-8970-44f4-93c4-9cb89601d026" containerName="cleanup"
Nov 02 03:30:30.628041 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:30.628029    4496 cpu_manager.go:414] "containerMap: removing container" podUID="d17fc996-41cb-4fae-bb3e-25ceefc1933c" containerName="cleanup"
Nov 02 03:30:30.628123 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:30.628112    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d17fc996-41cb-4fae-bb3e-25ceefc1933c" containerName="cleanup"
Nov 02 03:30:30.628215 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:30.628204    4496 cpu_manager.go:414] "containerMap: removing container" podUID="f35619a0-8970-44f4-93c4-9cb89601d026" containerName="cleanup"
Nov 02 03:30:30.628292 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:30.628281    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f35619a0-8970-44f4-93c4-9cb89601d026" containerName="cleanup"
Nov 02 03:30:30.628438 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:30.628425    4496 cpu_manager.go:414] "containerMap: removing container" podUID="ec2999b1-4a22-466d-94e4-76e47fce9a05" containerName="cleanup"
Nov 02 03:30:30.628537 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:30.628524    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec2999b1-4a22-466d-94e4-76e47fce9a05" containerName="cleanup"
Nov 02 03:30:30.628641 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:30.628630    4496 cpu_manager.go:414] "containerMap: removing container" podUID="4b6e12f8-0901-40ef-89f1-bbf0f5cfc6a6" containerName="cleanup"
Nov 02 03:30:30.628790 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:30.628777    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b6e12f8-0901-40ef-89f1-bbf0f5cfc6a6" containerName="cleanup"
Nov 02 03:30:30.628871 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:30.628861    4496 cpu_manager.go:414] "containerMap: removing container" podUID="ec2999b1-4a22-466d-94e4-76e47fce9a05" containerName="cleanup"
Nov 02 03:30:30.628948 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:30.628938    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec2999b1-4a22-466d-94e4-76e47fce9a05" containerName="cleanup"
Nov 02 03:30:30.629053 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:30.629035    4496 cpu_manager.go:414] "containerMap: removing container" podUID="d5629aca-5c6e-4681-883a-bd854c24fe0a" containerName="cleanup"
Nov 02 03:30:30.629217 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:30.629203    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5629aca-5c6e-4681-883a-bd854c24fe0a" containerName="cleanup"
Nov 02 03:30:30.629325 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:30.629313    4496 cpu_manager.go:414] "containerMap: removing container" podUID="d5629aca-5c6e-4681-883a-bd854c24fe0a" containerName="cleanup"
Nov 02 03:30:30.629404 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:30.629393    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5629aca-5c6e-4681-883a-bd854c24fe0a" containerName="cleanup"
Nov 02 03:30:30.629668 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:30.629651    4496 cpu_manager.go:414] "containerMap: removing container" podUID="f95982ce-8006-43da-bf24-f1ad6417e935" containerName="cleanup"
Nov 02 03:30:30.644401 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:30.644375    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f95982ce-8006-43da-bf24-f1ad6417e935" containerName="cleanup"
Nov 02 03:30:30.654064 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:30.647881    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="39d67039-b717-49ee-bb04-84c03d1cc400" containerName="cleanup"
Nov 02 03:30:30.654064 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:30.648040    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5629aca-5c6e-4681-883a-bd854c24fe0a" containerName="cleanup"
Nov 02 03:30:30.654064 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:30.648056    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="d17fc996-41cb-4fae-bb3e-25ceefc1933c" containerName="cleanup"
Nov 02 03:30:30.654064 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:30.648071    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="f35619a0-8970-44f4-93c4-9cb89601d026" containerName="cleanup"
Nov 02 03:30:30.654064 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:30.648082    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="d17fc996-41cb-4fae-bb3e-25ceefc1933c" containerName="cleanup"
Nov 02 03:30:30.654064 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:30.648093    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="f35619a0-8970-44f4-93c4-9cb89601d026" containerName="cleanup"
Nov 02 03:30:30.654064 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:30.648104    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec2999b1-4a22-466d-94e4-76e47fce9a05" containerName="cleanup"
Nov 02 03:30:30.654064 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:30.648112    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b6e12f8-0901-40ef-89f1-bbf0f5cfc6a6" containerName="cleanup"
Nov 02 03:30:30.654064 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:30.648121    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b6e12f8-0901-40ef-89f1-bbf0f5cfc6a6" containerName="cleanup"
Nov 02 03:30:30.654064 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:30.648131    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="a81713dd-5c95-42d8-ad64-2a7be6e342a9" containerName="cleanup"
Nov 02 03:30:30.654064 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:30.648141    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5629aca-5c6e-4681-883a-bd854c24fe0a" containerName="cleanup"
Nov 02 03:30:30.654064 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:30.648150    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5629aca-5c6e-4681-883a-bd854c24fe0a" containerName="cleanup"
Nov 02 03:30:30.654064 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:30.648162    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="f95982ce-8006-43da-bf24-f1ad6417e935" containerName="cleanup"
Nov 02 03:30:30.654064 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:30.648171    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="f95982ce-8006-43da-bf24-f1ad6417e935" containerName="cleanup"
Nov 02 03:30:30.654064 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:30.648184    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="f95982ce-8006-43da-bf24-f1ad6417e935" containerName="cleanup"
Nov 02 03:30:30.654064 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:30.648227    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="a81713dd-5c95-42d8-ad64-2a7be6e342a9" containerName="cleanup"
Nov 02 03:30:30.654064 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:30.648236    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="d17fc996-41cb-4fae-bb3e-25ceefc1933c" containerName="cleanup"
Nov 02 03:30:30.654064 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:30.648249    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec2999b1-4a22-466d-94e4-76e47fce9a05" containerName="cleanup"
Nov 02 03:30:30.654064 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:30.648260    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="39d67039-b717-49ee-bb04-84c03d1cc400" containerName="cleanup"
Nov 02 03:30:30.656917 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:30.656894    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-9d5625-54769b969-tfkvv"
Nov 02 03:30:30.660723 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:30.660095    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9d5625-54769b969-tfkvv"]
Nov 02 03:30:30.710823 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:30.710794    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-9d5625\""
Nov 02 03:30:30.744501 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:30.744468    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a50fee2e-7a02-4b8d-a228-3f0dfd42ce31-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-9d5625-54769b969-tfkvv\" (UID: \"a50fee2e-7a02-4b8d-a228-3f0dfd42ce31\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9d5625-54769b969-tfkvv"
Nov 02 03:30:30.744689 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:30.744670    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mjkc\" (UniqueName: \"kubernetes.io/projected/a50fee2e-7a02-4b8d-a228-3f0dfd42ce31-kube-api-access-4mjkc\") pod \"payload-processing-e2e-ipp-b-9d5625-54769b969-tfkvv\" (UID: \"a50fee2e-7a02-4b8d-a228-3f0dfd42ce31\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9d5625-54769b969-tfkvv"
Nov 02 03:30:30.761837 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:30.761806    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-9d5625-dockercfg-c2vt4\""
Nov 02 03:30:30.765584 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:30.765556    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9d5625-87d7dd87-24dnh"]
Nov 02 03:30:30.766233 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:30.766216    4496 cpu_manager.go:414] "containerMap: removing container" podUID="f35619a0-8970-44f4-93c4-9cb89601d026" containerName="cleanup"
Nov 02 03:30:30.766275 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:30.766234    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f35619a0-8970-44f4-93c4-9cb89601d026" containerName="cleanup"
Nov 02 03:30:30.766275 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:30.766262    4496 cpu_manager.go:414] "containerMap: removing container" podUID="f95982ce-8006-43da-bf24-f1ad6417e935" containerName="cleanup"
Nov 02 03:30:30.766275 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:30.766269    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f95982ce-8006-43da-bf24-f1ad6417e935" containerName="cleanup"
Nov 02 03:30:30.766373 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:30.766279    4496 cpu_manager.go:414] "containerMap: removing container" podUID="4b6e12f8-0901-40ef-89f1-bbf0f5cfc6a6" containerName="cleanup"
Nov 02 03:30:30.766373 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:30.766284    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b6e12f8-0901-40ef-89f1-bbf0f5cfc6a6" containerName="cleanup"
Nov 02 03:30:30.766373 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:30.766300    4496 cpu_manager.go:414] "containerMap: removing container" podUID="a81713dd-5c95-42d8-ad64-2a7be6e342a9" containerName="cleanup"
Nov 02 03:30:30.766373 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:30.766305    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a81713dd-5c95-42d8-ad64-2a7be6e342a9" containerName="cleanup"
Nov 02 03:30:30.766373 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:30.766315    4496 cpu_manager.go:414] "containerMap: removing container" podUID="39d67039-b717-49ee-bb04-84c03d1cc400" containerName="cleanup"
Nov 02 03:30:30.766373 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:30.766319    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39d67039-b717-49ee-bb04-84c03d1cc400" containerName="cleanup"
Nov 02 03:30:30.766555 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:30.766414    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="39d67039-b717-49ee-bb04-84c03d1cc400" containerName="cleanup"
Nov 02 03:30:30.766555 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:30.766425    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="f35619a0-8970-44f4-93c4-9cb89601d026" containerName="cleanup"
Nov 02 03:30:30.766555 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:30.766432    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec2999b1-4a22-466d-94e4-76e47fce9a05" containerName="cleanup"
Nov 02 03:30:30.766555 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:30.766438    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b6e12f8-0901-40ef-89f1-bbf0f5cfc6a6" containerName="cleanup"
Nov 02 03:30:30.766555 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:30.766444    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="a81713dd-5c95-42d8-ad64-2a7be6e342a9" containerName="cleanup"
Nov 02 03:30:30.770061 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:30.770040    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d5625-87d7dd87-24dnh"
Nov 02 03:30:30.825397 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:30.825371    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9d5625-87d7dd87-24dnh"]
Nov 02 03:30:30.846160 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:30.846136    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twl58\" (UniqueName: \"kubernetes.io/projected/24f29cce-d514-49e4-9359-e1efed242080-kube-api-access-twl58\") pod \"payload-pre-processing-e2e-ipp-b-9d5625-87d7dd87-24dnh\" (UID: \"24f29cce-d514-49e4-9359-e1efed242080\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d5625-87d7dd87-24dnh"
Nov 02 03:30:30.846333 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:30.846310    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a50fee2e-7a02-4b8d-a228-3f0dfd42ce31-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-9d5625-54769b969-tfkvv\" (UID: \"a50fee2e-7a02-4b8d-a228-3f0dfd42ce31\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9d5625-54769b969-tfkvv"
Nov 02 03:30:30.846432 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:30.846413    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24f29cce-d514-49e4-9359-e1efed242080-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-9d5625-87d7dd87-24dnh\" (UID: \"24f29cce-d514-49e4-9359-e1efed242080\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d5625-87d7dd87-24dnh"
Nov 02 03:30:30.846533 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:30.846517    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mjkc\" (UniqueName: \"kubernetes.io/projected/a50fee2e-7a02-4b8d-a228-3f0dfd42ce31-kube-api-access-4mjkc\") pod \"payload-processing-e2e-ipp-b-9d5625-54769b969-tfkvv\" (UID: \"a50fee2e-7a02-4b8d-a228-3f0dfd42ce31\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9d5625-54769b969-tfkvv"
Nov 02 03:30:30.847575 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:30.847545    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a50fee2e-7a02-4b8d-a228-3f0dfd42ce31-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-9d5625-54769b969-tfkvv\" (UID: \"a50fee2e-7a02-4b8d-a228-3f0dfd42ce31\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9d5625-54769b969-tfkvv"
Nov 02 03:30:30.904846 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:30.904776    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mjkc\" (UniqueName: \"kubernetes.io/projected/a50fee2e-7a02-4b8d-a228-3f0dfd42ce31-kube-api-access-4mjkc\") pod \"payload-processing-e2e-ipp-b-9d5625-54769b969-tfkvv\" (UID: \"a50fee2e-7a02-4b8d-a228-3f0dfd42ce31\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9d5625-54769b969-tfkvv"
Nov 02 03:30:30.947418 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:30.947382    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24f29cce-d514-49e4-9359-e1efed242080-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-9d5625-87d7dd87-24dnh\" (UID: \"24f29cce-d514-49e4-9359-e1efed242080\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d5625-87d7dd87-24dnh"
Nov 02 03:30:30.947778 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:30.947511    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twl58\" (UniqueName: \"kubernetes.io/projected/24f29cce-d514-49e4-9359-e1efed242080-kube-api-access-twl58\") pod \"payload-pre-processing-e2e-ipp-b-9d5625-87d7dd87-24dnh\" (UID: \"24f29cce-d514-49e4-9359-e1efed242080\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d5625-87d7dd87-24dnh"
Nov 02 03:30:30.948200 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:30.948176    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24f29cce-d514-49e4-9359-e1efed242080-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-9d5625-87d7dd87-24dnh\" (UID: \"24f29cce-d514-49e4-9359-e1efed242080\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d5625-87d7dd87-24dnh"
Nov 02 03:30:30.992670 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:30.992642    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-9d5625-54769b969-tfkvv"
Nov 02 03:30:31.005113 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:31.005086    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twl58\" (UniqueName: \"kubernetes.io/projected/24f29cce-d514-49e4-9359-e1efed242080-kube-api-access-twl58\") pod \"payload-pre-processing-e2e-ipp-b-9d5625-87d7dd87-24dnh\" (UID: \"24f29cce-d514-49e4-9359-e1efed242080\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d5625-87d7dd87-24dnh"
Nov 02 03:30:31.087283 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:31.087255    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d5625-87d7dd87-24dnh"
Nov 02 03:30:31.372343 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:31.372306    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9d5625-54769b969-tfkvv"]
Nov 02 03:30:31.376852 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:30:31.376819    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda50fee2e_7a02_4b8d_a228_3f0dfd42ce31.slice/crio-2756ae87ec560b0a57e9fb1d144147590d89cb3ca1e786366568090ef759d5fb WatchSource:0}: Error finding container 2756ae87ec560b0a57e9fb1d144147590d89cb3ca1e786366568090ef759d5fb: Status 404 returned error can't find the container with id 2756ae87ec560b0a57e9fb1d144147590d89cb3ca1e786366568090ef759d5fb
Nov 02 03:30:31.478265 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:30:31.478217    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24f29cce_d514_49e4_9359_e1efed242080.slice/crio-fd4d246964b5a7d31e22e664920e09d1d1bf96194677f68540018ca696310874 WatchSource:0}: Error finding container fd4d246964b5a7d31e22e664920e09d1d1bf96194677f68540018ca696310874: Status 404 returned error can't find the container with id fd4d246964b5a7d31e22e664920e09d1d1bf96194677f68540018ca696310874
Nov 02 03:30:31.478419 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:31.478295    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9d5625-87d7dd87-24dnh"]
Nov 02 03:30:31.936785 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:31.936747    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-9d5625-54769b969-tfkvv" event={"ID":"a50fee2e-7a02-4b8d-a228-3f0dfd42ce31","Type":"ContainerStarted","Data":"198cff1ef8a3928b389ef41811d2d12d383750a7622bde08154e6485d0c45da8"}
Nov 02 03:30:31.937261 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:31.936812    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-9d5625-54769b969-tfkvv" event={"ID":"a50fee2e-7a02-4b8d-a228-3f0dfd42ce31","Type":"ContainerStarted","Data":"2756ae87ec560b0a57e9fb1d144147590d89cb3ca1e786366568090ef759d5fb"}
Nov 02 03:30:31.937261 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:31.936968    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-9d5625-54769b969-tfkvv"
Nov 02 03:30:31.938558 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:31.938534    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d5625-87d7dd87-24dnh" event={"ID":"24f29cce-d514-49e4-9359-e1efed242080","Type":"ContainerStarted","Data":"6e99f1846e3108bb6d6c2f333ac48a8ed13ee6631a1a381141464def10ceeebb"}
Nov 02 03:30:31.938685 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:31.938579    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d5625-87d7dd87-24dnh" event={"ID":"24f29cce-d514-49e4-9359-e1efed242080","Type":"ContainerStarted","Data":"fd4d246964b5a7d31e22e664920e09d1d1bf96194677f68540018ca696310874"}
Nov 02 03:30:31.938685 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:31.938669    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d5625-87d7dd87-24dnh"
Nov 02 03:30:32.059457 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:32.058386    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-9d5625-54769b969-tfkvv" podStartSLOduration=2.058369505 podStartE2EDuration="2.058369505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:30:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:30:32.05661827 +0000 UTC m=+2572.635225651" watchObservedRunningTime="2026-11-02 03:30:32.058369505 +0000 UTC m=+2572.636976887"
Nov 02 03:30:32.184408 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:32.184341    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d5625-87d7dd87-24dnh" podStartSLOduration=2.184320886 podStartE2EDuration="2.184320886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:30:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:30:32.177616617 +0000 UTC m=+2572.756223998" watchObservedRunningTime="2026-11-02 03:30:32.184320886 +0000 UTC m=+2572.762928266"
Nov 02 03:30:33.530241 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:30:33.530206    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 03:30:41.286249 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:41.286215    4496 scope.go:122] "RemoveContainer" containerID="d335a6aa085bc63caa501612721dce29696ef8ab2d99e8114fa393ab340afa23"
Nov 02 03:30:41.296650 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:41.296629    4496 scope.go:122] "RemoveContainer" containerID="500b65846cb429f34e4b661eb2980f3faa430efbcd9cad4a3bc3b0f46608513b"
Nov 02 03:30:42.951795 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:42.951765    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d5625-87d7dd87-24dnh"
Nov 02 03:30:42.952253 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:42.951841    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-9d5625-54769b969-tfkvv"
Nov 02 03:30:48.358788 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:48.358746    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-dd87f0-5dbfc776f5-4b62z"]
Nov 02 03:30:48.364898 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:48.364873    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dd87f0-5dbfc776f5-4b62z"
Nov 02 03:30:48.421156 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:48.421122    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-dd87f0-5dbfc776f5-4b62z"]
Nov 02 03:30:48.466454 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:48.466423    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-dd87f0\""
Nov 02 03:30:48.466607 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:48.466515    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-dd87f0-dockercfg-tvr88\""
Nov 02 03:30:48.491555 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:48.491523    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7v9hl\" (UniqueName: \"kubernetes.io/projected/20f3f370-4233-4348-b0f6-176122fc91f8-kube-api-access-7v9hl\") pod \"payload-pre-processing-e2e-ipp-a-dd87f0-5dbfc776f5-4b62z\" (UID: \"20f3f370-4233-4348-b0f6-176122fc91f8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dd87f0-5dbfc776f5-4b62z"
Nov 02 03:30:48.491698 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:48.491560    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20f3f370-4233-4348-b0f6-176122fc91f8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-dd87f0-5dbfc776f5-4b62z\" (UID: \"20f3f370-4233-4348-b0f6-176122fc91f8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dd87f0-5dbfc776f5-4b62z"
Nov 02 03:30:48.593365 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:48.593327    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7v9hl\" (UniqueName: \"kubernetes.io/projected/20f3f370-4233-4348-b0f6-176122fc91f8-kube-api-access-7v9hl\") pod \"payload-pre-processing-e2e-ipp-a-dd87f0-5dbfc776f5-4b62z\" (UID: \"20f3f370-4233-4348-b0f6-176122fc91f8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dd87f0-5dbfc776f5-4b62z"
Nov 02 03:30:48.593365 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:48.593369    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20f3f370-4233-4348-b0f6-176122fc91f8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-dd87f0-5dbfc776f5-4b62z\" (UID: \"20f3f370-4233-4348-b0f6-176122fc91f8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dd87f0-5dbfc776f5-4b62z"
Nov 02 03:30:48.594063 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:48.594039    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20f3f370-4233-4348-b0f6-176122fc91f8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-dd87f0-5dbfc776f5-4b62z\" (UID: \"20f3f370-4233-4348-b0f6-176122fc91f8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dd87f0-5dbfc776f5-4b62z"
Nov 02 03:30:48.650970 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:48.650905    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7v9hl\" (UniqueName: \"kubernetes.io/projected/20f3f370-4233-4348-b0f6-176122fc91f8-kube-api-access-7v9hl\") pod \"payload-pre-processing-e2e-ipp-a-dd87f0-5dbfc776f5-4b62z\" (UID: \"20f3f370-4233-4348-b0f6-176122fc91f8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dd87f0-5dbfc776f5-4b62z"
Nov 02 03:30:48.680633 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:48.680606    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dd87f0-5dbfc776f5-4b62z"
Nov 02 03:30:49.058963 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:49.057929    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-dd87f0-5dbfc776f5-4b62z"]
Nov 02 03:30:49.061310 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:30:49.061280    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20f3f370_4233_4348_b0f6_176122fc91f8.slice/crio-33c593065195017c2f6e467b13c20dca025dbfdd132579d1590d8718562787f5 WatchSource:0}: Error finding container 33c593065195017c2f6e467b13c20dca025dbfdd132579d1590d8718562787f5: Status 404 returned error can't find the container with id 33c593065195017c2f6e467b13c20dca025dbfdd132579d1590d8718562787f5
Nov 02 03:30:49.063806 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:49.063774    4496 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 03:30:49.072967 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:49.072943    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dd87f0-5dbfc776f5-4b62z" event={"ID":"20f3f370-4233-4348-b0f6-176122fc91f8","Type":"ContainerStarted","Data":"33c593065195017c2f6e467b13c20dca025dbfdd132579d1590d8718562787f5"}
Nov 02 03:30:49.142136 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:49.142113    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-dd87f0-54b67c4cd8-k55c8"]
Nov 02 03:30:49.147562 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:49.147542    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-dd87f0-54b67c4cd8-k55c8"
Nov 02 03:30:49.200516 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:49.200487    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jphk\" (UniqueName: \"kubernetes.io/projected/0609027c-0577-4487-8d37-0c0316de5c5f-kube-api-access-9jphk\") pod \"payload-processing-e2e-ipp-a-dd87f0-54b67c4cd8-k55c8\" (UID: \"0609027c-0577-4487-8d37-0c0316de5c5f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-dd87f0-54b67c4cd8-k55c8"
Nov 02 03:30:49.200758 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:49.200733    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0609027c-0577-4487-8d37-0c0316de5c5f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-dd87f0-54b67c4cd8-k55c8\" (UID: \"0609027c-0577-4487-8d37-0c0316de5c5f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-dd87f0-54b67c4cd8-k55c8"
Nov 02 03:30:49.202099 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:49.202077    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-dd87f0-54b67c4cd8-k55c8"]
Nov 02 03:30:49.301855 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:49.301815    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0609027c-0577-4487-8d37-0c0316de5c5f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-dd87f0-54b67c4cd8-k55c8\" (UID: \"0609027c-0577-4487-8d37-0c0316de5c5f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-dd87f0-54b67c4cd8-k55c8"
Nov 02 03:30:49.302137 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:49.301920    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jphk\" (UniqueName: \"kubernetes.io/projected/0609027c-0577-4487-8d37-0c0316de5c5f-kube-api-access-9jphk\") pod \"payload-processing-e2e-ipp-a-dd87f0-54b67c4cd8-k55c8\" (UID: \"0609027c-0577-4487-8d37-0c0316de5c5f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-dd87f0-54b67c4cd8-k55c8"
Nov 02 03:30:49.302652 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:49.302627    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0609027c-0577-4487-8d37-0c0316de5c5f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-dd87f0-54b67c4cd8-k55c8\" (UID: \"0609027c-0577-4487-8d37-0c0316de5c5f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-dd87f0-54b67c4cd8-k55c8"
Nov 02 03:30:49.360504 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:49.360433    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jphk\" (UniqueName: \"kubernetes.io/projected/0609027c-0577-4487-8d37-0c0316de5c5f-kube-api-access-9jphk\") pod \"payload-processing-e2e-ipp-a-dd87f0-54b67c4cd8-k55c8\" (UID: \"0609027c-0577-4487-8d37-0c0316de5c5f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-dd87f0-54b67c4cd8-k55c8"
Nov 02 03:30:49.462090 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:49.462060    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-dd87f0-54b67c4cd8-k55c8"
Nov 02 03:30:49.840477 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:30:49.840430    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0609027c_0577_4487_8d37_0c0316de5c5f.slice/crio-8d849b8556432f23901781ccac241f3e18ffa496422a594f11d188bb2f944482 WatchSource:0}: Error finding container 8d849b8556432f23901781ccac241f3e18ffa496422a594f11d188bb2f944482: Status 404 returned error can't find the container with id 8d849b8556432f23901781ccac241f3e18ffa496422a594f11d188bb2f944482
Nov 02 03:30:49.840790 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:49.840763    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-dd87f0-54b67c4cd8-k55c8"]
Nov 02 03:30:50.083834 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:50.083798    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-dd87f0-54b67c4cd8-k55c8" event={"ID":"0609027c-0577-4487-8d37-0c0316de5c5f","Type":"ContainerStarted","Data":"8d849b8556432f23901781ccac241f3e18ffa496422a594f11d188bb2f944482"}
Nov 02 03:30:53.113889 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:53.113839    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dd87f0-5dbfc776f5-4b62z" event={"ID":"20f3f370-4233-4348-b0f6-176122fc91f8","Type":"ContainerStarted","Data":"a836d92cac3744936b23ae2ec6037127762fa40e928d5b80876d84ccdb3c7a8b"}
Nov 02 03:30:53.114371 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:53.113916    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dd87f0-5dbfc776f5-4b62z"
Nov 02 03:30:53.115756 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:53.115725    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-dd87f0-54b67c4cd8-k55c8" event={"ID":"0609027c-0577-4487-8d37-0c0316de5c5f","Type":"ContainerStarted","Data":"febe49dc3c122f0c5002cd3aa9b752ba54aa87bff947f046c61a840931b894b7"}
Nov 02 03:30:53.115873 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:53.115785    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-dd87f0-54b67c4cd8-k55c8"
Nov 02 03:30:53.230293 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:53.230237    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dd87f0-5dbfc776f5-4b62z" podStartSLOduration=2.073465066 podStartE2EDuration="5.230223242s" totalImagesPullingTime="3.156758176s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:30:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:30:53.228767887 +0000 UTC m=+2593.807375268" watchObservedRunningTime="2026-11-02 03:30:53.230223242 +0000 UTC m=+2593.808830621"
Nov 02 03:30:53.347217 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:30:53.345953    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-dd87f0-54b67c4cd8-k55c8" podStartSLOduration=2.07330638 podStartE2EDuration="4.345934908s" totalImagesPullingTime="2.272628528s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:30:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:30:53.344138312 +0000 UTC m=+2593.922745695" watchObservedRunningTime="2026-11-02 03:30:53.345934908 +0000 UTC m=+2593.924542292"
Nov 02 03:31:02.780339 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:31:02.780298    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6a5285-wvfdm"]
Nov 02 03:31:02.785626 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:31:02.785602    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6a5285-wvfdm"
Nov 02 03:31:02.841662 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:31:02.841634    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6a5285-wvfdm"]
Nov 02 03:31:02.874878 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:31:02.874832    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 03:31:02.874878 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:31:02.874832    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 03:31:02.875144 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:31:02.874842    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 03:31:02.887132 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:31:02.887102    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9z5bk\""
Nov 02 03:31:02.983460 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:31:02.983420    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7h2c\" (UniqueName: \"kubernetes.io/projected/cfc9bdd3-8312-426c-932a-230679dc4a8d-kube-api-access-z7h2c\") pod \"maas-api-revoke-keys-e2e-api-a-6a5285-wvfdm\" (UID: \"cfc9bdd3-8312-426c-932a-230679dc4a8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6a5285-wvfdm"
Nov 02 03:31:02.983651 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:31:02.983534    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfc9bdd3-8312-426c-932a-230679dc4a8d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-6a5285-wvfdm\" (UID: \"cfc9bdd3-8312-426c-932a-230679dc4a8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6a5285-wvfdm"
Nov 02 03:31:02.983651 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:31:02.983633    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfc9bdd3-8312-426c-932a-230679dc4a8d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-6a5285-wvfdm\" (UID: \"cfc9bdd3-8312-426c-932a-230679dc4a8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6a5285-wvfdm"
Nov 02 03:31:03.084642 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:31:03.084546    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7h2c\" (UniqueName: \"kubernetes.io/projected/cfc9bdd3-8312-426c-932a-230679dc4a8d-kube-api-access-z7h2c\") pod \"maas-api-revoke-keys-e2e-api-a-6a5285-wvfdm\" (UID: \"cfc9bdd3-8312-426c-932a-230679dc4a8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6a5285-wvfdm"
Nov 02 03:31:03.084642 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:31:03.084595    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfc9bdd3-8312-426c-932a-230679dc4a8d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-6a5285-wvfdm\" (UID: \"cfc9bdd3-8312-426c-932a-230679dc4a8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6a5285-wvfdm"
Nov 02 03:31:03.084642 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:31:03.084636    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfc9bdd3-8312-426c-932a-230679dc4a8d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-6a5285-wvfdm\" (UID: \"cfc9bdd3-8312-426c-932a-230679dc4a8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6a5285-wvfdm"
Nov 02 03:31:03.085286 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:31:03.085250    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfc9bdd3-8312-426c-932a-230679dc4a8d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-6a5285-wvfdm\" (UID: \"cfc9bdd3-8312-426c-932a-230679dc4a8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6a5285-wvfdm"
Nov 02 03:31:03.087128 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:31:03.087098    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfc9bdd3-8312-426c-932a-230679dc4a8d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-6a5285-wvfdm\" (UID: \"cfc9bdd3-8312-426c-932a-230679dc4a8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6a5285-wvfdm"
Nov 02 03:31:03.141912 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:31:03.141875    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7h2c\" (UniqueName: \"kubernetes.io/projected/cfc9bdd3-8312-426c-932a-230679dc4a8d-kube-api-access-z7h2c\") pod \"maas-api-revoke-keys-e2e-api-a-6a5285-wvfdm\" (UID: \"cfc9bdd3-8312-426c-932a-230679dc4a8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6a5285-wvfdm"
Nov 02 03:31:03.399535 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:31:03.399444    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6a5285-wvfdm"
Nov 02 03:31:03.533286 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:31:03.533248    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 03:31:03.778554 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:31:03.778529    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6a5285-wvfdm"]
Nov 02 03:31:03.779591 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:31:03.779563    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfc9bdd3_8312_426c_932a_230679dc4a8d.slice/crio-a61e851603c45469c30f37a2fea70a5a1394afae0f24cad2e52ab113e9160f84 WatchSource:0}: Error finding container a61e851603c45469c30f37a2fea70a5a1394afae0f24cad2e52ab113e9160f84: Status 404 returned error can't find the container with id a61e851603c45469c30f37a2fea70a5a1394afae0f24cad2e52ab113e9160f84
Nov 02 03:31:04.125230 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:31:04.125194    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dd87f0-5dbfc776f5-4b62z"
Nov 02 03:31:04.125626 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:31:04.125276    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-dd87f0-54b67c4cd8-k55c8"
Nov 02 03:31:04.209782 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:31:04.209747    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6a5285-wvfdm" event={"ID":"cfc9bdd3-8312-426c-932a-230679dc4a8d","Type":"ContainerStarted","Data":"b3c851a75a5cecf757f3f524b73730afa22f48e8df07fab14743883a0b03a4a4"}
Nov 02 03:31:04.209933 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:31:04.209806    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6a5285-wvfdm" event={"ID":"cfc9bdd3-8312-426c-932a-230679dc4a8d","Type":"ContainerStarted","Data":"a61e851603c45469c30f37a2fea70a5a1394afae0f24cad2e52ab113e9160f84"}
Nov 02 03:31:04.472587 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:31:04.472482    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6a5285-wvfdm" podStartSLOduration=2.47246843 podStartE2EDuration="2.47246843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:31:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:31:04.471572234 +0000 UTC m=+2605.050179614" watchObservedRunningTime="2026-11-02 03:31:04.47246843 +0000 UTC m=+2605.051075809"
Nov 02 03:31:05.218120 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:31:05.218087    4496 generic.go:356] "Generic (PLEG): container finished" podID="cfc9bdd3-8312-426c-932a-230679dc4a8d" containerID="b3c851a75a5cecf757f3f524b73730afa22f48e8df07fab14743883a0b03a4a4" exitCode=0
Nov 02 03:31:05.218525 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:31:05.218173    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6a5285-wvfdm" event={"ID":"cfc9bdd3-8312-426c-932a-230679dc4a8d","Type":"ContainerDied","Data":"b3c851a75a5cecf757f3f524b73730afa22f48e8df07fab14743883a0b03a4a4"}
Nov 02 03:31:06.380879 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:31:06.380851    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6a5285-wvfdm"
Nov 02 03:31:06.422834 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:31:06.422804    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cfc9bdd3-8312-426c-932a-230679dc4a8d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfc9bdd3-8312-426c-932a-230679dc4a8d-maas-api-xks-ca\") pod \"cfc9bdd3-8312-426c-932a-230679dc4a8d\" (UID: \"cfc9bdd3-8312-426c-932a-230679dc4a8d\") "
Nov 02 03:31:06.422970 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:31:06.422885    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfc9bdd3-8312-426c-932a-230679dc4a8d-kube-api-access-z7h2c\" (UniqueName: \"kubernetes.io/projected/cfc9bdd3-8312-426c-932a-230679dc4a8d-kube-api-access-z7h2c\") pod \"cfc9bdd3-8312-426c-932a-230679dc4a8d\" (UID: \"cfc9bdd3-8312-426c-932a-230679dc4a8d\") "
Nov 02 03:31:06.422970 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:31:06.422924    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfc9bdd3-8312-426c-932a-230679dc4a8d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfc9bdd3-8312-426c-932a-230679dc4a8d-maas-api-openshift-ca\") pod \"cfc9bdd3-8312-426c-932a-230679dc4a8d\" (UID: \"cfc9bdd3-8312-426c-932a-230679dc4a8d\") "
Nov 02 03:31:06.423346 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:31:06.423318    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfc9bdd3-8312-426c-932a-230679dc4a8d-maas-api-openshift-ca" pod "cfc9bdd3-8312-426c-932a-230679dc4a8d" (UID: "cfc9bdd3-8312-426c-932a-230679dc4a8d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:31:06.423528 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:31:06.423511    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfc9bdd3-8312-426c-932a-230679dc4a8d-maas-api-openshift-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:31:06.424931 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:31:06.424901    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfc9bdd3-8312-426c-932a-230679dc4a8d-kube-api-access-z7h2c" pod "cfc9bdd3-8312-426c-932a-230679dc4a8d" (UID: "cfc9bdd3-8312-426c-932a-230679dc4a8d"). InnerVolumeSpecName "kube-api-access-z7h2c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:31:06.425049 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:31:06.424972    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfc9bdd3-8312-426c-932a-230679dc4a8d-maas-api-xks-ca" pod "cfc9bdd3-8312-426c-932a-230679dc4a8d" (UID: "cfc9bdd3-8312-426c-932a-230679dc4a8d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 03:31:06.525277 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:31:06.525195    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7h2c\" (UniqueName: \"kubernetes.io/projected/cfc9bdd3-8312-426c-932a-230679dc4a8d-kube-api-access-z7h2c\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:31:06.525277 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:31:06.525233    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfc9bdd3-8312-426c-932a-230679dc4a8d-maas-api-xks-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:31:07.234435 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:31:07.234400    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6a5285-wvfdm"
Nov 02 03:31:07.234658 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:31:07.234432    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6a5285-wvfdm" event={"ID":"cfc9bdd3-8312-426c-932a-230679dc4a8d","Type":"ContainerDied","Data":"a61e851603c45469c30f37a2fea70a5a1394afae0f24cad2e52ab113e9160f84"}
Nov 02 03:31:07.234658 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:31:07.234480    4496 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a61e851603c45469c30f37a2fea70a5a1394afae0f24cad2e52ab113e9160f84"
Nov 02 03:31:20.831183 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:31:20.831146    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-0809c8-64cf8df667-gp4jm"]
Nov 02 03:31:20.832112 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:31:20.832088    4496 cpu_manager.go:414] "containerMap: removing container" podUID="cfc9bdd3-8312-426c-932a-230679dc4a8d" containerName="revoke-keys"
Nov 02 03:31:20.832112 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:31:20.832112    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfc9bdd3-8312-426c-932a-230679dc4a8d" containerName="revoke-keys"
Nov 02 03:31:20.832307 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:31:20.832286    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfc9bdd3-8312-426c-932a-230679dc4a8d" containerName="revoke-keys"
Nov 02 03:31:20.855040 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:31:20.845287    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-0809c8-64cf8df667-gp4jm"
Nov 02 03:31:20.885932 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:31:20.885666    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mj8t\" (UniqueName: \"kubernetes.io/projected/763d2941-009d-48c5-9f8b-99e5fdbd3949-kube-api-access-7mj8t\") pod \"payload-processing-e2e-ipp-b-0809c8-64cf8df667-gp4jm\" (UID: \"763d2941-009d-48c5-9f8b-99e5fdbd3949\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-0809c8-64cf8df667-gp4jm"
Nov 02 03:31:20.886306 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:31:20.886272    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/763d2941-009d-48c5-9f8b-99e5fdbd3949-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-0809c8-64cf8df667-gp4jm\" (UID: \"763d2941-009d-48c5-9f8b-99e5fdbd3949\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-0809c8-64cf8df667-gp4jm"
Nov 02 03:31:20.887329 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:31:20.887305    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-0809c8-64cf8df667-gp4jm"]
Nov 02 03:31:20.903424 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:31:20.903399    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-0809c8\""
Nov 02 03:31:20.903577 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:31:20.903546    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-0809c8-dockercfg-5d9dt\""
Nov 02 03:31:20.988276 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:31:20.988241    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mj8t\" (UniqueName: \"kubernetes.io/projected/763d2941-009d-48c5-9f8b-99e5fdbd3949-kube-api-access-7mj8t\") pod \"payload-processing-e2e-ipp-b-0809c8-64cf8df667-gp4jm\" (UID: \"763d2941-009d-48c5-9f8b-99e5fdbd3949\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-0809c8-64cf8df667-gp4jm"
Nov 02 03:31:20.988276 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:31:20.988284    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/763d2941-009d-48c5-9f8b-99e5fdbd3949-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-0809c8-64cf8df667-gp4jm\" (UID: \"763d2941-009d-48c5-9f8b-99e5fdbd3949\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-0809c8-64cf8df667-gp4jm"
Nov 02 03:31:20.989047 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:31:20.989001    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/763d2941-009d-48c5-9f8b-99e5fdbd3949-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-0809c8-64cf8df667-gp4jm\" (UID: \"763d2941-009d-48c5-9f8b-99e5fdbd3949\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-0809c8-64cf8df667-gp4jm"
Nov 02 03:31:21.058889 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:31:21.058848    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mj8t\" (UniqueName: \"kubernetes.io/projected/763d2941-009d-48c5-9f8b-99e5fdbd3949-kube-api-access-7mj8t\") pod \"payload-processing-e2e-ipp-b-0809c8-64cf8df667-gp4jm\" (UID: \"763d2941-009d-48c5-9f8b-99e5fdbd3949\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-0809c8-64cf8df667-gp4jm"
Nov 02 03:31:21.214075 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:31:21.213968    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-0809c8-64cf8df667-gp4jm"
Nov 02 03:31:21.635045 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:31:21.633771    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-0809c8-64cf8df667-gp4jm"]
Nov 02 03:31:22.460043 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:31:22.459908    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-0809c8-64cf8df667-gp4jm" event={"ID":"763d2941-009d-48c5-9f8b-99e5fdbd3949","Type":"ContainerStarted","Data":"49d32d0dcd573e1c9b771de7de0e8019a0301a44a6e3855eb874a7769f71f04f"}
Nov 02 03:31:22.460043 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:31:22.459989    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-0809c8-64cf8df667-gp4jm" event={"ID":"763d2941-009d-48c5-9f8b-99e5fdbd3949","Type":"ContainerStarted","Data":"033a3e9b6ccdd1b2e319f166b8d76c77ae34b2864c587251be771084e216bc48"}
Nov 02 03:31:22.466674 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:31:22.466636    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-0809c8-64cf8df667-gp4jm"
Nov 02 03:31:22.582347 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:31:22.582297    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-0809c8-64cf8df667-gp4jm" podStartSLOduration=2.582282023 podStartE2EDuration="2.582282023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:31:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:31:22.581075518 +0000 UTC m=+2623.159682899" watchObservedRunningTime="2026-11-02 03:31:22.582282023 +0000 UTC m=+2623.160889399"
Nov 02 03:31:27.508651 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:31:27.508619    4496 generic.go:356] "Generic (PLEG): container finished" podID="ecfffbc9-4809-49a3-aed3-35b771fcb388" containerID="04e1a6d2ce978d286dad971f200a968a4b40fbf2ffdd3299875e06eafd008708" exitCode=0
Nov 02 03:31:27.509232 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:31:27.508688    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-018375-openshift-default-7fbc48965-7mlcx" event={"ID":"ecfffbc9-4809-49a3-aed3-35b771fcb388","Type":"ContainerDied","Data":"04e1a6d2ce978d286dad971f200a968a4b40fbf2ffdd3299875e06eafd008708"}
Nov 02 03:31:27.509330 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:31:27.509242    4496 scope.go:122] "RemoveContainer" containerID="04e1a6d2ce978d286dad971f200a968a4b40fbf2ffdd3299875e06eafd008708"
Nov 02 03:31:27.510943 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:31:27.510914    4496 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 03:31:27.511049 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:31:27.510981    4496 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 03:31:27.511049 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:31:27.511030    4496 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 03:31:28.517737 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:31:28.517699    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-018375-openshift-default-7fbc48965-7mlcx" event={"ID":"ecfffbc9-4809-49a3-aed3-35b771fcb388","Type":"ContainerStarted","Data":"5f93f8ac2177bb1a83fc1c46c39136cefcc673e66b6d3c1a766de5c75cd65a70"}
Nov 02 03:31:29.457105 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:31:29.457069    4496 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-018375-openshift-default-7fbc48965-7mlcx"
Nov 02 03:31:29.458202 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:31:29.458171    4496 patch_prober.go:29] interesting pod/e2e-ipp-b-018375-openshift-default-7fbc48965-7mlcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body=
Nov 02 03:31:29.458439 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:31:29.458410    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-018375-openshift-default-7fbc48965-7mlcx" podUID="ecfffbc9-4809-49a3-aed3-35b771fcb388" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused"
Nov 02 03:31:30.458039 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:31:30.457348    4496 patch_prober.go:29] interesting pod/e2e-ipp-b-018375-openshift-default-7fbc48965-7mlcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body=
Nov 02 03:31:30.458039 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:31:30.457420    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-018375-openshift-default-7fbc48965-7mlcx" podUID="ecfffbc9-4809-49a3-aed3-35b771fcb388" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused"
Nov 02 03:31:31.461037 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:31:31.460659    4496 patch_prober.go:29] interesting pod/e2e-ipp-b-018375-openshift-default-7fbc48965-7mlcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body=
Nov 02 03:31:31.461037 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:31:31.460729    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-018375-openshift-default-7fbc48965-7mlcx" podUID="ecfffbc9-4809-49a3-aed3-35b771fcb388" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused"
Nov 02 03:31:31.706231 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:31:31.706191    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-045653-7c865bfc67-tphl9"]
Nov 02 03:31:31.711684 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:31:31.711625    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-045653-7c865bfc67-tphl9"
Nov 02 03:31:31.765315 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:31:31.765272    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-045653-dockercfg-9sgx6\""
Nov 02 03:31:31.765510 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:31:31.765349    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-045653\""
Nov 02 03:31:31.766891 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:31:31.766863    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-045653-7c865bfc67-tphl9"]
Nov 02 03:31:31.830624 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:31:31.830575    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61f56105-4f55-4dd8-96bc-53c8ab4e4964-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-045653-7c865bfc67-tphl9\" (UID: \"61f56105-4f55-4dd8-96bc-53c8ab4e4964\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-045653-7c865bfc67-tphl9"
Nov 02 03:31:31.830799 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:31:31.830711    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmrfj\" (UniqueName: \"kubernetes.io/projected/61f56105-4f55-4dd8-96bc-53c8ab4e4964-kube-api-access-fmrfj\") pod \"payload-pre-processing-e2e-ipp-a-045653-7c865bfc67-tphl9\" (UID: \"61f56105-4f55-4dd8-96bc-53c8ab4e4964\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-045653-7c865bfc67-tphl9"
Nov 02 03:31:31.931964 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:31:31.931930    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61f56105-4f55-4dd8-96bc-53c8ab4e4964-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-045653-7c865bfc67-tphl9\" (UID: \"61f56105-4f55-4dd8-96bc-53c8ab4e4964\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-045653-7c865bfc67-tphl9"
Nov 02 03:31:31.932192 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:31:31.932011    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmrfj\" (UniqueName: \"kubernetes.io/projected/61f56105-4f55-4dd8-96bc-53c8ab4e4964-kube-api-access-fmrfj\") pod \"payload-pre-processing-e2e-ipp-a-045653-7c865bfc67-tphl9\" (UID: \"61f56105-4f55-4dd8-96bc-53c8ab4e4964\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-045653-7c865bfc67-tphl9"
Nov 02 03:31:31.932681 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:31:31.932660    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61f56105-4f55-4dd8-96bc-53c8ab4e4964-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-045653-7c865bfc67-tphl9\" (UID: \"61f56105-4f55-4dd8-96bc-53c8ab4e4964\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-045653-7c865bfc67-tphl9"
Nov 02 03:31:31.990437 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:31:31.990398    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmrfj\" (UniqueName: \"kubernetes.io/projected/61f56105-4f55-4dd8-96bc-53c8ab4e4964-kube-api-access-fmrfj\") pod \"payload-pre-processing-e2e-ipp-a-045653-7c865bfc67-tphl9\" (UID: \"61f56105-4f55-4dd8-96bc-53c8ab4e4964\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-045653-7c865bfc67-tphl9"
Nov 02 03:31:32.026398 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:31:32.026367    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-045653-7c865bfc67-tphl9"
Nov 02 03:31:32.355176 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:31:32.354121    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-018375-openshift-default-7fbc48965-7mlcx"
Nov 02 03:31:32.465031 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:31:32.464977    4496 patch_prober.go:29] interesting pod/e2e-ipp-b-018375-openshift-default-7fbc48965-7mlcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body=
Nov 02 03:31:32.465582 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:31:32.465552    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-018375-openshift-default-7fbc48965-7mlcx" podUID="ecfffbc9-4809-49a3-aed3-35b771fcb388" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused"
Nov 02 03:31:32.469476 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:31:32.469446    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61f56105_4f55_4dd8_96bc_53c8ab4e4964.slice/crio-5bfb43cade3069264d1f66b51714c17f0fd970c3c1c9951d53bbd1499e715b8a WatchSource:0}: Error finding container 5bfb43cade3069264d1f66b51714c17f0fd970c3c1c9951d53bbd1499e715b8a: Status 404 returned error can't find the container with id 5bfb43cade3069264d1f66b51714c17f0fd970c3c1c9951d53bbd1499e715b8a
Nov 02 03:31:32.471909 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:31:32.471879    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-045653-7c865bfc67-tphl9"]
Nov 02 03:31:32.478506 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:31:32.478480    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-045653-6b5cfcf4d-d85kt"]
Nov 02 03:31:32.484881 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:31:32.484862    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-045653-6b5cfcf4d-d85kt"
Nov 02 03:31:32.567238 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:31:32.567202    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-045653-6b5cfcf4d-d85kt"]
Nov 02 03:31:32.567907 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:31:32.567877    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6l9x\" (UniqueName: \"kubernetes.io/projected/0ccc5f9c-bfeb-4076-ae4f-741cb1fbe76b-kube-api-access-q6l9x\") pod \"payload-processing-e2e-ipp-a-045653-6b5cfcf4d-d85kt\" (UID: \"0ccc5f9c-bfeb-4076-ae4f-741cb1fbe76b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-045653-6b5cfcf4d-d85kt"
Nov 02 03:31:32.576884 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:31:32.576600    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ccc5f9c-bfeb-4076-ae4f-741cb1fbe76b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-045653-6b5cfcf4d-d85kt\" (UID: \"0ccc5f9c-bfeb-4076-ae4f-741cb1fbe76b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-045653-6b5cfcf4d-d85kt"
Nov 02 03:31:32.607579 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:31:32.607414    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-045653-7c865bfc67-tphl9" event={"ID":"61f56105-4f55-4dd8-96bc-53c8ab4e4964","Type":"ContainerStarted","Data":"5bfb43cade3069264d1f66b51714c17f0fd970c3c1c9951d53bbd1499e715b8a"}
Nov 02 03:31:32.680038 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:31:32.679584    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6l9x\" (UniqueName: \"kubernetes.io/projected/0ccc5f9c-bfeb-4076-ae4f-741cb1fbe76b-kube-api-access-q6l9x\") pod \"payload-processing-e2e-ipp-a-045653-6b5cfcf4d-d85kt\" (UID: \"0ccc5f9c-bfeb-4076-ae4f-741cb1fbe76b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-045653-6b5cfcf4d-d85kt"
Nov 02 03:31:32.680038 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:31:32.679647    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ccc5f9c-bfeb-4076-ae4f-741cb1fbe76b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-045653-6b5cfcf4d-d85kt\" (UID: \"0ccc5f9c-bfeb-4076-ae4f-741cb1fbe76b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-045653-6b5cfcf4d-d85kt"
Nov 02 03:31:32.680636 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:31:32.680579    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ccc5f9c-bfeb-4076-ae4f-741cb1fbe76b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-045653-6b5cfcf4d-d85kt\" (UID: \"0ccc5f9c-bfeb-4076-ae4f-741cb1fbe76b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-045653-6b5cfcf4d-d85kt"
Nov 02 03:31:32.760795 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:31:32.760755    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6l9x\" (UniqueName: \"kubernetes.io/projected/0ccc5f9c-bfeb-4076-ae4f-741cb1fbe76b-kube-api-access-q6l9x\") pod \"payload-processing-e2e-ipp-a-045653-6b5cfcf4d-d85kt\" (UID: \"0ccc5f9c-bfeb-4076-ae4f-741cb1fbe76b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-045653-6b5cfcf4d-d85kt"
Nov 02 03:31:32.828916 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:31:32.828872    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-045653-6b5cfcf4d-d85kt"
Nov 02 03:31:33.236046 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:31:33.232991    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-045653-6b5cfcf4d-d85kt"]
Nov 02 03:31:33.236046 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:31:33.233483    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ccc5f9c_bfeb_4076_ae4f_741cb1fbe76b.slice/crio-a599205dcbf7ba68c2d06d9fe0edcaa57581bfbab3bb276c62f87d8ddab9932b WatchSource:0}: Error finding container a599205dcbf7ba68c2d06d9fe0edcaa57581bfbab3bb276c62f87d8ddab9932b: Status 404 returned error can't find the container with id a599205dcbf7ba68c2d06d9fe0edcaa57581bfbab3bb276c62f87d8ddab9932b
Nov 02 03:31:33.456058 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:31:33.455168    4496 patch_prober.go:29] interesting pod/e2e-ipp-b-018375-openshift-default-7fbc48965-7mlcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body=
Nov 02 03:31:33.456058 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:31:33.455239    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-018375-openshift-default-7fbc48965-7mlcx" podUID="ecfffbc9-4809-49a3-aed3-35b771fcb388" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused"
Nov 02 03:31:33.529837 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:31:33.529757    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 03:31:33.619753 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:31:33.619712    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-045653-6b5cfcf4d-d85kt" event={"ID":"0ccc5f9c-bfeb-4076-ae4f-741cb1fbe76b","Type":"ContainerStarted","Data":"aac907948cbf751d304b8be7b502ff49f94fe9c1c22fa067ca2255155dbb3efa"}
Nov 02 03:31:33.620101 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:31:33.619779    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-045653-6b5cfcf4d-d85kt" event={"ID":"0ccc5f9c-bfeb-4076-ae4f-741cb1fbe76b","Type":"ContainerStarted","Data":"a599205dcbf7ba68c2d06d9fe0edcaa57581bfbab3bb276c62f87d8ddab9932b"}
Nov 02 03:31:33.620101 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:31:33.619929    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-045653-6b5cfcf4d-d85kt"
Nov 02 03:31:33.621655 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:31:33.621633    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-045653-7c865bfc67-tphl9" event={"ID":"61f56105-4f55-4dd8-96bc-53c8ab4e4964","Type":"ContainerStarted","Data":"b4bb31b576d1b045df77ff6b2fed580126723437171ab960c0e123de78568ebb"}
Nov 02 03:31:33.621778 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:31:33.621767    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-045653-7c865bfc67-tphl9"
Nov 02 03:31:33.738613 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:31:33.738559    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-045653-6b5cfcf4d-d85kt" podStartSLOduration=1.738544041 podStartE2EDuration="1.738544041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:31:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:31:33.734355639 +0000 UTC m=+2634.312963021" watchObservedRunningTime="2026-11-02 03:31:33.738544041 +0000 UTC m=+2634.317151420"
Nov 02 03:31:33.855241 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:31:33.855130    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-045653-7c865bfc67-tphl9" podStartSLOduration=2.855111303 podStartE2EDuration="2.855111303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:31:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:31:33.84957916 +0000 UTC m=+2634.428186542" watchObservedRunningTime="2026-11-02 03:31:33.855111303 +0000 UTC m=+2634.433718690"
Nov 02 03:31:34.460050 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:31:34.460001    4496 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-018375-openshift-default-7fbc48965-7mlcx"
Nov 02 03:31:34.461219 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:31:34.461198    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-018375-openshift-default-7fbc48965-7mlcx"
Nov 02 03:31:34.486595 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:31:34.486563    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-0809c8-64cf8df667-gp4jm"
Nov 02 03:31:41.380713 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:31:41.380677    4496 scope.go:122] "RemoveContainer" containerID="52b6c092bea015c787a59f09cc5577660e29e5bf7bb382a097f596b7143a3af5"
Nov 02 03:31:41.391198 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:31:41.391176    4496 scope.go:122] "RemoveContainer" containerID="816c8e4e99ebba8d7ba2e16ac4877f6c12488e11dd2c1e7bbe44fefc9ec54b84"
Nov 02 03:31:44.630939 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:31:44.630906    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-045653-6b5cfcf4d-d85kt"
Nov 02 03:31:44.631446 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:31:44.631082    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-045653-7c865bfc67-tphl9"
Nov 02 03:32:03.527853 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:32:03.527813    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 03:32:04.991433 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:04.991400    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-4c3ea5/tenantcfg-model-4c3ea5-kserve-77d88775bf-gsngn"]
Nov 02 03:32:04.997140 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:04.997113    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-4c3ea5/tenantcfg-model-4c3ea5-kserve-77d88775bf-gsngn"
Nov 02 03:32:05.053701 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:05.053668    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-4c3ea5/tenantcfg-model-4c3ea5-kserve-77d88775bf-gsngn"]
Nov 02 03:32:05.098702 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:05.098663    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-4c3ea5\"/\"openshift-service-ca.crt\""
Nov 02 03:32:05.098865 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:05.098716    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-4c3ea5\"/\"kube-root-ca.crt\""
Nov 02 03:32:05.098865 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:05.098756    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-4c3ea5\"/\"tenantcfg-model-4c3ea5-kserve-self-signed-certs\""
Nov 02 03:32:05.098865 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:05.098834    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-4c3ea5\"/\"default-dockercfg-vlchs\""
Nov 02 03:32:05.194870 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:05.194832    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c08a1ba9-e651-4714-b84d-9b8161f6ae8f-dshm\") pod \"tenantcfg-model-4c3ea5-kserve-77d88775bf-gsngn\" (UID: \"c08a1ba9-e651-4714-b84d-9b8161f6ae8f\") " pod="ai-tenant-e2e-shared-a-w2-4c3ea5/tenantcfg-model-4c3ea5-kserve-77d88775bf-gsngn"
Nov 02 03:32:05.195078 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:05.195052    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c08a1ba9-e651-4714-b84d-9b8161f6ae8f-model-cache\") pod \"tenantcfg-model-4c3ea5-kserve-77d88775bf-gsngn\" (UID: \"c08a1ba9-e651-4714-b84d-9b8161f6ae8f\") " pod="ai-tenant-e2e-shared-a-w2-4c3ea5/tenantcfg-model-4c3ea5-kserve-77d88775bf-gsngn"
Nov 02 03:32:05.195131 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:05.195114    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c08a1ba9-e651-4714-b84d-9b8161f6ae8f-tls-certs\") pod \"tenantcfg-model-4c3ea5-kserve-77d88775bf-gsngn\" (UID: \"c08a1ba9-e651-4714-b84d-9b8161f6ae8f\") " pod="ai-tenant-e2e-shared-a-w2-4c3ea5/tenantcfg-model-4c3ea5-kserve-77d88775bf-gsngn"
Nov 02 03:32:05.195210 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:05.195197    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c08a1ba9-e651-4714-b84d-9b8161f6ae8f-tmp-dir\") pod \"tenantcfg-model-4c3ea5-kserve-77d88775bf-gsngn\" (UID: \"c08a1ba9-e651-4714-b84d-9b8161f6ae8f\") " pod="ai-tenant-e2e-shared-a-w2-4c3ea5/tenantcfg-model-4c3ea5-kserve-77d88775bf-gsngn"
Nov 02 03:32:05.195260 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:05.195246    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpcwd\" (UniqueName: \"kubernetes.io/projected/c08a1ba9-e651-4714-b84d-9b8161f6ae8f-kube-api-access-kpcwd\") pod \"tenantcfg-model-4c3ea5-kserve-77d88775bf-gsngn\" (UID: \"c08a1ba9-e651-4714-b84d-9b8161f6ae8f\") " pod="ai-tenant-e2e-shared-a-w2-4c3ea5/tenantcfg-model-4c3ea5-kserve-77d88775bf-gsngn"
Nov 02 03:32:05.195313 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:05.195271    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c08a1ba9-e651-4714-b84d-9b8161f6ae8f-home\") pod \"tenantcfg-model-4c3ea5-kserve-77d88775bf-gsngn\" (UID: \"c08a1ba9-e651-4714-b84d-9b8161f6ae8f\") " pod="ai-tenant-e2e-shared-a-w2-4c3ea5/tenantcfg-model-4c3ea5-kserve-77d88775bf-gsngn"
Nov 02 03:32:05.296072 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:05.295991    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpcwd\" (UniqueName: \"kubernetes.io/projected/c08a1ba9-e651-4714-b84d-9b8161f6ae8f-kube-api-access-kpcwd\") pod \"tenantcfg-model-4c3ea5-kserve-77d88775bf-gsngn\" (UID: \"c08a1ba9-e651-4714-b84d-9b8161f6ae8f\") " pod="ai-tenant-e2e-shared-a-w2-4c3ea5/tenantcfg-model-4c3ea5-kserve-77d88775bf-gsngn"
Nov 02 03:32:05.296072 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:05.296061    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c08a1ba9-e651-4714-b84d-9b8161f6ae8f-home\") pod \"tenantcfg-model-4c3ea5-kserve-77d88775bf-gsngn\" (UID: \"c08a1ba9-e651-4714-b84d-9b8161f6ae8f\") " pod="ai-tenant-e2e-shared-a-w2-4c3ea5/tenantcfg-model-4c3ea5-kserve-77d88775bf-gsngn"
Nov 02 03:32:05.296307 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:05.296152    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c08a1ba9-e651-4714-b84d-9b8161f6ae8f-dshm\") pod \"tenantcfg-model-4c3ea5-kserve-77d88775bf-gsngn\" (UID: \"c08a1ba9-e651-4714-b84d-9b8161f6ae8f\") " pod="ai-tenant-e2e-shared-a-w2-4c3ea5/tenantcfg-model-4c3ea5-kserve-77d88775bf-gsngn"
Nov 02 03:32:05.296368 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:05.296305    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c08a1ba9-e651-4714-b84d-9b8161f6ae8f-model-cache\") pod \"tenantcfg-model-4c3ea5-kserve-77d88775bf-gsngn\" (UID: \"c08a1ba9-e651-4714-b84d-9b8161f6ae8f\") " pod="ai-tenant-e2e-shared-a-w2-4c3ea5/tenantcfg-model-4c3ea5-kserve-77d88775bf-gsngn"
Nov 02 03:32:05.296368 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:05.296337    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c08a1ba9-e651-4714-b84d-9b8161f6ae8f-tls-certs\") pod \"tenantcfg-model-4c3ea5-kserve-77d88775bf-gsngn\" (UID: \"c08a1ba9-e651-4714-b84d-9b8161f6ae8f\") " pod="ai-tenant-e2e-shared-a-w2-4c3ea5/tenantcfg-model-4c3ea5-kserve-77d88775bf-gsngn"
Nov 02 03:32:05.296480 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:05.296419    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c08a1ba9-e651-4714-b84d-9b8161f6ae8f-tmp-dir\") pod \"tenantcfg-model-4c3ea5-kserve-77d88775bf-gsngn\" (UID: \"c08a1ba9-e651-4714-b84d-9b8161f6ae8f\") " pod="ai-tenant-e2e-shared-a-w2-4c3ea5/tenantcfg-model-4c3ea5-kserve-77d88775bf-gsngn"
Nov 02 03:32:05.296573 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:05.296545    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c08a1ba9-e651-4714-b84d-9b8161f6ae8f-home\") pod \"tenantcfg-model-4c3ea5-kserve-77d88775bf-gsngn\" (UID: \"c08a1ba9-e651-4714-b84d-9b8161f6ae8f\") " pod="ai-tenant-e2e-shared-a-w2-4c3ea5/tenantcfg-model-4c3ea5-kserve-77d88775bf-gsngn"
Nov 02 03:32:05.296717 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:05.296639    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c08a1ba9-e651-4714-b84d-9b8161f6ae8f-model-cache\") pod \"tenantcfg-model-4c3ea5-kserve-77d88775bf-gsngn\" (UID: \"c08a1ba9-e651-4714-b84d-9b8161f6ae8f\") " pod="ai-tenant-e2e-shared-a-w2-4c3ea5/tenantcfg-model-4c3ea5-kserve-77d88775bf-gsngn"
Nov 02 03:32:05.296772 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:05.296756    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c08a1ba9-e651-4714-b84d-9b8161f6ae8f-tmp-dir\") pod \"tenantcfg-model-4c3ea5-kserve-77d88775bf-gsngn\" (UID: \"c08a1ba9-e651-4714-b84d-9b8161f6ae8f\") " pod="ai-tenant-e2e-shared-a-w2-4c3ea5/tenantcfg-model-4c3ea5-kserve-77d88775bf-gsngn"
Nov 02 03:32:05.298395 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:05.298375    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c08a1ba9-e651-4714-b84d-9b8161f6ae8f-dshm\") pod \"tenantcfg-model-4c3ea5-kserve-77d88775bf-gsngn\" (UID: \"c08a1ba9-e651-4714-b84d-9b8161f6ae8f\") " pod="ai-tenant-e2e-shared-a-w2-4c3ea5/tenantcfg-model-4c3ea5-kserve-77d88775bf-gsngn"
Nov 02 03:32:05.298733 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:05.298715    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c08a1ba9-e651-4714-b84d-9b8161f6ae8f-tls-certs\") pod \"tenantcfg-model-4c3ea5-kserve-77d88775bf-gsngn\" (UID: \"c08a1ba9-e651-4714-b84d-9b8161f6ae8f\") " pod="ai-tenant-e2e-shared-a-w2-4c3ea5/tenantcfg-model-4c3ea5-kserve-77d88775bf-gsngn"
Nov 02 03:32:05.353082 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:05.353052    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpcwd\" (UniqueName: \"kubernetes.io/projected/c08a1ba9-e651-4714-b84d-9b8161f6ae8f-kube-api-access-kpcwd\") pod \"tenantcfg-model-4c3ea5-kserve-77d88775bf-gsngn\" (UID: \"c08a1ba9-e651-4714-b84d-9b8161f6ae8f\") " pod="ai-tenant-e2e-shared-a-w2-4c3ea5/tenantcfg-model-4c3ea5-kserve-77d88775bf-gsngn"
Nov 02 03:32:05.610538 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:05.610466    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-4c3ea5/tenantcfg-model-4c3ea5-kserve-77d88775bf-gsngn"
Nov 02 03:32:05.994653 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:32:05.994620    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc08a1ba9_e651_4714_b84d_9b8161f6ae8f.slice/crio-58d616bea10a38e92f9dce668e14f88f51843ec206b49f8e15dd3985bcda3af4 WatchSource:0}: Error finding container 58d616bea10a38e92f9dce668e14f88f51843ec206b49f8e15dd3985bcda3af4: Status 404 returned error can't find the container with id 58d616bea10a38e92f9dce668e14f88f51843ec206b49f8e15dd3985bcda3af4
Nov 02 03:32:05.994976 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:05.994886    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-4c3ea5/tenantcfg-model-4c3ea5-kserve-77d88775bf-gsngn"]
Nov 02 03:32:06.885663 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:06.885614    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-4c3ea5/tenantcfg-model-4c3ea5-kserve-77d88775bf-gsngn" event={"ID":"c08a1ba9-e651-4714-b84d-9b8161f6ae8f","Type":"ContainerStarted","Data":"2cd43900264fe04ae9022fbbb3e46bc3c626dc980336a09be1adbcb51086a9cc"}
Nov 02 03:32:06.885856 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:06.885676    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-4c3ea5/tenantcfg-model-4c3ea5-kserve-77d88775bf-gsngn" event={"ID":"c08a1ba9-e651-4714-b84d-9b8161f6ae8f","Type":"ContainerStarted","Data":"58d616bea10a38e92f9dce668e14f88f51843ec206b49f8e15dd3985bcda3af4"}
Nov 02 03:32:07.005008 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:07.004948    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-4c3ea5/tenantcfg-model-4c3ea5-kserve-77d88775bf-gsngn" podStartSLOduration=3.004933255 podStartE2EDuration="3.004933255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:32:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:32:07.001256478 +0000 UTC m=+2667.579863865" watchObservedRunningTime="2026-11-02 03:32:07.004933255 +0000 UTC m=+2667.583540635"
Nov 02 03:32:07.611066 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:07.611008    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-4c3ea5/tenantcfg-model-4c3ea5-kserve-77d88775bf-gsngn"
Nov 02 03:32:15.611259 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:15.611211    4496 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-4c3ea5/tenantcfg-model-4c3ea5-kserve-77d88775bf-gsngn"
Nov 02 03:32:15.623487 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:15.623450    4496 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-4c3ea5/tenantcfg-model-4c3ea5-kserve-77d88775bf-gsngn"
Nov 02 03:32:15.961145 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:15.961069    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-4c3ea5/tenantcfg-model-4c3ea5-kserve-77d88775bf-gsngn"
Nov 02 03:32:23.132527 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:23.132491    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-018375-6b46774f5f-dd49t"]
Nov 02 03:32:23.233770 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:23.233736    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-018375-778658cdb7-cr9bl"]
Nov 02 03:32:24.779335 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:24.779294    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-abc348-6fbfd887f5-4zkk2"]
Nov 02 03:32:24.787703 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:24.787667    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-abc348-6fbfd887f5-4zkk2"
Nov 02 03:32:24.826339 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:24.826306    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-abc348-6fbfd887f5-4zkk2"]
Nov 02 03:32:24.884401 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:24.884371    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-abc348-dockercfg-gs696\""
Nov 02 03:32:24.886229 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:24.886196    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-abc348\""
Nov 02 03:32:24.887700 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:24.887663    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwft2\" (UniqueName: \"kubernetes.io/projected/90ea88ed-de1d-490d-8b24-03dcdacca466-kube-api-access-hwft2\") pod \"payload-pre-processing-e2e-ipp-a-abc348-6fbfd887f5-4zkk2\" (UID: \"90ea88ed-de1d-490d-8b24-03dcdacca466\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-abc348-6fbfd887f5-4zkk2"
Nov 02 03:32:24.888212 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:24.888182    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90ea88ed-de1d-490d-8b24-03dcdacca466-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-abc348-6fbfd887f5-4zkk2\" (UID: \"90ea88ed-de1d-490d-8b24-03dcdacca466\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-abc348-6fbfd887f5-4zkk2"
Nov 02 03:32:24.989245 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:24.989211    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90ea88ed-de1d-490d-8b24-03dcdacca466-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-abc348-6fbfd887f5-4zkk2\" (UID: \"90ea88ed-de1d-490d-8b24-03dcdacca466\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-abc348-6fbfd887f5-4zkk2"
Nov 02 03:32:24.989397 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:24.989255    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwft2\" (UniqueName: \"kubernetes.io/projected/90ea88ed-de1d-490d-8b24-03dcdacca466-kube-api-access-hwft2\") pod \"payload-pre-processing-e2e-ipp-a-abc348-6fbfd887f5-4zkk2\" (UID: \"90ea88ed-de1d-490d-8b24-03dcdacca466\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-abc348-6fbfd887f5-4zkk2"
Nov 02 03:32:24.989945 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:24.989925    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90ea88ed-de1d-490d-8b24-03dcdacca466-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-abc348-6fbfd887f5-4zkk2\" (UID: \"90ea88ed-de1d-490d-8b24-03dcdacca466\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-abc348-6fbfd887f5-4zkk2"
Nov 02 03:32:25.046765 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:25.046687    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwft2\" (UniqueName: \"kubernetes.io/projected/90ea88ed-de1d-490d-8b24-03dcdacca466-kube-api-access-hwft2\") pod \"payload-pre-processing-e2e-ipp-a-abc348-6fbfd887f5-4zkk2\" (UID: \"90ea88ed-de1d-490d-8b24-03dcdacca466\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-abc348-6fbfd887f5-4zkk2"
Nov 02 03:32:25.105163 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:25.105128    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-abc348-6fbfd887f5-4zkk2"
Nov 02 03:32:25.491195 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:32:25.491163    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90ea88ed_de1d_490d_8b24_03dcdacca466.slice/crio-b254748b83aac2f6ee1dbc13d6329788b8a79c42c49ba6cf3589046209e7029f WatchSource:0}: Error finding container b254748b83aac2f6ee1dbc13d6329788b8a79c42c49ba6cf3589046209e7029f: Status 404 returned error can't find the container with id b254748b83aac2f6ee1dbc13d6329788b8a79c42c49ba6cf3589046209e7029f
Nov 02 03:32:25.491438 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:25.491410    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-abc348-6fbfd887f5-4zkk2"]
Nov 02 03:32:25.517777 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:25.517751    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-abc348-b97879cd4-b9nkq"]
Nov 02 03:32:25.523388 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:25.523368    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-abc348-b97879cd4-b9nkq"
Nov 02 03:32:25.596722 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:25.596683    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8237198-5111-435d-a8d8-237ab0bbc48e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-abc348-b97879cd4-b9nkq\" (UID: \"f8237198-5111-435d-a8d8-237ab0bbc48e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-abc348-b97879cd4-b9nkq"
Nov 02 03:32:25.597064 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:25.597034    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrf2m\" (UniqueName: \"kubernetes.io/projected/f8237198-5111-435d-a8d8-237ab0bbc48e-kube-api-access-rrf2m\") pod \"payload-processing-e2e-ipp-a-abc348-b97879cd4-b9nkq\" (UID: \"f8237198-5111-435d-a8d8-237ab0bbc48e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-abc348-b97879cd4-b9nkq"
Nov 02 03:32:25.602620 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:25.602598    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-abc348-b97879cd4-b9nkq"]
Nov 02 03:32:25.698074 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:25.698045    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrf2m\" (UniqueName: \"kubernetes.io/projected/f8237198-5111-435d-a8d8-237ab0bbc48e-kube-api-access-rrf2m\") pod \"payload-processing-e2e-ipp-a-abc348-b97879cd4-b9nkq\" (UID: \"f8237198-5111-435d-a8d8-237ab0bbc48e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-abc348-b97879cd4-b9nkq"
Nov 02 03:32:25.698245 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:25.698122    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8237198-5111-435d-a8d8-237ab0bbc48e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-abc348-b97879cd4-b9nkq\" (UID: \"f8237198-5111-435d-a8d8-237ab0bbc48e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-abc348-b97879cd4-b9nkq"
Nov 02 03:32:25.698819 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:25.698800    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8237198-5111-435d-a8d8-237ab0bbc48e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-abc348-b97879cd4-b9nkq\" (UID: \"f8237198-5111-435d-a8d8-237ab0bbc48e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-abc348-b97879cd4-b9nkq"
Nov 02 03:32:25.756747 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:25.756681    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrf2m\" (UniqueName: \"kubernetes.io/projected/f8237198-5111-435d-a8d8-237ab0bbc48e-kube-api-access-rrf2m\") pod \"payload-processing-e2e-ipp-a-abc348-b97879cd4-b9nkq\" (UID: \"f8237198-5111-435d-a8d8-237ab0bbc48e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-abc348-b97879cd4-b9nkq"
Nov 02 03:32:25.865800 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:25.865760    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-abc348-b97879cd4-b9nkq"
Nov 02 03:32:26.037076 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:26.036961    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-abc348-6fbfd887f5-4zkk2" event={"ID":"90ea88ed-de1d-490d-8b24-03dcdacca466","Type":"ContainerStarted","Data":"30c1b987bfbbc906f7d3360824c039251075387a9770d8f2bb22287e9824c8c9"}
Nov 02 03:32:26.037241 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:26.037099    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-abc348-6fbfd887f5-4zkk2" event={"ID":"90ea88ed-de1d-490d-8b24-03dcdacca466","Type":"ContainerStarted","Data":"b254748b83aac2f6ee1dbc13d6329788b8a79c42c49ba6cf3589046209e7029f"}
Nov 02 03:32:26.037313 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:26.037286    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-abc348-6fbfd887f5-4zkk2"
Nov 02 03:32:26.155380 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:26.155318    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-abc348-6fbfd887f5-4zkk2" podStartSLOduration=2.155298408 podStartE2EDuration="2.155298408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:32:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:32:26.151182479 +0000 UTC m=+2686.729789858" watchObservedRunningTime="2026-11-02 03:32:26.155298408 +0000 UTC m=+2686.733905792"
Nov 02 03:32:26.299402 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:26.297597    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-abc348-b97879cd4-b9nkq"]
Nov 02 03:32:26.304009 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:32:26.302625    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8237198_5111_435d_a8d8_237ab0bbc48e.slice/crio-feb22109ea8c0c2c21346981b682c6814962d60a8f031ba9bd404fad590ce50a WatchSource:0}: Error finding container feb22109ea8c0c2c21346981b682c6814962d60a8f031ba9bd404fad590ce50a: Status 404 returned error can't find the container with id feb22109ea8c0c2c21346981b682c6814962d60a8f031ba9bd404fad590ce50a
Nov 02 03:32:26.774150 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:26.774121    4496 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-gzc2f_bd8a1b95-491e-44ea-988a-e4437c386a1a/payload-processing/0.log"
Nov 02 03:32:27.026373 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:27.026283    4496 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-gzc2f_bd8a1b95-491e-44ea-988a-e4437c386a1a/payload-processing/0.log"
Nov 02 03:32:27.050115 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:27.050078    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-abc348-b97879cd4-b9nkq" event={"ID":"f8237198-5111-435d-a8d8-237ab0bbc48e","Type":"ContainerStarted","Data":"a00df95dee4852332e6fe55c2734e2cbe2ee04ef44aee1e01a6ace8016386136"}
Nov 02 03:32:27.050394 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:27.050368    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-abc348-b97879cd4-b9nkq" event={"ID":"f8237198-5111-435d-a8d8-237ab0bbc48e","Type":"ContainerStarted","Data":"feb22109ea8c0c2c21346981b682c6814962d60a8f031ba9bd404fad590ce50a"}
Nov 02 03:32:28.062100 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:28.062062    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-abc348-b97879cd4-b9nkq"
Nov 02 03:32:28.133099 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:28.133054    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-018375-6b46774f5f-dd49t" podUID="ab645ce9-6586-4caa-b502-421e3c95652d" containerName="payload-processing" containerID="cri-o://41d06a0e8a7f3cb07c6754850aa0a15da30fc9e04934c1435a6f198918baea82" gracePeriod=25
Nov 02 03:32:28.234968 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:28.234925    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-018375-778658cdb7-cr9bl" podUID="4880f85e-a7cd-43b2-8040-9ca6dbae82ae" containerName="payload-pre-processing" containerID="cri-o://99a397351ac2f2ed9c1c14e7afa8e4ec605c019ff86f98d6700f92b6fda718cf" gracePeriod=25
Nov 02 03:32:28.465137 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:28.464785    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-018375-6b46774f5f-dd49t"
Nov 02 03:32:28.543655 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:28.543622    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab645ce9-6586-4caa-b502-421e3c95652d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab645ce9-6586-4caa-b502-421e3c95652d-plugins-config-volume\") pod \"ab645ce9-6586-4caa-b502-421e3c95652d\" (UID: \"ab645ce9-6586-4caa-b502-421e3c95652d\") "
Nov 02 03:32:28.543807 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:28.543780    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab645ce9-6586-4caa-b502-421e3c95652d-kube-api-access-cssdm\" (UniqueName: \"kubernetes.io/projected/ab645ce9-6586-4caa-b502-421e3c95652d-kube-api-access-cssdm\") pod \"ab645ce9-6586-4caa-b502-421e3c95652d\" (UID: \"ab645ce9-6586-4caa-b502-421e3c95652d\") "
Nov 02 03:32:28.544051 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:28.543990    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab645ce9-6586-4caa-b502-421e3c95652d-plugins-config-volume" pod "ab645ce9-6586-4caa-b502-421e3c95652d" (UID: "ab645ce9-6586-4caa-b502-421e3c95652d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:32:28.544392 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:28.544373    4496 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab645ce9-6586-4caa-b502-421e3c95652d-plugins-config-volume\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:32:28.546410 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:28.546389    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab645ce9-6586-4caa-b502-421e3c95652d-kube-api-access-cssdm" pod "ab645ce9-6586-4caa-b502-421e3c95652d" (UID: "ab645ce9-6586-4caa-b502-421e3c95652d"). InnerVolumeSpecName "kube-api-access-cssdm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:32:28.552473 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:28.552454    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-018375-778658cdb7-cr9bl"
Nov 02 03:32:28.584853 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:28.584757    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-abc348-b97879cd4-b9nkq" podStartSLOduration=3.584739882 podStartE2EDuration="3.584739882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:32:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:32:27.172522646 +0000 UTC m=+2687.751130030" watchObservedRunningTime="2026-11-02 03:32:28.584739882 +0000 UTC m=+2689.163347262"
Nov 02 03:32:28.681863 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:28.681453    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4880f85e-a7cd-43b2-8040-9ca6dbae82ae-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4880f85e-a7cd-43b2-8040-9ca6dbae82ae-plugins-config-volume\") pod \"4880f85e-a7cd-43b2-8040-9ca6dbae82ae\" (UID: \"4880f85e-a7cd-43b2-8040-9ca6dbae82ae\") "
Nov 02 03:32:28.681863 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:28.681609    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4880f85e-a7cd-43b2-8040-9ca6dbae82ae-kube-api-access-jjkw5\" (UniqueName: \"kubernetes.io/projected/4880f85e-a7cd-43b2-8040-9ca6dbae82ae-kube-api-access-jjkw5\") pod \"4880f85e-a7cd-43b2-8040-9ca6dbae82ae\" (UID: \"4880f85e-a7cd-43b2-8040-9ca6dbae82ae\") "
Nov 02 03:32:28.682712 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:28.682287    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cssdm\" (UniqueName: \"kubernetes.io/projected/ab645ce9-6586-4caa-b502-421e3c95652d-kube-api-access-cssdm\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:32:28.695671 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:28.695639    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4880f85e-a7cd-43b2-8040-9ca6dbae82ae-plugins-config-volume" pod "4880f85e-a7cd-43b2-8040-9ca6dbae82ae" (UID: "4880f85e-a7cd-43b2-8040-9ca6dbae82ae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:32:28.731202 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:28.731163    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4880f85e-a7cd-43b2-8040-9ca6dbae82ae-kube-api-access-jjkw5" pod "4880f85e-a7cd-43b2-8040-9ca6dbae82ae" (UID: "4880f85e-a7cd-43b2-8040-9ca6dbae82ae"). InnerVolumeSpecName "kube-api-access-jjkw5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:32:28.783560 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:28.783523    4496 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4880f85e-a7cd-43b2-8040-9ca6dbae82ae-plugins-config-volume\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:32:28.783560 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:28.783552    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjkw5\" (UniqueName: \"kubernetes.io/projected/4880f85e-a7cd-43b2-8040-9ca6dbae82ae-kube-api-access-jjkw5\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:32:28.797824 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:28.797789    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-nmzqt"]
Nov 02 03:32:28.798079 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:28.798041    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-nmzqt" podUID="9b9ba5f1-2280-4916-81d7-231b805e959e" containerName="istio-proxy" containerID="cri-o://244e379c2d48153953ac24746d19b3947af2d45cee8f5d73a391c6723770c63b" gracePeriod=30
Nov 02 03:32:29.072457 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:29.072418    4496 generic.go:356] "Generic (PLEG): container finished" podID="ab645ce9-6586-4caa-b502-421e3c95652d" containerID="41d06a0e8a7f3cb07c6754850aa0a15da30fc9e04934c1435a6f198918baea82" exitCode=0
Nov 02 03:32:29.072969 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:29.072494    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-018375-6b46774f5f-dd49t"
Nov 02 03:32:29.072969 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:29.072504    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-018375-6b46774f5f-dd49t" event={"ID":"ab645ce9-6586-4caa-b502-421e3c95652d","Type":"ContainerDied","Data":"41d06a0e8a7f3cb07c6754850aa0a15da30fc9e04934c1435a6f198918baea82"}
Nov 02 03:32:29.072969 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:29.072583    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-018375-6b46774f5f-dd49t" event={"ID":"ab645ce9-6586-4caa-b502-421e3c95652d","Type":"ContainerDied","Data":"e16387a16c4fca72fa58d479a2b51ca370df090b972830fbbaf676272aab8b7c"}
Nov 02 03:32:29.072969 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:29.072620    4496 scope.go:122] "RemoveContainer" containerID="41d06a0e8a7f3cb07c6754850aa0a15da30fc9e04934c1435a6f198918baea82"
Nov 02 03:32:29.074230 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:29.074207    4496 generic.go:356] "Generic (PLEG): container finished" podID="4880f85e-a7cd-43b2-8040-9ca6dbae82ae" containerID="99a397351ac2f2ed9c1c14e7afa8e4ec605c019ff86f98d6700f92b6fda718cf" exitCode=0
Nov 02 03:32:29.074314 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:29.074256    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-018375-778658cdb7-cr9bl"
Nov 02 03:32:29.074314 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:29.074286    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-018375-778658cdb7-cr9bl" event={"ID":"4880f85e-a7cd-43b2-8040-9ca6dbae82ae","Type":"ContainerDied","Data":"99a397351ac2f2ed9c1c14e7afa8e4ec605c019ff86f98d6700f92b6fda718cf"}
Nov 02 03:32:29.074417 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:29.074336    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-018375-778658cdb7-cr9bl" event={"ID":"4880f85e-a7cd-43b2-8040-9ca6dbae82ae","Type":"ContainerDied","Data":"49d1cf7540791754be18938539a9ce9e178f7bd520a9201b204fb11e4bd72ad4"}
Nov 02 03:32:29.092873 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:29.092855    4496 scope.go:122] "RemoveContainer" containerID="41d06a0e8a7f3cb07c6754850aa0a15da30fc9e04934c1435a6f198918baea82"
Nov 02 03:32:29.093167 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:32:29.093145    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41d06a0e8a7f3cb07c6754850aa0a15da30fc9e04934c1435a6f198918baea82\": container with ID starting with 41d06a0e8a7f3cb07c6754850aa0a15da30fc9e04934c1435a6f198918baea82 not found: ID does not exist" containerID="41d06a0e8a7f3cb07c6754850aa0a15da30fc9e04934c1435a6f198918baea82"
Nov 02 03:32:29.093236 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:29.093179    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41d06a0e8a7f3cb07c6754850aa0a15da30fc9e04934c1435a6f198918baea82"} err="failed to get container status \"41d06a0e8a7f3cb07c6754850aa0a15da30fc9e04934c1435a6f198918baea82\": rpc error: code = NotFound desc = could not find container \"41d06a0e8a7f3cb07c6754850aa0a15da30fc9e04934c1435a6f198918baea82\": container with ID starting with 41d06a0e8a7f3cb07c6754850aa0a15da30fc9e04934c1435a6f198918baea82 not found: ID does not exist"
Nov 02 03:32:29.093236 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:29.093202    4496 scope.go:122] "RemoveContainer" containerID="99a397351ac2f2ed9c1c14e7afa8e4ec605c019ff86f98d6700f92b6fda718cf"
Nov 02 03:32:29.104483 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:29.104465    4496 scope.go:122] "RemoveContainer" containerID="99a397351ac2f2ed9c1c14e7afa8e4ec605c019ff86f98d6700f92b6fda718cf"
Nov 02 03:32:29.104811 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:32:29.104790    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99a397351ac2f2ed9c1c14e7afa8e4ec605c019ff86f98d6700f92b6fda718cf\": container with ID starting with 99a397351ac2f2ed9c1c14e7afa8e4ec605c019ff86f98d6700f92b6fda718cf not found: ID does not exist" containerID="99a397351ac2f2ed9c1c14e7afa8e4ec605c019ff86f98d6700f92b6fda718cf"
Nov 02 03:32:29.104885 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:29.104819    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99a397351ac2f2ed9c1c14e7afa8e4ec605c019ff86f98d6700f92b6fda718cf"} err="failed to get container status \"99a397351ac2f2ed9c1c14e7afa8e4ec605c019ff86f98d6700f92b6fda718cf\": rpc error: code = NotFound desc = could not find container \"99a397351ac2f2ed9c1c14e7afa8e4ec605c019ff86f98d6700f92b6fda718cf\": container with ID starting with 99a397351ac2f2ed9c1c14e7afa8e4ec605c019ff86f98d6700f92b6fda718cf not found: ID does not exist"
Nov 02 03:32:29.240857 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:29.240827    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-018375-778658cdb7-cr9bl"]
Nov 02 03:32:29.246415 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:29.246387    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-018375-778658cdb7-cr9bl"]
Nov 02 03:32:29.362277 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:29.362198    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-018375-6b46774f5f-dd49t"]
Nov 02 03:32:29.367731 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:29.367703    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-018375-6b46774f5f-dd49t"]
Nov 02 03:32:29.577119 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:29.577086    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4880f85e-a7cd-43b2-8040-9ca6dbae82ae" path="/var/lib/kubelet/pods/4880f85e-a7cd-43b2-8040-9ca6dbae82ae/volumes"
Nov 02 03:32:29.577521 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:29.577506    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab645ce9-6586-4caa-b502-421e3c95652d" path="/var/lib/kubelet/pods/ab645ce9-6586-4caa-b502-421e3c95652d/volumes"
Nov 02 03:32:30.037403 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:30.037358    4496 patch_prober.go:29] interesting pod/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-nmzqt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 03:32:30.037617 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:30.037436    4496 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-nmzqt" podUID="9b9ba5f1-2280-4916-81d7-231b805e959e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 03:32:33.528160 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:32:33.528125    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 03:32:34.052914 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:34.052881    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4bf04623-hfwcl"]
Nov 02 03:32:34.071334 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:34.071279    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-nmzqt"
Nov 02 03:32:34.101218 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:34.101189    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4bf04623-hfwcl"]
Nov 02 03:32:34.119547 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:34.119521    4496 generic.go:356] "Generic (PLEG): container finished" podID="9b9ba5f1-2280-4916-81d7-231b805e959e" containerID="244e379c2d48153953ac24746d19b3947af2d45cee8f5d73a391c6723770c63b" exitCode=0
Nov 02 03:32:34.119684 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:34.119567    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-nmzqt"
Nov 02 03:32:34.119684 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:34.119600    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-nmzqt" event={"ID":"9b9ba5f1-2280-4916-81d7-231b805e959e","Type":"ContainerDied","Data":"244e379c2d48153953ac24746d19b3947af2d45cee8f5d73a391c6723770c63b"}
Nov 02 03:32:34.119684 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:34.119654    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-nmzqt" event={"ID":"9b9ba5f1-2280-4916-81d7-231b805e959e","Type":"ContainerDied","Data":"dcb74053a57dc3f16abc881aac6fc309a39c967a2c8fd6d53cb448fca8958050"}
Nov 02 03:32:34.119868 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:34.119685    4496 scope.go:122] "RemoveContainer" containerID="244e379c2d48153953ac24746d19b3947af2d45cee8f5d73a391c6723770c63b"
Nov 02 03:32:34.130619 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:34.130593    4496 scope.go:122] "RemoveContainer" containerID="244e379c2d48153953ac24746d19b3947af2d45cee8f5d73a391c6723770c63b"
Nov 02 03:32:34.130903 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:32:34.130884    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"244e379c2d48153953ac24746d19b3947af2d45cee8f5d73a391c6723770c63b\": container with ID starting with 244e379c2d48153953ac24746d19b3947af2d45cee8f5d73a391c6723770c63b not found: ID does not exist" containerID="244e379c2d48153953ac24746d19b3947af2d45cee8f5d73a391c6723770c63b"
Nov 02 03:32:34.130954 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:34.130912    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"244e379c2d48153953ac24746d19b3947af2d45cee8f5d73a391c6723770c63b"} err="failed to get container status \"244e379c2d48153953ac24746d19b3947af2d45cee8f5d73a391c6723770c63b\": rpc error: code = NotFound desc = could not find container \"244e379c2d48153953ac24746d19b3947af2d45cee8f5d73a391c6723770c63b\": container with ID starting with 244e379c2d48153953ac24746d19b3947af2d45cee8f5d73a391c6723770c63b not found: ID does not exist"
Nov 02 03:32:34.147871 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:34.147849    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b9ba5f1-2280-4916-81d7-231b805e959e-istio-token\" (UniqueName: \"kubernetes.io/projected/9b9ba5f1-2280-4916-81d7-231b805e959e-istio-token\") pod \"9b9ba5f1-2280-4916-81d7-231b805e959e\" (UID: \"9b9ba5f1-2280-4916-81d7-231b805e959e\") "
Nov 02 03:32:34.147962 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:34.147922    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b9ba5f1-2280-4916-81d7-231b805e959e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b9ba5f1-2280-4916-81d7-231b805e959e-workload-certs\") pod \"9b9ba5f1-2280-4916-81d7-231b805e959e\" (UID: \"9b9ba5f1-2280-4916-81d7-231b805e959e\") "
Nov 02 03:32:34.148003 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:34.147974    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b9ba5f1-2280-4916-81d7-231b805e959e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b9ba5f1-2280-4916-81d7-231b805e959e-workload-socket\") pod \"9b9ba5f1-2280-4916-81d7-231b805e959e\" (UID: \"9b9ba5f1-2280-4916-81d7-231b805e959e\") "
Nov 02 03:32:34.148081 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:34.148069    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b9ba5f1-2280-4916-81d7-231b805e959e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b9ba5f1-2280-4916-81d7-231b805e959e-istio-envoy\") pod \"9b9ba5f1-2280-4916-81d7-231b805e959e\" (UID: \"9b9ba5f1-2280-4916-81d7-231b805e959e\") "
Nov 02 03:32:34.148141 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:34.148131    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9b9ba5f1-2280-4916-81d7-231b805e959e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b9ba5f1-2280-4916-81d7-231b805e959e-istio-podinfo\") pod \"9b9ba5f1-2280-4916-81d7-231b805e959e\" (UID: \"9b9ba5f1-2280-4916-81d7-231b805e959e\") "
Nov 02 03:32:34.148207 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:34.148189    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b9ba5f1-2280-4916-81d7-231b805e959e-kube-api-access-ts4t5\" (UniqueName: \"kubernetes.io/projected/9b9ba5f1-2280-4916-81d7-231b805e959e-kube-api-access-ts4t5\") pod \"9b9ba5f1-2280-4916-81d7-231b805e959e\" (UID: \"9b9ba5f1-2280-4916-81d7-231b805e959e\") "
Nov 02 03:32:34.148330 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:34.148226    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b9ba5f1-2280-4916-81d7-231b805e959e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b9ba5f1-2280-4916-81d7-231b805e959e-credential-socket\") pod \"9b9ba5f1-2280-4916-81d7-231b805e959e\" (UID: \"9b9ba5f1-2280-4916-81d7-231b805e959e\") "
Nov 02 03:32:34.148330 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:34.148263    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b9ba5f1-2280-4916-81d7-231b805e959e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b9ba5f1-2280-4916-81d7-231b805e959e-istio-data\") pod \"9b9ba5f1-2280-4916-81d7-231b805e959e\" (UID: \"9b9ba5f1-2280-4916-81d7-231b805e959e\") "
Nov 02 03:32:34.148330 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:34.148296    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b9ba5f1-2280-4916-81d7-231b805e959e-workload-certs" pod "9b9ba5f1-2280-4916-81d7-231b805e959e" (UID: "9b9ba5f1-2280-4916-81d7-231b805e959e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:32:34.148518 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:34.148314    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b9ba5f1-2280-4916-81d7-231b805e959e-workload-socket" pod "9b9ba5f1-2280-4916-81d7-231b805e959e" (UID: "9b9ba5f1-2280-4916-81d7-231b805e959e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:32:34.148518 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:34.148385    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b9ba5f1-2280-4916-81d7-231b805e959e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b9ba5f1-2280-4916-81d7-231b805e959e-istiod-ca-cert\") pod \"9b9ba5f1-2280-4916-81d7-231b805e959e\" (UID: \"9b9ba5f1-2280-4916-81d7-231b805e959e\") "
Nov 02 03:32:34.148630 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:34.148514    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b9ba5f1-2280-4916-81d7-231b805e959e-istio-data" pod "9b9ba5f1-2280-4916-81d7-231b805e959e" (UID: "9b9ba5f1-2280-4916-81d7-231b805e959e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:32:34.148682 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:34.148661    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b9ba5f1-2280-4916-81d7-231b805e959e-credential-socket" pod "9b9ba5f1-2280-4916-81d7-231b805e959e" (UID: "9b9ba5f1-2280-4916-81d7-231b805e959e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:32:34.148765 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:34.148742    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b9ba5f1-2280-4916-81d7-231b805e959e-istiod-ca-cert" pod "9b9ba5f1-2280-4916-81d7-231b805e959e" (UID: "9b9ba5f1-2280-4916-81d7-231b805e959e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:32:34.149090 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:34.149067    4496 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b9ba5f1-2280-4916-81d7-231b805e959e-istiod-ca-cert\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:32:34.149156 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:34.149098    4496 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b9ba5f1-2280-4916-81d7-231b805e959e-workload-certs\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:32:34.149156 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:34.149117    4496 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b9ba5f1-2280-4916-81d7-231b805e959e-workload-socket\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:32:34.149156 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:34.149134    4496 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b9ba5f1-2280-4916-81d7-231b805e959e-credential-socket\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:32:34.149156 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:34.149149    4496 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b9ba5f1-2280-4916-81d7-231b805e959e-istio-data\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:32:34.150117 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:34.150096    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b9ba5f1-2280-4916-81d7-231b805e959e-istio-token" pod "9b9ba5f1-2280-4916-81d7-231b805e959e" (UID: "9b9ba5f1-2280-4916-81d7-231b805e959e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:32:34.150284 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:34.150260    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b9ba5f1-2280-4916-81d7-231b805e959e-istio-envoy" pod "9b9ba5f1-2280-4916-81d7-231b805e959e" (UID: "9b9ba5f1-2280-4916-81d7-231b805e959e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:32:34.150713 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:34.150678    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9b9ba5f1-2280-4916-81d7-231b805e959e-istio-podinfo" pod "9b9ba5f1-2280-4916-81d7-231b805e959e" (UID: "9b9ba5f1-2280-4916-81d7-231b805e959e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 03:32:34.150713 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:34.150698    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b9ba5f1-2280-4916-81d7-231b805e959e-kube-api-access-ts4t5" pod "9b9ba5f1-2280-4916-81d7-231b805e959e" (UID: "9b9ba5f1-2280-4916-81d7-231b805e959e"). InnerVolumeSpecName "kube-api-access-ts4t5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:32:34.250697 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:34.250664    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ts4t5\" (UniqueName: \"kubernetes.io/projected/9b9ba5f1-2280-4916-81d7-231b805e959e-kube-api-access-ts4t5\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:32:34.250697 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:34.250694    4496 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b9ba5f1-2280-4916-81d7-231b805e959e-istio-token\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:32:34.250697 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:34.250704    4496 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b9ba5f1-2280-4916-81d7-231b805e959e-istio-envoy\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:32:34.250916 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:34.250714    4496 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b9ba5f1-2280-4916-81d7-231b805e959e-istio-podinfo\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:32:34.547549 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:34.547517    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-nmzqt"]
Nov 02 03:32:34.552916 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:34.552889    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-nmzqt"]
Nov 02 03:32:35.574840 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:35.574804    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96ab300a-48eb-4c1e-9c97-db8e6cec62c0" path="/var/lib/kubelet/pods/96ab300a-48eb-4c1e-9c97-db8e6cec62c0/volumes"
Nov 02 03:32:35.575378 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:35.575360    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b9ba5f1-2280-4916-81d7-231b805e959e" path="/var/lib/kubelet/pods/9b9ba5f1-2280-4916-81d7-231b805e959e/volumes"
Nov 02 03:32:37.052144 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:37.052113    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-abc348-6fbfd887f5-4zkk2"
Nov 02 03:32:39.077842 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:39.077803    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-abc348-b97879cd4-b9nkq"
Nov 02 03:32:40.161484 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:40.161454    4496 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hww8q_5c797b08-a314-44ed-8026-8a0ce1f6c36f/ovn-acl-logging/0.log"
Nov 02 03:32:40.165406 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:40.165378    4496 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hww8q_5c797b08-a314-44ed-8026-8a0ce1f6c36f/ovn-acl-logging/0.log"
Nov 02 03:32:41.425273 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:41.425240    4496 scope.go:122] "RemoveContainer" containerID="84057766c252316ad4078d71c351e0dc98e136fca66cf5887088b9b5ef328009"
Nov 02 03:32:41.435741 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:41.435717    4496 scope.go:122] "RemoveContainer" containerID="e3c7e86c2a879a568b82c2d3bdaab03da451416ebe6baae03022c6c6514252ef"
Nov 02 03:32:41.447791 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:32:41.447765    4496 scope.go:122] "RemoveContainer" containerID="969edc3ab36084d081a94c359c09108acb2507149716b733ba329d3da5607e53"
Nov 02 03:33:03.528527 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:33:03.528489    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 03:33:11.147751 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:11.147704    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9d5625-54769b969-tfkvv"]
Nov 02 03:33:11.249226 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:11.249190    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9d5625-87d7dd87-24dnh"]
Nov 02 03:33:11.805935 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:11.805891    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-045653-fvbq7"]
Nov 02 03:33:11.806601 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:11.806583    4496 cpu_manager.go:414] "containerMap: removing container" podUID="4880f85e-a7cd-43b2-8040-9ca6dbae82ae" containerName="payload-pre-processing"
Nov 02 03:33:11.806601 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:11.806601    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4880f85e-a7cd-43b2-8040-9ca6dbae82ae" containerName="payload-pre-processing"
Nov 02 03:33:11.806760 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:11.806644    4496 cpu_manager.go:414] "containerMap: removing container" podUID="ab645ce9-6586-4caa-b502-421e3c95652d" containerName="payload-processing"
Nov 02 03:33:11.806760 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:11.806650    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab645ce9-6586-4caa-b502-421e3c95652d" containerName="payload-processing"
Nov 02 03:33:11.806760 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:11.806663    4496 cpu_manager.go:414] "containerMap: removing container" podUID="9b9ba5f1-2280-4916-81d7-231b805e959e" containerName="istio-proxy"
Nov 02 03:33:11.806760 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:11.806669    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b9ba5f1-2280-4916-81d7-231b805e959e" containerName="istio-proxy"
Nov 02 03:33:11.806973 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:11.806796    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab645ce9-6586-4caa-b502-421e3c95652d" containerName="payload-processing"
Nov 02 03:33:11.806973 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:11.806805    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b9ba5f1-2280-4916-81d7-231b805e959e" containerName="istio-proxy"
Nov 02 03:33:11.806973 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:11.806814    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="4880f85e-a7cd-43b2-8040-9ca6dbae82ae" containerName="payload-pre-processing"
Nov 02 03:33:11.810300 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:11.810280    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-045653-fvbq7"
Nov 02 03:33:11.863846 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:11.863821    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9z5bk\""
Nov 02 03:33:11.864070 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:11.864046    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 03:33:11.864070 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:11.863993    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 03:33:11.864287 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:11.864266    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 03:33:11.865482 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:11.865460    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-045653-fvbq7"]
Nov 02 03:33:11.934974 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:11.934941    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5x65k\" (UniqueName: \"kubernetes.io/projected/e5b8152f-c332-42be-a2de-1fb845cdc2bf-kube-api-access-5x65k\") pod \"maas-api-revoke-keys-e2e-ipp-a-045653-fvbq7\" (UID: \"e5b8152f-c332-42be-a2de-1fb845cdc2bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-045653-fvbq7"
Nov 02 03:33:11.935186 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:11.935122    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5b8152f-c332-42be-a2de-1fb845cdc2bf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-045653-fvbq7\" (UID: \"e5b8152f-c332-42be-a2de-1fb845cdc2bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-045653-fvbq7"
Nov 02 03:33:11.935285 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:11.935267    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5b8152f-c332-42be-a2de-1fb845cdc2bf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-045653-fvbq7\" (UID: \"e5b8152f-c332-42be-a2de-1fb845cdc2bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-045653-fvbq7"
Nov 02 03:33:12.037064 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:12.036994    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5b8152f-c332-42be-a2de-1fb845cdc2bf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-045653-fvbq7\" (UID: \"e5b8152f-c332-42be-a2de-1fb845cdc2bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-045653-fvbq7"
Nov 02 03:33:12.037258 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:12.037120    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5x65k\" (UniqueName: \"kubernetes.io/projected/e5b8152f-c332-42be-a2de-1fb845cdc2bf-kube-api-access-5x65k\") pod \"maas-api-revoke-keys-e2e-ipp-a-045653-fvbq7\" (UID: \"e5b8152f-c332-42be-a2de-1fb845cdc2bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-045653-fvbq7"
Nov 02 03:33:12.037323 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:12.037258    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5b8152f-c332-42be-a2de-1fb845cdc2bf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-045653-fvbq7\" (UID: \"e5b8152f-c332-42be-a2de-1fb845cdc2bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-045653-fvbq7"
Nov 02 03:33:12.037774 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:12.037754    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5b8152f-c332-42be-a2de-1fb845cdc2bf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-045653-fvbq7\" (UID: \"e5b8152f-c332-42be-a2de-1fb845cdc2bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-045653-fvbq7"
Nov 02 03:33:12.039976 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:12.039948    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5b8152f-c332-42be-a2de-1fb845cdc2bf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-045653-fvbq7\" (UID: \"e5b8152f-c332-42be-a2de-1fb845cdc2bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-045653-fvbq7"
Nov 02 03:33:12.094787 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:12.094724    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5x65k\" (UniqueName: \"kubernetes.io/projected/e5b8152f-c332-42be-a2de-1fb845cdc2bf-kube-api-access-5x65k\") pod \"maas-api-revoke-keys-e2e-ipp-a-045653-fvbq7\" (UID: \"e5b8152f-c332-42be-a2de-1fb845cdc2bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-045653-fvbq7"
Nov 02 03:33:12.123807 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:12.123778    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-045653-fvbq7"
Nov 02 03:33:12.532973 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:12.532940    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-045653-fvbq7"]
Nov 02 03:33:12.533427 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:33:12.533407    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5b8152f_c332_42be_a2de_1fb845cdc2bf.slice/crio-fa58fbc7193c91f0fb80858e7702ced6c0e58c4d654ce1ec01fa4cc7785f92b5 WatchSource:0}: Error finding container fa58fbc7193c91f0fb80858e7702ced6c0e58c4d654ce1ec01fa4cc7785f92b5: Status 404 returned error can't find the container with id fa58fbc7193c91f0fb80858e7702ced6c0e58c4d654ce1ec01fa4cc7785f92b5
Nov 02 03:33:13.421089 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:13.421048    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-045653-fvbq7" event={"ID":"e5b8152f-c332-42be-a2de-1fb845cdc2bf","Type":"ContainerStarted","Data":"6d9c8f6917cb46568343696957cf6816a16612aec548514bba6054065d1f9a35"}
Nov 02 03:33:13.421285 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:13.421114    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-045653-fvbq7" event={"ID":"e5b8152f-c332-42be-a2de-1fb845cdc2bf","Type":"ContainerStarted","Data":"fa58fbc7193c91f0fb80858e7702ced6c0e58c4d654ce1ec01fa4cc7785f92b5"}
Nov 02 03:33:13.536472 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:13.536429    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-045653-fvbq7" podStartSLOduration=2.536413462 podStartE2EDuration="2.536413462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:33:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:33:13.535509437 +0000 UTC m=+2734.114116818" watchObservedRunningTime="2026-11-02 03:33:13.536413462 +0000 UTC m=+2734.115020841"
Nov 02 03:33:14.395774 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:14.395740    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6a5285-wvfdm"]
Nov 02 03:33:14.430751 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:14.430718    4496 generic.go:356] "Generic (PLEG): container finished" podID="e5b8152f-c332-42be-a2de-1fb845cdc2bf" containerID="6d9c8f6917cb46568343696957cf6816a16612aec548514bba6054065d1f9a35" exitCode=0
Nov 02 03:33:14.430930 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:14.430817    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-045653-fvbq7" event={"ID":"e5b8152f-c332-42be-a2de-1fb845cdc2bf","Type":"ContainerDied","Data":"6d9c8f6917cb46568343696957cf6816a16612aec548514bba6054065d1f9a35"}
Nov 02 03:33:14.443908 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:14.443881    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6a5285-wvfdm"]
Nov 02 03:33:15.574850 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:15.574814    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfc9bdd3-8312-426c-932a-230679dc4a8d" path="/var/lib/kubelet/pods/cfc9bdd3-8312-426c-932a-230679dc4a8d/volumes"
Nov 02 03:33:15.590892 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:15.590868    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-045653-fvbq7"
Nov 02 03:33:15.682907 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:15.682871    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5b8152f-c332-42be-a2de-1fb845cdc2bf-kube-api-access-5x65k\" (UniqueName: \"kubernetes.io/projected/e5b8152f-c332-42be-a2de-1fb845cdc2bf-kube-api-access-5x65k\") pod \"e5b8152f-c332-42be-a2de-1fb845cdc2bf\" (UID: \"e5b8152f-c332-42be-a2de-1fb845cdc2bf\") "
Nov 02 03:33:15.683085 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:15.682942    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5b8152f-c332-42be-a2de-1fb845cdc2bf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5b8152f-c332-42be-a2de-1fb845cdc2bf-maas-api-xks-ca\") pod \"e5b8152f-c332-42be-a2de-1fb845cdc2bf\" (UID: \"e5b8152f-c332-42be-a2de-1fb845cdc2bf\") "
Nov 02 03:33:15.683085 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:15.683062    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5b8152f-c332-42be-a2de-1fb845cdc2bf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5b8152f-c332-42be-a2de-1fb845cdc2bf-maas-api-openshift-ca\") pod \"e5b8152f-c332-42be-a2de-1fb845cdc2bf\" (UID: \"e5b8152f-c332-42be-a2de-1fb845cdc2bf\") "
Nov 02 03:33:15.683391 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:15.683368    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5b8152f-c332-42be-a2de-1fb845cdc2bf-maas-api-openshift-ca" pod "e5b8152f-c332-42be-a2de-1fb845cdc2bf" (UID: "e5b8152f-c332-42be-a2de-1fb845cdc2bf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:33:15.684929 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:15.684905    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5b8152f-c332-42be-a2de-1fb845cdc2bf-maas-api-xks-ca" pod "e5b8152f-c332-42be-a2de-1fb845cdc2bf" (UID: "e5b8152f-c332-42be-a2de-1fb845cdc2bf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 03:33:15.685047 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:15.685009    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5b8152f-c332-42be-a2de-1fb845cdc2bf-kube-api-access-5x65k" pod "e5b8152f-c332-42be-a2de-1fb845cdc2bf" (UID: "e5b8152f-c332-42be-a2de-1fb845cdc2bf"). InnerVolumeSpecName "kube-api-access-5x65k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:33:15.784974 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:15.784939    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5b8152f-c332-42be-a2de-1fb845cdc2bf-maas-api-openshift-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:33:15.784974 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:15.784969    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5x65k\" (UniqueName: \"kubernetes.io/projected/e5b8152f-c332-42be-a2de-1fb845cdc2bf-kube-api-access-5x65k\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:33:15.784974 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:15.784981    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5b8152f-c332-42be-a2de-1fb845cdc2bf-maas-api-xks-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:33:16.148481 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:16.148428    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-9d5625-54769b969-tfkvv" podUID="a50fee2e-7a02-4b8d-a228-3f0dfd42ce31" containerName="payload-processing" containerID="cri-o://198cff1ef8a3928b389ef41811d2d12d383750a7622bde08154e6485d0c45da8" gracePeriod=25
Nov 02 03:33:16.250246 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:16.250210    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d5625-87d7dd87-24dnh" podUID="24f29cce-d514-49e4-9359-e1efed242080" containerName="payload-pre-processing" containerID="cri-o://6e99f1846e3108bb6d6c2f333ac48a8ed13ee6631a1a381141464def10ceeebb" gracePeriod=25
Nov 02 03:33:16.441298 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:16.441001    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-9d5625-54769b969-tfkvv"
Nov 02 03:33:16.450809 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:16.450773    4496 generic.go:356] "Generic (PLEG): container finished" podID="a50fee2e-7a02-4b8d-a228-3f0dfd42ce31" containerID="198cff1ef8a3928b389ef41811d2d12d383750a7622bde08154e6485d0c45da8" exitCode=0
Nov 02 03:33:16.450960 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:16.450826    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-9d5625-54769b969-tfkvv"
Nov 02 03:33:16.450960 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:16.450851    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-9d5625-54769b969-tfkvv" event={"ID":"a50fee2e-7a02-4b8d-a228-3f0dfd42ce31","Type":"ContainerDied","Data":"198cff1ef8a3928b389ef41811d2d12d383750a7622bde08154e6485d0c45da8"}
Nov 02 03:33:16.450960 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:16.450919    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-9d5625-54769b969-tfkvv" event={"ID":"a50fee2e-7a02-4b8d-a228-3f0dfd42ce31","Type":"ContainerDied","Data":"2756ae87ec560b0a57e9fb1d144147590d89cb3ca1e786366568090ef759d5fb"}
Nov 02 03:33:16.450960 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:16.450948    4496 scope.go:122] "RemoveContainer" containerID="198cff1ef8a3928b389ef41811d2d12d383750a7622bde08154e6485d0c45da8"
Nov 02 03:33:16.452855 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:16.452832    4496 generic.go:356] "Generic (PLEG): container finished" podID="24f29cce-d514-49e4-9359-e1efed242080" containerID="6e99f1846e3108bb6d6c2f333ac48a8ed13ee6631a1a381141464def10ceeebb" exitCode=0
Nov 02 03:33:16.452954 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:16.452902    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d5625-87d7dd87-24dnh" event={"ID":"24f29cce-d514-49e4-9359-e1efed242080","Type":"ContainerDied","Data":"6e99f1846e3108bb6d6c2f333ac48a8ed13ee6631a1a381141464def10ceeebb"}
Nov 02 03:33:16.454533 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:16.454517    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-045653-fvbq7"
Nov 02 03:33:16.454625 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:16.454562    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-045653-fvbq7" event={"ID":"e5b8152f-c332-42be-a2de-1fb845cdc2bf","Type":"ContainerDied","Data":"fa58fbc7193c91f0fb80858e7702ced6c0e58c4d654ce1ec01fa4cc7785f92b5"}
Nov 02 03:33:16.454625 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:16.454613    4496 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fa58fbc7193c91f0fb80858e7702ced6c0e58c4d654ce1ec01fa4cc7785f92b5"
Nov 02 03:33:16.464647 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:16.464528    4496 scope.go:122] "RemoveContainer" containerID="198cff1ef8a3928b389ef41811d2d12d383750a7622bde08154e6485d0c45da8"
Nov 02 03:33:16.464986 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:33:16.464958    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"198cff1ef8a3928b389ef41811d2d12d383750a7622bde08154e6485d0c45da8\": container with ID starting with 198cff1ef8a3928b389ef41811d2d12d383750a7622bde08154e6485d0c45da8 not found: ID does not exist" containerID="198cff1ef8a3928b389ef41811d2d12d383750a7622bde08154e6485d0c45da8"
Nov 02 03:33:16.465329 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:16.464997    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"198cff1ef8a3928b389ef41811d2d12d383750a7622bde08154e6485d0c45da8"} err="failed to get container status \"198cff1ef8a3928b389ef41811d2d12d383750a7622bde08154e6485d0c45da8\": rpc error: code = NotFound desc = could not find container \"198cff1ef8a3928b389ef41811d2d12d383750a7622bde08154e6485d0c45da8\": container with ID starting with 198cff1ef8a3928b389ef41811d2d12d383750a7622bde08154e6485d0c45da8 not found: ID does not exist"
Nov 02 03:33:16.493710 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:16.493653    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a50fee2e-7a02-4b8d-a228-3f0dfd42ce31-kube-api-access-4mjkc\" (UniqueName: \"kubernetes.io/projected/a50fee2e-7a02-4b8d-a228-3f0dfd42ce31-kube-api-access-4mjkc\") pod \"a50fee2e-7a02-4b8d-a228-3f0dfd42ce31\" (UID: \"a50fee2e-7a02-4b8d-a228-3f0dfd42ce31\") "
Nov 02 03:33:16.493874 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:16.493806    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a50fee2e-7a02-4b8d-a228-3f0dfd42ce31-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a50fee2e-7a02-4b8d-a228-3f0dfd42ce31-plugins-config-volume\") pod \"a50fee2e-7a02-4b8d-a228-3f0dfd42ce31\" (UID: \"a50fee2e-7a02-4b8d-a228-3f0dfd42ce31\") "
Nov 02 03:33:16.494259 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:16.494229    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a50fee2e-7a02-4b8d-a228-3f0dfd42ce31-plugins-config-volume" pod "a50fee2e-7a02-4b8d-a228-3f0dfd42ce31" (UID: "a50fee2e-7a02-4b8d-a228-3f0dfd42ce31"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:33:16.494475 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:16.494450    4496 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a50fee2e-7a02-4b8d-a228-3f0dfd42ce31-plugins-config-volume\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:33:16.496551 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:16.496524    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a50fee2e-7a02-4b8d-a228-3f0dfd42ce31-kube-api-access-4mjkc" pod "a50fee2e-7a02-4b8d-a228-3f0dfd42ce31" (UID: "a50fee2e-7a02-4b8d-a228-3f0dfd42ce31"). InnerVolumeSpecName "kube-api-access-4mjkc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:33:16.588549 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:16.588517    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d5625-87d7dd87-24dnh"
Nov 02 03:33:16.595843 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:16.595819    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mjkc\" (UniqueName: \"kubernetes.io/projected/a50fee2e-7a02-4b8d-a228-3f0dfd42ce31-kube-api-access-4mjkc\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:33:16.696884 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:16.696798    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24f29cce-d514-49e4-9359-e1efed242080-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24f29cce-d514-49e4-9359-e1efed242080-plugins-config-volume\") pod \"24f29cce-d514-49e4-9359-e1efed242080\" (UID: \"24f29cce-d514-49e4-9359-e1efed242080\") "
Nov 02 03:33:16.697073 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:16.696938    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24f29cce-d514-49e4-9359-e1efed242080-kube-api-access-twl58\" (UniqueName: \"kubernetes.io/projected/24f29cce-d514-49e4-9359-e1efed242080-kube-api-access-twl58\") pod \"24f29cce-d514-49e4-9359-e1efed242080\" (UID: \"24f29cce-d514-49e4-9359-e1efed242080\") "
Nov 02 03:33:16.697150 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:16.697128    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24f29cce-d514-49e4-9359-e1efed242080-plugins-config-volume" pod "24f29cce-d514-49e4-9359-e1efed242080" (UID: "24f29cce-d514-49e4-9359-e1efed242080"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:33:16.697699 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:16.697654    4496 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24f29cce-d514-49e4-9359-e1efed242080-plugins-config-volume\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:33:16.699182 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:16.699162    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24f29cce-d514-49e4-9359-e1efed242080-kube-api-access-twl58" pod "24f29cce-d514-49e4-9359-e1efed242080" (UID: "24f29cce-d514-49e4-9359-e1efed242080"). InnerVolumeSpecName "kube-api-access-twl58". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:33:16.799211 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:16.799182    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twl58\" (UniqueName: \"kubernetes.io/projected/24f29cce-d514-49e4-9359-e1efed242080-kube-api-access-twl58\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:33:16.892177 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:16.892144    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9d5625-54769b969-tfkvv"]
Nov 02 03:33:16.896352 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:16.896323    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9d5625-54769b969-tfkvv"]
Nov 02 03:33:17.464947 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:17.464910    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d5625-87d7dd87-24dnh" event={"ID":"24f29cce-d514-49e4-9359-e1efed242080","Type":"ContainerDied","Data":"fd4d246964b5a7d31e22e664920e09d1d1bf96194677f68540018ca696310874"}
Nov 02 03:33:17.464947 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:17.464929    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d5625-87d7dd87-24dnh"
Nov 02 03:33:17.465257 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:17.464977    4496 scope.go:122] "RemoveContainer" containerID="6e99f1846e3108bb6d6c2f333ac48a8ed13ee6631a1a381141464def10ceeebb"
Nov 02 03:33:17.578387 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:17.578354    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a50fee2e-7a02-4b8d-a228-3f0dfd42ce31" path="/var/lib/kubelet/pods/a50fee2e-7a02-4b8d-a228-3f0dfd42ce31/volumes"
Nov 02 03:33:17.633083 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:17.633046    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9d5625-87d7dd87-24dnh"]
Nov 02 03:33:17.636465 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:17.636436    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9d5625-87d7dd87-24dnh"]
Nov 02 03:33:19.585999 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:19.585965    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24f29cce-d514-49e4-9359-e1efed242080" path="/var/lib/kubelet/pods/24f29cce-d514-49e4-9359-e1efed242080/volumes"
Nov 02 03:33:21.662456 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:21.662419    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f1a692-zdkxn"]
Nov 02 03:33:21.663440 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:21.663416    4496 cpu_manager.go:414] "containerMap: removing container" podUID="a50fee2e-7a02-4b8d-a228-3f0dfd42ce31" containerName="payload-processing"
Nov 02 03:33:21.663519 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:21.663441    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a50fee2e-7a02-4b8d-a228-3f0dfd42ce31" containerName="payload-processing"
Nov 02 03:33:21.663519 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:21.663465    4496 cpu_manager.go:414] "containerMap: removing container" podUID="24f29cce-d514-49e4-9359-e1efed242080" containerName="payload-pre-processing"
Nov 02 03:33:21.663519 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:21.663473    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24f29cce-d514-49e4-9359-e1efed242080" containerName="payload-pre-processing"
Nov 02 03:33:21.663519 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:21.663504    4496 cpu_manager.go:414] "containerMap: removing container" podUID="e5b8152f-c332-42be-a2de-1fb845cdc2bf" containerName="revoke-keys"
Nov 02 03:33:21.663519 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:21.663513    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5b8152f-c332-42be-a2de-1fb845cdc2bf" containerName="revoke-keys"
Nov 02 03:33:21.663718 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:21.663664    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="a50fee2e-7a02-4b8d-a228-3f0dfd42ce31" containerName="payload-processing"
Nov 02 03:33:21.663718 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:21.663679    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="24f29cce-d514-49e4-9359-e1efed242080" containerName="payload-pre-processing"
Nov 02 03:33:21.663718 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:21.663692    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5b8152f-c332-42be-a2de-1fb845cdc2bf" containerName="revoke-keys"
Nov 02 03:33:21.669124 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:21.669096    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f1a692-zdkxn"
Nov 02 03:33:21.719658 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:21.719626    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f1a692-zdkxn"]
Nov 02 03:33:21.761813 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:21.761769    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmq9d\" (UniqueName: \"kubernetes.io/projected/9049d7e3-323a-4d1d-90db-79bdbc2c6c68-kube-api-access-hmq9d\") pod \"maas-api-revoke-keys-e2e-api-b-f1a692-zdkxn\" (UID: \"9049d7e3-323a-4d1d-90db-79bdbc2c6c68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f1a692-zdkxn"
Nov 02 03:33:21.761981 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:21.761936    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9049d7e3-323a-4d1d-90db-79bdbc2c6c68-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f1a692-zdkxn\" (UID: \"9049d7e3-323a-4d1d-90db-79bdbc2c6c68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f1a692-zdkxn"
Nov 02 03:33:21.762063 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:21.762001    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9049d7e3-323a-4d1d-90db-79bdbc2c6c68-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f1a692-zdkxn\" (UID: \"9049d7e3-323a-4d1d-90db-79bdbc2c6c68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f1a692-zdkxn"
Nov 02 03:33:21.766099 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:21.766074    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 03:33:21.766099 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:21.766084    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 03:33:21.772884 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:21.772855    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 03:33:21.773031 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:21.772974    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9z5bk\""
Nov 02 03:33:21.863232 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:21.863201    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9049d7e3-323a-4d1d-90db-79bdbc2c6c68-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f1a692-zdkxn\" (UID: \"9049d7e3-323a-4d1d-90db-79bdbc2c6c68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f1a692-zdkxn"
Nov 02 03:33:21.863438 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:21.863269    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9049d7e3-323a-4d1d-90db-79bdbc2c6c68-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f1a692-zdkxn\" (UID: \"9049d7e3-323a-4d1d-90db-79bdbc2c6c68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f1a692-zdkxn"
Nov 02 03:33:21.863438 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:21.863320    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmq9d\" (UniqueName: \"kubernetes.io/projected/9049d7e3-323a-4d1d-90db-79bdbc2c6c68-kube-api-access-hmq9d\") pod \"maas-api-revoke-keys-e2e-api-b-f1a692-zdkxn\" (UID: \"9049d7e3-323a-4d1d-90db-79bdbc2c6c68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f1a692-zdkxn"
Nov 02 03:33:21.864161 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:21.864130    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9049d7e3-323a-4d1d-90db-79bdbc2c6c68-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f1a692-zdkxn\" (UID: \"9049d7e3-323a-4d1d-90db-79bdbc2c6c68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f1a692-zdkxn"
Nov 02 03:33:21.865659 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:21.865635    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9049d7e3-323a-4d1d-90db-79bdbc2c6c68-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f1a692-zdkxn\" (UID: \"9049d7e3-323a-4d1d-90db-79bdbc2c6c68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f1a692-zdkxn"
Nov 02 03:33:21.920698 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:21.920628    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmq9d\" (UniqueName: \"kubernetes.io/projected/9049d7e3-323a-4d1d-90db-79bdbc2c6c68-kube-api-access-hmq9d\") pod \"maas-api-revoke-keys-e2e-api-b-f1a692-zdkxn\" (UID: \"9049d7e3-323a-4d1d-90db-79bdbc2c6c68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f1a692-zdkxn"
Nov 02 03:33:21.983420 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:21.983383    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f1a692-zdkxn"
Nov 02 03:33:22.362486 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:33:22.362448    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9049d7e3_323a_4d1d_90db_79bdbc2c6c68.slice/crio-b0781a3589fb03167163739c76a3f3580c95df35278ab96fa22eac5f105f9d9c WatchSource:0}: Error finding container b0781a3589fb03167163739c76a3f3580c95df35278ab96fa22eac5f105f9d9c: Status 404 returned error can't find the container with id b0781a3589fb03167163739c76a3f3580c95df35278ab96fa22eac5f105f9d9c
Nov 02 03:33:22.363802 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:22.363771    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f1a692-zdkxn"]
Nov 02 03:33:22.506177 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:22.506105    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f1a692-zdkxn" event={"ID":"9049d7e3-323a-4d1d-90db-79bdbc2c6c68","Type":"ContainerStarted","Data":"b0781a3589fb03167163739c76a3f3580c95df35278ab96fa22eac5f105f9d9c"}
Nov 02 03:33:23.527511 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:23.527409    4496 generic.go:356] "Generic (PLEG): container finished" podID="9049d7e3-323a-4d1d-90db-79bdbc2c6c68" containerID="78a0b6d27239fcf8d5a8b0a897d462ceb5e501b5b877af016da19766336a01d5" exitCode=0
Nov 02 03:33:23.528123 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:23.527553    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f1a692-zdkxn" event={"ID":"9049d7e3-323a-4d1d-90db-79bdbc2c6c68","Type":"ContainerDied","Data":"78a0b6d27239fcf8d5a8b0a897d462ceb5e501b5b877af016da19766336a01d5"}
Nov 02 03:33:24.728350 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:24.728321    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f1a692-zdkxn"
Nov 02 03:33:24.802345 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:24.802252    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9049d7e3-323a-4d1d-90db-79bdbc2c6c68-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9049d7e3-323a-4d1d-90db-79bdbc2c6c68-maas-api-xks-ca\") pod \"9049d7e3-323a-4d1d-90db-79bdbc2c6c68\" (UID: \"9049d7e3-323a-4d1d-90db-79bdbc2c6c68\") "
Nov 02 03:33:24.802345 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:24.802292    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9049d7e3-323a-4d1d-90db-79bdbc2c6c68-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9049d7e3-323a-4d1d-90db-79bdbc2c6c68-maas-api-openshift-ca\") pod \"9049d7e3-323a-4d1d-90db-79bdbc2c6c68\" (UID: \"9049d7e3-323a-4d1d-90db-79bdbc2c6c68\") "
Nov 02 03:33:24.802596 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:24.802431    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9049d7e3-323a-4d1d-90db-79bdbc2c6c68-kube-api-access-hmq9d\" (UniqueName: \"kubernetes.io/projected/9049d7e3-323a-4d1d-90db-79bdbc2c6c68-kube-api-access-hmq9d\") pod \"9049d7e3-323a-4d1d-90db-79bdbc2c6c68\" (UID: \"9049d7e3-323a-4d1d-90db-79bdbc2c6c68\") "
Nov 02 03:33:24.802685 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:24.802655    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9049d7e3-323a-4d1d-90db-79bdbc2c6c68-maas-api-openshift-ca" pod "9049d7e3-323a-4d1d-90db-79bdbc2c6c68" (UID: "9049d7e3-323a-4d1d-90db-79bdbc2c6c68"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:33:24.803156 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:24.803131    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9049d7e3-323a-4d1d-90db-79bdbc2c6c68-maas-api-openshift-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:33:24.805088 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:24.805059    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9049d7e3-323a-4d1d-90db-79bdbc2c6c68-kube-api-access-hmq9d" pod "9049d7e3-323a-4d1d-90db-79bdbc2c6c68" (UID: "9049d7e3-323a-4d1d-90db-79bdbc2c6c68"). InnerVolumeSpecName "kube-api-access-hmq9d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:33:24.805176 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:24.805083    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9049d7e3-323a-4d1d-90db-79bdbc2c6c68-maas-api-xks-ca" pod "9049d7e3-323a-4d1d-90db-79bdbc2c6c68" (UID: "9049d7e3-323a-4d1d-90db-79bdbc2c6c68"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 03:33:24.904761 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:24.904724    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmq9d\" (UniqueName: \"kubernetes.io/projected/9049d7e3-323a-4d1d-90db-79bdbc2c6c68-kube-api-access-hmq9d\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:33:24.904761 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:24.904755    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9049d7e3-323a-4d1d-90db-79bdbc2c6c68-maas-api-xks-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:33:25.544395 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:25.544363    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f1a692-zdkxn"
Nov 02 03:33:25.544596 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:25.544391    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f1a692-zdkxn" event={"ID":"9049d7e3-323a-4d1d-90db-79bdbc2c6c68","Type":"ContainerDied","Data":"b0781a3589fb03167163739c76a3f3580c95df35278ab96fa22eac5f105f9d9c"}
Nov 02 03:33:25.544596 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:25.544448    4496 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b0781a3589fb03167163739c76a3f3580c95df35278ab96fa22eac5f105f9d9c"
Nov 02 03:33:33.527184 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:33:33.527143    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 03:33:41.307226 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:41.307193    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-018375-6495ddb8c5-r44mj"]
Nov 02 03:33:41.307911 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:41.307896    4496 cpu_manager.go:414] "containerMap: removing container" podUID="9049d7e3-323a-4d1d-90db-79bdbc2c6c68" containerName="revoke-keys"
Nov 02 03:33:41.307960 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:41.307913    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9049d7e3-323a-4d1d-90db-79bdbc2c6c68" containerName="revoke-keys"
Nov 02 03:33:41.308048 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:41.308035    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="9049d7e3-323a-4d1d-90db-79bdbc2c6c68" containerName="revoke-keys"
Nov 02 03:33:41.311538 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:41.311521    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-018375-6495ddb8c5-r44mj"
Nov 02 03:33:41.367048 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:41.366980    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-018375-6495ddb8c5-r44mj"]
Nov 02 03:33:41.413007 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:41.412977    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-018375\""
Nov 02 03:33:41.413183 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:41.413077    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-018375-dockercfg-nslrm\""
Nov 02 03:33:41.413511 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:41.413488    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdbc2cc9-af01-41e7-b360-4ab8b73a113c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-018375-6495ddb8c5-r44mj\" (UID: \"cdbc2cc9-af01-41e7-b360-4ab8b73a113c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-018375-6495ddb8c5-r44mj"
Nov 02 03:33:41.413662 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:41.413649    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nqtt\" (UniqueName: \"kubernetes.io/projected/cdbc2cc9-af01-41e7-b360-4ab8b73a113c-kube-api-access-9nqtt\") pod \"payload-pre-processing-e2e-ipp-b-018375-6495ddb8c5-r44mj\" (UID: \"cdbc2cc9-af01-41e7-b360-4ab8b73a113c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-018375-6495ddb8c5-r44mj"
Nov 02 03:33:41.515540 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:41.515509    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nqtt\" (UniqueName: \"kubernetes.io/projected/cdbc2cc9-af01-41e7-b360-4ab8b73a113c-kube-api-access-9nqtt\") pod \"payload-pre-processing-e2e-ipp-b-018375-6495ddb8c5-r44mj\" (UID: \"cdbc2cc9-af01-41e7-b360-4ab8b73a113c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-018375-6495ddb8c5-r44mj"
Nov 02 03:33:41.515736 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:41.515650    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdbc2cc9-af01-41e7-b360-4ab8b73a113c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-018375-6495ddb8c5-r44mj\" (UID: \"cdbc2cc9-af01-41e7-b360-4ab8b73a113c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-018375-6495ddb8c5-r44mj"
Nov 02 03:33:41.516367 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:41.516347    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdbc2cc9-af01-41e7-b360-4ab8b73a113c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-018375-6495ddb8c5-r44mj\" (UID: \"cdbc2cc9-af01-41e7-b360-4ab8b73a113c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-018375-6495ddb8c5-r44mj"
Nov 02 03:33:41.529510 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:41.529483    4496 scope.go:122] "RemoveContainer" containerID="4af1dd372de685837c767002e08d69eea3521d167d4f7f2768d7a62ea6e121f0"
Nov 02 03:33:41.539725 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:41.539703    4496 scope.go:122] "RemoveContainer" containerID="0e3609de0d6b09ed48163dde2fcb03fdb6bcc2d7bf1a3debeb6a4dcfabb00caa"
Nov 02 03:33:41.576667 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:41.576640    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nqtt\" (UniqueName: \"kubernetes.io/projected/cdbc2cc9-af01-41e7-b360-4ab8b73a113c-kube-api-access-9nqtt\") pod \"payload-pre-processing-e2e-ipp-b-018375-6495ddb8c5-r44mj\" (UID: \"cdbc2cc9-af01-41e7-b360-4ab8b73a113c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-018375-6495ddb8c5-r44mj"
Nov 02 03:33:41.626044 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:41.625983    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-018375-6495ddb8c5-r44mj"
Nov 02 03:33:42.010731 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:42.010702    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-018375-6495ddb8c5-r44mj"]
Nov 02 03:33:42.011341 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:33:42.011318    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdbc2cc9_af01_41e7_b360_4ab8b73a113c.slice/crio-299ace3957518c11988b52e3eaef9bffabfe01c223f67df3ec613a68364fd680 WatchSource:0}: Error finding container 299ace3957518c11988b52e3eaef9bffabfe01c223f67df3ec613a68364fd680: Status 404 returned error can't find the container with id 299ace3957518c11988b52e3eaef9bffabfe01c223f67df3ec613a68364fd680
Nov 02 03:33:42.722788 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:42.722748    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-018375-6495ddb8c5-r44mj" event={"ID":"cdbc2cc9-af01-41e7-b360-4ab8b73a113c","Type":"ContainerStarted","Data":"e2fbaed9c5a3b19b864f4625edcf13d59a9ab2849c33476ffac18f2e6ed9c730"}
Nov 02 03:33:42.723218 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:42.722809    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-018375-6495ddb8c5-r44mj" event={"ID":"cdbc2cc9-af01-41e7-b360-4ab8b73a113c","Type":"ContainerStarted","Data":"299ace3957518c11988b52e3eaef9bffabfe01c223f67df3ec613a68364fd680"}
Nov 02 03:33:42.723218 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:42.722845    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-018375-6495ddb8c5-r44mj"
Nov 02 03:33:42.838556 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:42.838501    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-018375-6495ddb8c5-r44mj" podStartSLOduration=1.838484488 podStartE2EDuration="1.838484488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:33:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:33:42.837593565 +0000 UTC m=+2763.416200936" watchObservedRunningTime="2026-11-02 03:33:42.838484488 +0000 UTC m=+2763.417091872"
Nov 02 03:33:53.731472 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:33:53.731442    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-018375-6495ddb8c5-r44mj"
Nov 02 03:34:03.526253 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:34:03.526215    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 03:34:05.463327 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:05.463291    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-0809c8-64cf8df667-gp4jm"]
Nov 02 03:34:08.611090 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:08.611051    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-045653-6b5cfcf4d-d85kt"]
Nov 02 03:34:08.611540 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:08.611258    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-045653-6b5cfcf4d-d85kt" podUID="0ccc5f9c-bfeb-4076-ae4f-741cb1fbe76b" containerName="payload-processing" containerID="cri-o://aac907948cbf751d304b8be7b502ff49f94fe9c1c22fa067ca2255155dbb3efa" gracePeriod=30
Nov 02 03:34:08.666121 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:08.666092    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-045653-7c865bfc67-tphl9"]
Nov 02 03:34:08.666345 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:08.666323    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-045653-7c865bfc67-tphl9" podUID="61f56105-4f55-4dd8-96bc-53c8ab4e4964" containerName="payload-pre-processing" containerID="cri-o://b4bb31b576d1b045df77ff6b2fed580126723437171ab960c0e123de78568ebb" gracePeriod=30
Nov 02 03:34:08.975881 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:08.975702    4496 generic.go:356] "Generic (PLEG): container finished" podID="0ccc5f9c-bfeb-4076-ae4f-741cb1fbe76b" containerID="aac907948cbf751d304b8be7b502ff49f94fe9c1c22fa067ca2255155dbb3efa" exitCode=0
Nov 02 03:34:08.975881 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:08.975785    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-045653-6b5cfcf4d-d85kt" event={"ID":"0ccc5f9c-bfeb-4076-ae4f-741cb1fbe76b","Type":"ContainerDied","Data":"aac907948cbf751d304b8be7b502ff49f94fe9c1c22fa067ca2255155dbb3efa"}
Nov 02 03:34:08.975881 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:08.975855    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-045653-6b5cfcf4d-d85kt" event={"ID":"0ccc5f9c-bfeb-4076-ae4f-741cb1fbe76b","Type":"ContainerDied","Data":"a599205dcbf7ba68c2d06d9fe0edcaa57581bfbab3bb276c62f87d8ddab9932b"}
Nov 02 03:34:08.975881 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:08.975881    4496 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a599205dcbf7ba68c2d06d9fe0edcaa57581bfbab3bb276c62f87d8ddab9932b"
Nov 02 03:34:08.978123 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:08.978101    4496 generic.go:356] "Generic (PLEG): container finished" podID="61f56105-4f55-4dd8-96bc-53c8ab4e4964" containerID="b4bb31b576d1b045df77ff6b2fed580126723437171ab960c0e123de78568ebb" exitCode=0
Nov 02 03:34:08.978242 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:08.978183    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-045653-7c865bfc67-tphl9" event={"ID":"61f56105-4f55-4dd8-96bc-53c8ab4e4964","Type":"ContainerDied","Data":"b4bb31b576d1b045df77ff6b2fed580126723437171ab960c0e123de78568ebb"}
Nov 02 03:34:08.979766 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:08.979751    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-045653-6b5cfcf4d-d85kt"
Nov 02 03:34:09.005102 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:09.005082    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-045653-7c865bfc67-tphl9"
Nov 02 03:34:09.068403 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:09.068373    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61f56105-4f55-4dd8-96bc-53c8ab4e4964-kube-api-access-fmrfj\" (UniqueName: \"kubernetes.io/projected/61f56105-4f55-4dd8-96bc-53c8ab4e4964-kube-api-access-fmrfj\") pod \"61f56105-4f55-4dd8-96bc-53c8ab4e4964\" (UID: \"61f56105-4f55-4dd8-96bc-53c8ab4e4964\") "
Nov 02 03:34:09.068571 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:09.068426    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ccc5f9c-bfeb-4076-ae4f-741cb1fbe76b-kube-api-access-q6l9x\" (UniqueName: \"kubernetes.io/projected/0ccc5f9c-bfeb-4076-ae4f-741cb1fbe76b-kube-api-access-q6l9x\") pod \"0ccc5f9c-bfeb-4076-ae4f-741cb1fbe76b\" (UID: \"0ccc5f9c-bfeb-4076-ae4f-741cb1fbe76b\") "
Nov 02 03:34:09.068641 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:09.068567    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ccc5f9c-bfeb-4076-ae4f-741cb1fbe76b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ccc5f9c-bfeb-4076-ae4f-741cb1fbe76b-plugins-config-volume\") pod \"0ccc5f9c-bfeb-4076-ae4f-741cb1fbe76b\" (UID: \"0ccc5f9c-bfeb-4076-ae4f-741cb1fbe76b\") "
Nov 02 03:34:09.068698 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:09.068648    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61f56105-4f55-4dd8-96bc-53c8ab4e4964-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61f56105-4f55-4dd8-96bc-53c8ab4e4964-plugins-config-volume\") pod \"61f56105-4f55-4dd8-96bc-53c8ab4e4964\" (UID: \"61f56105-4f55-4dd8-96bc-53c8ab4e4964\") "
Nov 02 03:34:09.069051 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:09.069002    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ccc5f9c-bfeb-4076-ae4f-741cb1fbe76b-plugins-config-volume" pod "0ccc5f9c-bfeb-4076-ae4f-741cb1fbe76b" (UID: "0ccc5f9c-bfeb-4076-ae4f-741cb1fbe76b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:34:09.069051 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:09.069008    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61f56105-4f55-4dd8-96bc-53c8ab4e4964-plugins-config-volume" pod "61f56105-4f55-4dd8-96bc-53c8ab4e4964" (UID: "61f56105-4f55-4dd8-96bc-53c8ab4e4964"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:34:09.069373 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:09.069349    4496 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61f56105-4f55-4dd8-96bc-53c8ab4e4964-plugins-config-volume\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:34:09.069446 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:09.069379    4496 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ccc5f9c-bfeb-4076-ae4f-741cb1fbe76b-plugins-config-volume\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:34:09.070720 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:09.070693    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ccc5f9c-bfeb-4076-ae4f-741cb1fbe76b-kube-api-access-q6l9x" pod "0ccc5f9c-bfeb-4076-ae4f-741cb1fbe76b" (UID: "0ccc5f9c-bfeb-4076-ae4f-741cb1fbe76b"). InnerVolumeSpecName "kube-api-access-q6l9x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:34:09.070827 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:09.070730    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61f56105-4f55-4dd8-96bc-53c8ab4e4964-kube-api-access-fmrfj" pod "61f56105-4f55-4dd8-96bc-53c8ab4e4964" (UID: "61f56105-4f55-4dd8-96bc-53c8ab4e4964"). InnerVolumeSpecName "kube-api-access-fmrfj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:34:09.177107 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:09.176916    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmrfj\" (UniqueName: \"kubernetes.io/projected/61f56105-4f55-4dd8-96bc-53c8ab4e4964-kube-api-access-fmrfj\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:34:09.177107 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:09.176997    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6l9x\" (UniqueName: \"kubernetes.io/projected/0ccc5f9c-bfeb-4076-ae4f-741cb1fbe76b-kube-api-access-q6l9x\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:34:09.988515 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:09.988481    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-045653-7c865bfc67-tphl9" event={"ID":"61f56105-4f55-4dd8-96bc-53c8ab4e4964","Type":"ContainerDied","Data":"5bfb43cade3069264d1f66b51714c17f0fd970c3c1c9951d53bbd1499e715b8a"}
Nov 02 03:34:09.988515 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:09.988504    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-045653-7c865bfc67-tphl9"
Nov 02 03:34:09.989088 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:09.988575    4496 scope.go:122] "RemoveContainer" containerID="b4bb31b576d1b045df77ff6b2fed580126723437171ab960c0e123de78568ebb"
Nov 02 03:34:09.989088 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:09.988717    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-045653-6b5cfcf4d-d85kt"
Nov 02 03:34:10.157318 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:10.157275    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-045653-7c865bfc67-tphl9"]
Nov 02 03:34:10.163445 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:10.163416    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-045653-7c865bfc67-tphl9"]
Nov 02 03:34:10.286896 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:10.286804    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-045653-6b5cfcf4d-d85kt"]
Nov 02 03:34:10.291926 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:10.291891    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-045653-6b5cfcf4d-d85kt"]
Nov 02 03:34:10.441365 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:10.441326    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-045653-fvbq7"]
Nov 02 03:34:10.445660 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:10.445629    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-045653-fvbq7"]
Nov 02 03:34:10.463617 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:10.463580    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-0809c8-64cf8df667-gp4jm" podUID="763d2941-009d-48c5-9f8b-99e5fdbd3949" containerName="payload-processing" containerID="cri-o://49d32d0dcd573e1c9b771de7de0e8019a0301a44a6e3855eb874a7769f71f04f" gracePeriod=25
Nov 02 03:34:10.755128 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:10.755097    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-0809c8-64cf8df667-gp4jm"
Nov 02 03:34:10.898626 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:10.898549    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/763d2941-009d-48c5-9f8b-99e5fdbd3949-kube-api-access-7mj8t\" (UniqueName: \"kubernetes.io/projected/763d2941-009d-48c5-9f8b-99e5fdbd3949-kube-api-access-7mj8t\") pod \"763d2941-009d-48c5-9f8b-99e5fdbd3949\" (UID: \"763d2941-009d-48c5-9f8b-99e5fdbd3949\") "
Nov 02 03:34:10.898776 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:10.898635    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/763d2941-009d-48c5-9f8b-99e5fdbd3949-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/763d2941-009d-48c5-9f8b-99e5fdbd3949-plugins-config-volume\") pod \"763d2941-009d-48c5-9f8b-99e5fdbd3949\" (UID: \"763d2941-009d-48c5-9f8b-99e5fdbd3949\") "
Nov 02 03:34:10.899084 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:10.899056    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/763d2941-009d-48c5-9f8b-99e5fdbd3949-plugins-config-volume" pod "763d2941-009d-48c5-9f8b-99e5fdbd3949" (UID: "763d2941-009d-48c5-9f8b-99e5fdbd3949"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:34:10.900789 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:10.900767    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/763d2941-009d-48c5-9f8b-99e5fdbd3949-kube-api-access-7mj8t" pod "763d2941-009d-48c5-9f8b-99e5fdbd3949" (UID: "763d2941-009d-48c5-9f8b-99e5fdbd3949"). InnerVolumeSpecName "kube-api-access-7mj8t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:34:10.997554 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:10.997517    4496 generic.go:356] "Generic (PLEG): container finished" podID="763d2941-009d-48c5-9f8b-99e5fdbd3949" containerID="49d32d0dcd573e1c9b771de7de0e8019a0301a44a6e3855eb874a7769f71f04f" exitCode=0
Nov 02 03:34:10.998111 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:10.997583    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-0809c8-64cf8df667-gp4jm"
Nov 02 03:34:10.998111 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:10.997601    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-0809c8-64cf8df667-gp4jm" event={"ID":"763d2941-009d-48c5-9f8b-99e5fdbd3949","Type":"ContainerDied","Data":"49d32d0dcd573e1c9b771de7de0e8019a0301a44a6e3855eb874a7769f71f04f"}
Nov 02 03:34:10.998111 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:10.997658    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-0809c8-64cf8df667-gp4jm" event={"ID":"763d2941-009d-48c5-9f8b-99e5fdbd3949","Type":"ContainerDied","Data":"033a3e9b6ccdd1b2e319f166b8d76c77ae34b2864c587251be771084e216bc48"}
Nov 02 03:34:10.998111 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:10.997689    4496 scope.go:122] "RemoveContainer" containerID="49d32d0dcd573e1c9b771de7de0e8019a0301a44a6e3855eb874a7769f71f04f"
Nov 02 03:34:11.000241 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:11.000214    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mj8t\" (UniqueName: \"kubernetes.io/projected/763d2941-009d-48c5-9f8b-99e5fdbd3949-kube-api-access-7mj8t\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:34:11.000377 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:11.000244    4496 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/763d2941-009d-48c5-9f8b-99e5fdbd3949-plugins-config-volume\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:34:11.008528 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:11.008512    4496 scope.go:122] "RemoveContainer" containerID="49d32d0dcd573e1c9b771de7de0e8019a0301a44a6e3855eb874a7769f71f04f"
Nov 02 03:34:11.008776 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:34:11.008753    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49d32d0dcd573e1c9b771de7de0e8019a0301a44a6e3855eb874a7769f71f04f\": container with ID starting with 49d32d0dcd573e1c9b771de7de0e8019a0301a44a6e3855eb874a7769f71f04f not found: ID does not exist" containerID="49d32d0dcd573e1c9b771de7de0e8019a0301a44a6e3855eb874a7769f71f04f"
Nov 02 03:34:11.008841 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:11.008787    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49d32d0dcd573e1c9b771de7de0e8019a0301a44a6e3855eb874a7769f71f04f"} err="failed to get container status \"49d32d0dcd573e1c9b771de7de0e8019a0301a44a6e3855eb874a7769f71f04f\": rpc error: code = NotFound desc = could not find container \"49d32d0dcd573e1c9b771de7de0e8019a0301a44a6e3855eb874a7769f71f04f\": container with ID starting with 49d32d0dcd573e1c9b771de7de0e8019a0301a44a6e3855eb874a7769f71f04f not found: ID does not exist"
Nov 02 03:34:11.168301 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:11.168211    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-0809c8-64cf8df667-gp4jm"]
Nov 02 03:34:11.173598 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:11.173569    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-0809c8-64cf8df667-gp4jm"]
Nov 02 03:34:11.574846 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:11.574807    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ccc5f9c-bfeb-4076-ae4f-741cb1fbe76b" path="/var/lib/kubelet/pods/0ccc5f9c-bfeb-4076-ae4f-741cb1fbe76b/volumes"
Nov 02 03:34:11.575510 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:11.575484    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61f56105-4f55-4dd8-96bc-53c8ab4e4964" path="/var/lib/kubelet/pods/61f56105-4f55-4dd8-96bc-53c8ab4e4964/volumes"
Nov 02 03:34:11.575996 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:11.575963    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="763d2941-009d-48c5-9f8b-99e5fdbd3949" path="/var/lib/kubelet/pods/763d2941-009d-48c5-9f8b-99e5fdbd3949/volumes"
Nov 02 03:34:11.576462 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:11.576441    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5b8152f-c332-42be-a2de-1fb845cdc2bf" path="/var/lib/kubelet/pods/e5b8152f-c332-42be-a2de-1fb845cdc2bf/volumes"
Nov 02 03:34:12.263346 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:12.263307    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-2ktj7"]
Nov 02 03:34:12.263809 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:12.263536    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-2ktj7" podUID="a8d9e090-928d-489e-9eb4-82ad15bb6aa5" containerName="istio-proxy" containerID="cri-o://e1e157fedc734b8e972890434e5dbf002a597948de2ffecde03d9b19eb081831" gracePeriod=30
Nov 02 03:34:12.315476 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:12.315441    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-zf5k4"]
Nov 02 03:34:12.316065 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:12.315994    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-zf5k4" podUID="d695fa41-7976-4a64-b9e2-4642ab0ed15f" containerName="istio-proxy" containerID="cri-o://efe2ac77c34e85a4a29e78deb7391c2336e8ed6476be5ba780462d7ee01b78cb" gracePeriod=30
Nov 02 03:34:14.016624 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:14.016582    4496 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-zf5k4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 03:34:14.017063 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:14.016644    4496 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-zf5k4" podUID="d695fa41-7976-4a64-b9e2-4642ab0ed15f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 03:34:15.584877 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:15.584840    4496 patch_prober.go:29] interesting pod/e2e-worker-w3-858236-openshift-default-74fcd64bf5-2ktj7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 03:34:15.585289 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:15.584893    4496 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-2ktj7" podUID="a8d9e090-928d-489e-9eb4-82ad15bb6aa5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 03:34:17.603504 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:17.603460    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-2ktj7"
Nov 02 03:34:17.649673 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:17.649648    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-zf5k4"
Nov 02 03:34:17.692578 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:17.692550    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8d9e090-928d-489e-9eb4-82ad15bb6aa5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8d9e090-928d-489e-9eb4-82ad15bb6aa5-workload-socket\") pod \"a8d9e090-928d-489e-9eb4-82ad15bb6aa5\" (UID: \"a8d9e090-928d-489e-9eb4-82ad15bb6aa5\") "
Nov 02 03:34:17.692769 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:17.692587    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8d9e090-928d-489e-9eb4-82ad15bb6aa5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8d9e090-928d-489e-9eb4-82ad15bb6aa5-istio-envoy\") pod \"a8d9e090-928d-489e-9eb4-82ad15bb6aa5\" (UID: \"a8d9e090-928d-489e-9eb4-82ad15bb6aa5\") "
Nov 02 03:34:17.692769 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:17.692609    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a8d9e090-928d-489e-9eb4-82ad15bb6aa5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8d9e090-928d-489e-9eb4-82ad15bb6aa5-istio-podinfo\") pod \"a8d9e090-928d-489e-9eb4-82ad15bb6aa5\" (UID: \"a8d9e090-928d-489e-9eb4-82ad15bb6aa5\") "
Nov 02 03:34:17.692769 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:17.692693    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8d9e090-928d-489e-9eb4-82ad15bb6aa5-kube-api-access-746t5\" (UniqueName: \"kubernetes.io/projected/a8d9e090-928d-489e-9eb4-82ad15bb6aa5-kube-api-access-746t5\") pod \"a8d9e090-928d-489e-9eb4-82ad15bb6aa5\" (UID: \"a8d9e090-928d-489e-9eb4-82ad15bb6aa5\") "
Nov 02 03:34:17.692962 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:17.692774    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8d9e090-928d-489e-9eb4-82ad15bb6aa5-istio-token\" (UniqueName: \"kubernetes.io/projected/a8d9e090-928d-489e-9eb4-82ad15bb6aa5-istio-token\") pod \"a8d9e090-928d-489e-9eb4-82ad15bb6aa5\" (UID: \"a8d9e090-928d-489e-9eb4-82ad15bb6aa5\") "
Nov 02 03:34:17.692962 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:17.692897    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8d9e090-928d-489e-9eb4-82ad15bb6aa5-workload-socket" pod "a8d9e090-928d-489e-9eb4-82ad15bb6aa5" (UID: "a8d9e090-928d-489e-9eb4-82ad15bb6aa5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:34:17.693109 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:17.693062    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8d9e090-928d-489e-9eb4-82ad15bb6aa5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8d9e090-928d-489e-9eb4-82ad15bb6aa5-credential-socket\") pod \"a8d9e090-928d-489e-9eb4-82ad15bb6aa5\" (UID: \"a8d9e090-928d-489e-9eb4-82ad15bb6aa5\") "
Nov 02 03:34:17.693170 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:17.693128    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8d9e090-928d-489e-9eb4-82ad15bb6aa5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8d9e090-928d-489e-9eb4-82ad15bb6aa5-workload-certs\") pod \"a8d9e090-928d-489e-9eb4-82ad15bb6aa5\" (UID: \"a8d9e090-928d-489e-9eb4-82ad15bb6aa5\") "
Nov 02 03:34:17.693225 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:17.693209    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8d9e090-928d-489e-9eb4-82ad15bb6aa5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8d9e090-928d-489e-9eb4-82ad15bb6aa5-istio-data\") pod \"a8d9e090-928d-489e-9eb4-82ad15bb6aa5\" (UID: \"a8d9e090-928d-489e-9eb4-82ad15bb6aa5\") "
Nov 02 03:34:17.693280 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:17.693260    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8d9e090-928d-489e-9eb4-82ad15bb6aa5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8d9e090-928d-489e-9eb4-82ad15bb6aa5-istiod-ca-cert\") pod \"a8d9e090-928d-489e-9eb4-82ad15bb6aa5\" (UID: \"a8d9e090-928d-489e-9eb4-82ad15bb6aa5\") "
Nov 02 03:34:17.693367 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:17.693346    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8d9e090-928d-489e-9eb4-82ad15bb6aa5-credential-socket" pod "a8d9e090-928d-489e-9eb4-82ad15bb6aa5" (UID: "a8d9e090-928d-489e-9eb4-82ad15bb6aa5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:34:17.693551 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:17.693520    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8d9e090-928d-489e-9eb4-82ad15bb6aa5-workload-certs" pod "a8d9e090-928d-489e-9eb4-82ad15bb6aa5" (UID: "a8d9e090-928d-489e-9eb4-82ad15bb6aa5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:34:17.693942 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:17.693921    4496 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8d9e090-928d-489e-9eb4-82ad15bb6aa5-workload-certs\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:34:17.694048 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:17.693948    4496 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8d9e090-928d-489e-9eb4-82ad15bb6aa5-workload-socket\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:34:17.694048 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:17.693963    4496 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8d9e090-928d-489e-9eb4-82ad15bb6aa5-credential-socket\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:34:17.694394 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:17.694344    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8d9e090-928d-489e-9eb4-82ad15bb6aa5-istio-data" pod "a8d9e090-928d-489e-9eb4-82ad15bb6aa5" (UID: "a8d9e090-928d-489e-9eb4-82ad15bb6aa5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:34:17.694606 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:17.694537    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8d9e090-928d-489e-9eb4-82ad15bb6aa5-istiod-ca-cert" pod "a8d9e090-928d-489e-9eb4-82ad15bb6aa5" (UID: "a8d9e090-928d-489e-9eb4-82ad15bb6aa5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:34:17.695295 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:17.695255    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8d9e090-928d-489e-9eb4-82ad15bb6aa5-istio-envoy" pod "a8d9e090-928d-489e-9eb4-82ad15bb6aa5" (UID: "a8d9e090-928d-489e-9eb4-82ad15bb6aa5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:34:17.695408 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:17.695361    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a8d9e090-928d-489e-9eb4-82ad15bb6aa5-istio-podinfo" pod "a8d9e090-928d-489e-9eb4-82ad15bb6aa5" (UID: "a8d9e090-928d-489e-9eb4-82ad15bb6aa5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 03:34:17.695408 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:17.695389    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8d9e090-928d-489e-9eb4-82ad15bb6aa5-kube-api-access-746t5" pod "a8d9e090-928d-489e-9eb4-82ad15bb6aa5" (UID: "a8d9e090-928d-489e-9eb4-82ad15bb6aa5"). InnerVolumeSpecName "kube-api-access-746t5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:34:17.696155 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:17.696135    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8d9e090-928d-489e-9eb4-82ad15bb6aa5-istio-token" pod "a8d9e090-928d-489e-9eb4-82ad15bb6aa5" (UID: "a8d9e090-928d-489e-9eb4-82ad15bb6aa5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:34:17.794616 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:17.794580    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d695fa41-7976-4a64-b9e2-4642ab0ed15f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d695fa41-7976-4a64-b9e2-4642ab0ed15f-istio-envoy\") pod \"d695fa41-7976-4a64-b9e2-4642ab0ed15f\" (UID: \"d695fa41-7976-4a64-b9e2-4642ab0ed15f\") "
Nov 02 03:34:17.794792 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:17.794653    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d695fa41-7976-4a64-b9e2-4642ab0ed15f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d695fa41-7976-4a64-b9e2-4642ab0ed15f-istio-data\") pod \"d695fa41-7976-4a64-b9e2-4642ab0ed15f\" (UID: \"d695fa41-7976-4a64-b9e2-4642ab0ed15f\") "
Nov 02 03:34:17.794792 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:17.794724    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d695fa41-7976-4a64-b9e2-4642ab0ed15f-istio-token\" (UniqueName: \"kubernetes.io/projected/d695fa41-7976-4a64-b9e2-4642ab0ed15f-istio-token\") pod \"d695fa41-7976-4a64-b9e2-4642ab0ed15f\" (UID: \"d695fa41-7976-4a64-b9e2-4642ab0ed15f\") "
Nov 02 03:34:17.794908 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:17.794806    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d695fa41-7976-4a64-b9e2-4642ab0ed15f-kube-api-access-2knn7\" (UniqueName: \"kubernetes.io/projected/d695fa41-7976-4a64-b9e2-4642ab0ed15f-kube-api-access-2knn7\") pod \"d695fa41-7976-4a64-b9e2-4642ab0ed15f\" (UID: \"d695fa41-7976-4a64-b9e2-4642ab0ed15f\") "
Nov 02 03:34:17.794908 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:17.794875    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d695fa41-7976-4a64-b9e2-4642ab0ed15f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d695fa41-7976-4a64-b9e2-4642ab0ed15f-istiod-ca-cert\") pod \"d695fa41-7976-4a64-b9e2-4642ab0ed15f\" (UID: \"d695fa41-7976-4a64-b9e2-4642ab0ed15f\") "
Nov 02 03:34:17.795038 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:17.794971    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d695fa41-7976-4a64-b9e2-4642ab0ed15f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d695fa41-7976-4a64-b9e2-4642ab0ed15f-credential-socket\") pod \"d695fa41-7976-4a64-b9e2-4642ab0ed15f\" (UID: \"d695fa41-7976-4a64-b9e2-4642ab0ed15f\") "
Nov 02 03:34:17.795106 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:17.795006    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d695fa41-7976-4a64-b9e2-4642ab0ed15f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d695fa41-7976-4a64-b9e2-4642ab0ed15f-workload-socket\") pod \"d695fa41-7976-4a64-b9e2-4642ab0ed15f\" (UID: \"d695fa41-7976-4a64-b9e2-4642ab0ed15f\") "
Nov 02 03:34:17.795106 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:17.795056    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d695fa41-7976-4a64-b9e2-4642ab0ed15f-istio-data" pod "d695fa41-7976-4a64-b9e2-4642ab0ed15f" (UID: "d695fa41-7976-4a64-b9e2-4642ab0ed15f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:34:17.795226 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:17.795106    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d695fa41-7976-4a64-b9e2-4642ab0ed15f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d695fa41-7976-4a64-b9e2-4642ab0ed15f-istio-podinfo\") pod \"d695fa41-7976-4a64-b9e2-4642ab0ed15f\" (UID: \"d695fa41-7976-4a64-b9e2-4642ab0ed15f\") "
Nov 02 03:34:17.795226 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:17.795182    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d695fa41-7976-4a64-b9e2-4642ab0ed15f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d695fa41-7976-4a64-b9e2-4642ab0ed15f-workload-certs\") pod \"d695fa41-7976-4a64-b9e2-4642ab0ed15f\" (UID: \"d695fa41-7976-4a64-b9e2-4642ab0ed15f\") "
Nov 02 03:34:17.795417 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:17.795391    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d695fa41-7976-4a64-b9e2-4642ab0ed15f-istiod-ca-cert" pod "d695fa41-7976-4a64-b9e2-4642ab0ed15f" (UID: "d695fa41-7976-4a64-b9e2-4642ab0ed15f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:34:17.795670 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:17.795641    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d695fa41-7976-4a64-b9e2-4642ab0ed15f-credential-socket" pod "d695fa41-7976-4a64-b9e2-4642ab0ed15f" (UID: "d695fa41-7976-4a64-b9e2-4642ab0ed15f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:34:17.795760 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:17.795675    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d695fa41-7976-4a64-b9e2-4642ab0ed15f-workload-certs" pod "d695fa41-7976-4a64-b9e2-4642ab0ed15f" (UID: "d695fa41-7976-4a64-b9e2-4642ab0ed15f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:34:17.795873 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:17.795846    4496 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8d9e090-928d-489e-9eb4-82ad15bb6aa5-istio-data\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:34:17.795873 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:17.795874    4496 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d695fa41-7976-4a64-b9e2-4642ab0ed15f-workload-certs\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:34:17.796114 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:17.795890    4496 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8d9e090-928d-489e-9eb4-82ad15bb6aa5-istiod-ca-cert\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:34:17.796114 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:17.795904    4496 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d695fa41-7976-4a64-b9e2-4642ab0ed15f-istio-data\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:34:17.796114 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:17.795916    4496 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8d9e090-928d-489e-9eb4-82ad15bb6aa5-istio-envoy\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:34:17.796114 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:17.795930    4496 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8d9e090-928d-489e-9eb4-82ad15bb6aa5-istio-podinfo\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:34:17.796114 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:17.795945    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-746t5\" (UniqueName: \"kubernetes.io/projected/a8d9e090-928d-489e-9eb4-82ad15bb6aa5-kube-api-access-746t5\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:34:17.796114 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:17.795961    4496 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d695fa41-7976-4a64-b9e2-4642ab0ed15f-istiod-ca-cert\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:34:17.796114 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:17.795973    4496 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8d9e090-928d-489e-9eb4-82ad15bb6aa5-istio-token\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:34:17.796114 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:17.795987    4496 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d695fa41-7976-4a64-b9e2-4642ab0ed15f-credential-socket\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:34:17.796114 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:17.796094    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d695fa41-7976-4a64-b9e2-4642ab0ed15f-workload-socket" pod "d695fa41-7976-4a64-b9e2-4642ab0ed15f" (UID: "d695fa41-7976-4a64-b9e2-4642ab0ed15f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:34:17.797602 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:17.797579    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d695fa41-7976-4a64-b9e2-4642ab0ed15f-istio-envoy" pod "d695fa41-7976-4a64-b9e2-4642ab0ed15f" (UID: "d695fa41-7976-4a64-b9e2-4642ab0ed15f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:34:17.797758 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:17.797732    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d695fa41-7976-4a64-b9e2-4642ab0ed15f-kube-api-access-2knn7" pod "d695fa41-7976-4a64-b9e2-4642ab0ed15f" (UID: "d695fa41-7976-4a64-b9e2-4642ab0ed15f"). InnerVolumeSpecName "kube-api-access-2knn7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:34:17.797818 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:17.797772    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d695fa41-7976-4a64-b9e2-4642ab0ed15f-istio-token" pod "d695fa41-7976-4a64-b9e2-4642ab0ed15f" (UID: "d695fa41-7976-4a64-b9e2-4642ab0ed15f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:34:17.797913 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:17.797893    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d695fa41-7976-4a64-b9e2-4642ab0ed15f-istio-podinfo" pod "d695fa41-7976-4a64-b9e2-4642ab0ed15f" (UID: "d695fa41-7976-4a64-b9e2-4642ab0ed15f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 03:34:17.896751 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:17.896675    4496 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d695fa41-7976-4a64-b9e2-4642ab0ed15f-workload-socket\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:34:17.896751 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:17.896709    4496 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d695fa41-7976-4a64-b9e2-4642ab0ed15f-istio-podinfo\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:34:17.896751 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:17.896718    4496 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d695fa41-7976-4a64-b9e2-4642ab0ed15f-istio-envoy\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:34:17.896751 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:17.896729    4496 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d695fa41-7976-4a64-b9e2-4642ab0ed15f-istio-token\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:34:17.896751 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:17.896738    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2knn7\" (UniqueName: \"kubernetes.io/projected/d695fa41-7976-4a64-b9e2-4642ab0ed15f-kube-api-access-2knn7\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:34:17.935216 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:17.935183    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9d5625-5hfhp"]
Nov 02 03:34:17.935801 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:17.935787    4496 cpu_manager.go:414] "containerMap: removing container" podUID="a8d9e090-928d-489e-9eb4-82ad15bb6aa5" containerName="istio-proxy"
Nov 02 03:34:17.935857 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:17.935803    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8d9e090-928d-489e-9eb4-82ad15bb6aa5" containerName="istio-proxy"
Nov 02 03:34:17.935857 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:17.935817    4496 cpu_manager.go:414] "containerMap: removing container" podUID="763d2941-009d-48c5-9f8b-99e5fdbd3949" containerName="payload-processing"
Nov 02 03:34:17.935857 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:17.935823    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="763d2941-009d-48c5-9f8b-99e5fdbd3949" containerName="payload-processing"
Nov 02 03:34:17.935857 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:17.935843    4496 cpu_manager.go:414] "containerMap: removing container" podUID="0ccc5f9c-bfeb-4076-ae4f-741cb1fbe76b" containerName="payload-processing"
Nov 02 03:34:17.935857 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:17.935848    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ccc5f9c-bfeb-4076-ae4f-741cb1fbe76b" containerName="payload-processing"
Nov 02 03:34:17.935857 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:17.935856    4496 cpu_manager.go:414] "containerMap: removing container" podUID="d695fa41-7976-4a64-b9e2-4642ab0ed15f" containerName="istio-proxy"
Nov 02 03:34:17.936075 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:17.935862    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d695fa41-7976-4a64-b9e2-4642ab0ed15f" containerName="istio-proxy"
Nov 02 03:34:17.936075 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:17.935880    4496 cpu_manager.go:414] "containerMap: removing container" podUID="61f56105-4f55-4dd8-96bc-53c8ab4e4964" containerName="payload-pre-processing"
Nov 02 03:34:17.936075 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:17.935885    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61f56105-4f55-4dd8-96bc-53c8ab4e4964" containerName="payload-pre-processing"
Nov 02 03:34:17.936075 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:17.935978    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8d9e090-928d-489e-9eb4-82ad15bb6aa5" containerName="istio-proxy"
Nov 02 03:34:17.936075 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:17.935985    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="763d2941-009d-48c5-9f8b-99e5fdbd3949" containerName="payload-processing"
Nov 02 03:34:17.936075 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:17.935995    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ccc5f9c-bfeb-4076-ae4f-741cb1fbe76b" containerName="payload-processing"
Nov 02 03:34:17.936075 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:17.936001    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="d695fa41-7976-4a64-b9e2-4642ab0ed15f" containerName="istio-proxy"
Nov 02 03:34:17.936075 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:17.936010    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="61f56105-4f55-4dd8-96bc-53c8ab4e4964" containerName="payload-pre-processing"
Nov 02 03:34:17.940817 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:17.940799    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9d5625-5hfhp"
Nov 02 03:34:17.996312 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:17.996206    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 03:34:17.996312 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:17.996261    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 03:34:17.996544 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:17.996448    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 03:34:17.996544 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:17.996537    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9z5bk\""
Nov 02 03:34:18.001203 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:18.001179    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9d5625-5hfhp"]
Nov 02 03:34:18.055743 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:18.055709    4496 generic.go:356] "Generic (PLEG): container finished" podID="d695fa41-7976-4a64-b9e2-4642ab0ed15f" containerID="efe2ac77c34e85a4a29e78deb7391c2336e8ed6476be5ba780462d7ee01b78cb" exitCode=0
Nov 02 03:34:18.055963 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:18.055770    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-zf5k4"
Nov 02 03:34:18.055963 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:18.055847    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-zf5k4" event={"ID":"d695fa41-7976-4a64-b9e2-4642ab0ed15f","Type":"ContainerDied","Data":"efe2ac77c34e85a4a29e78deb7391c2336e8ed6476be5ba780462d7ee01b78cb"}
Nov 02 03:34:18.055963 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:18.055906    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-zf5k4" event={"ID":"d695fa41-7976-4a64-b9e2-4642ab0ed15f","Type":"ContainerDied","Data":"010412e342a85f05f004fd724ce4da9b2101998c52abdcbb4c54ab189f77fd35"}
Nov 02 03:34:18.055963 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:18.055939    4496 scope.go:122] "RemoveContainer" containerID="efe2ac77c34e85a4a29e78deb7391c2336e8ed6476be5ba780462d7ee01b78cb"
Nov 02 03:34:18.059421 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:18.059391    4496 generic.go:356] "Generic (PLEG): container finished" podID="a8d9e090-928d-489e-9eb4-82ad15bb6aa5" containerID="e1e157fedc734b8e972890434e5dbf002a597948de2ffecde03d9b19eb081831" exitCode=0
Nov 02 03:34:18.059526 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:18.059458    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-2ktj7"
Nov 02 03:34:18.059631 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:18.059609    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-2ktj7" event={"ID":"a8d9e090-928d-489e-9eb4-82ad15bb6aa5","Type":"ContainerDied","Data":"e1e157fedc734b8e972890434e5dbf002a597948de2ffecde03d9b19eb081831"}
Nov 02 03:34:18.059764 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:18.059740    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-2ktj7" event={"ID":"a8d9e090-928d-489e-9eb4-82ad15bb6aa5","Type":"ContainerDied","Data":"b9cdedd775b1a44c925047fe2e40810d3a8b8e5ce75195c7f59b8e48357593d1"}
Nov 02 03:34:18.070823 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:18.070805    4496 scope.go:122] "RemoveContainer" containerID="efe2ac77c34e85a4a29e78deb7391c2336e8ed6476be5ba780462d7ee01b78cb"
Nov 02 03:34:18.071146 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:34:18.071125    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efe2ac77c34e85a4a29e78deb7391c2336e8ed6476be5ba780462d7ee01b78cb\": container with ID starting with efe2ac77c34e85a4a29e78deb7391c2336e8ed6476be5ba780462d7ee01b78cb not found: ID does not exist" containerID="efe2ac77c34e85a4a29e78deb7391c2336e8ed6476be5ba780462d7ee01b78cb"
Nov 02 03:34:18.071217 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:18.071153    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efe2ac77c34e85a4a29e78deb7391c2336e8ed6476be5ba780462d7ee01b78cb"} err="failed to get container status \"efe2ac77c34e85a4a29e78deb7391c2336e8ed6476be5ba780462d7ee01b78cb\": rpc error: code = NotFound desc = could not find container \"efe2ac77c34e85a4a29e78deb7391c2336e8ed6476be5ba780462d7ee01b78cb\": container with ID starting with efe2ac77c34e85a4a29e78deb7391c2336e8ed6476be5ba780462d7ee01b78cb not found: ID does not exist"
Nov 02 03:34:18.071217 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:18.071171    4496 scope.go:122] "RemoveContainer" containerID="e1e157fedc734b8e972890434e5dbf002a597948de2ffecde03d9b19eb081831"
Nov 02 03:34:18.081075 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:18.081058    4496 scope.go:122] "RemoveContainer" containerID="e1e157fedc734b8e972890434e5dbf002a597948de2ffecde03d9b19eb081831"
Nov 02 03:34:18.081393 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:34:18.081361    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1e157fedc734b8e972890434e5dbf002a597948de2ffecde03d9b19eb081831\": container with ID starting with e1e157fedc734b8e972890434e5dbf002a597948de2ffecde03d9b19eb081831 not found: ID does not exist" containerID="e1e157fedc734b8e972890434e5dbf002a597948de2ffecde03d9b19eb081831"
Nov 02 03:34:18.081446 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:18.081401    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1e157fedc734b8e972890434e5dbf002a597948de2ffecde03d9b19eb081831"} err="failed to get container status \"e1e157fedc734b8e972890434e5dbf002a597948de2ffecde03d9b19eb081831\": rpc error: code = NotFound desc = could not find container \"e1e157fedc734b8e972890434e5dbf002a597948de2ffecde03d9b19eb081831\": container with ID starting with e1e157fedc734b8e972890434e5dbf002a597948de2ffecde03d9b19eb081831 not found: ID does not exist"
Nov 02 03:34:18.099066 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:18.099036    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvttn\" (UniqueName: \"kubernetes.io/projected/2114be44-517b-4282-ab24-db10a832039d-kube-api-access-wvttn\") pod \"maas-api-revoke-keys-e2e-ipp-b-9d5625-5hfhp\" (UID: \"2114be44-517b-4282-ab24-db10a832039d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9d5625-5hfhp"
Nov 02 03:34:18.099195 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:18.099102    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2114be44-517b-4282-ab24-db10a832039d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-9d5625-5hfhp\" (UID: \"2114be44-517b-4282-ab24-db10a832039d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9d5625-5hfhp"
Nov 02 03:34:18.099279 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:18.099254    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2114be44-517b-4282-ab24-db10a832039d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-9d5625-5hfhp\" (UID: \"2114be44-517b-4282-ab24-db10a832039d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9d5625-5hfhp"
Nov 02 03:34:18.200955 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:18.200865    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvttn\" (UniqueName: \"kubernetes.io/projected/2114be44-517b-4282-ab24-db10a832039d-kube-api-access-wvttn\") pod \"maas-api-revoke-keys-e2e-ipp-b-9d5625-5hfhp\" (UID: \"2114be44-517b-4282-ab24-db10a832039d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9d5625-5hfhp"
Nov 02 03:34:18.200955 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:18.200935    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2114be44-517b-4282-ab24-db10a832039d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-9d5625-5hfhp\" (UID: \"2114be44-517b-4282-ab24-db10a832039d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9d5625-5hfhp"
Nov 02 03:34:18.201209 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:18.200984    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2114be44-517b-4282-ab24-db10a832039d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-9d5625-5hfhp\" (UID: \"2114be44-517b-4282-ab24-db10a832039d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9d5625-5hfhp"
Nov 02 03:34:18.201743 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:18.201712    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2114be44-517b-4282-ab24-db10a832039d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-9d5625-5hfhp\" (UID: \"2114be44-517b-4282-ab24-db10a832039d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9d5625-5hfhp"
Nov 02 03:34:18.203442 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:18.203420    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2114be44-517b-4282-ab24-db10a832039d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-9d5625-5hfhp\" (UID: \"2114be44-517b-4282-ab24-db10a832039d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9d5625-5hfhp"
Nov 02 03:34:18.225603 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:18.225566    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-zf5k4"]
Nov 02 03:34:18.228898 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:18.228871    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-zf5k4"]
Nov 02 03:34:18.260508 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:18.260473    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvttn\" (UniqueName: \"kubernetes.io/projected/2114be44-517b-4282-ab24-db10a832039d-kube-api-access-wvttn\") pod \"maas-api-revoke-keys-e2e-ipp-b-9d5625-5hfhp\" (UID: \"2114be44-517b-4282-ab24-db10a832039d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9d5625-5hfhp"
Nov 02 03:34:18.339846 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:18.339815    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-2ktj7"]
Nov 02 03:34:18.343389 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:18.343365    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-2ktj7"]
Nov 02 03:34:18.555245 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:18.555214    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9d5625-5hfhp"
Nov 02 03:34:18.936657 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:18.936620    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9d5625-5hfhp"]
Nov 02 03:34:18.937095 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:34:18.937066    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2114be44_517b_4282_ab24_db10a832039d.slice/crio-4cdebb29db26b4cfdb04ce83a269196000deaf68ab91be2b19b9b1aa47021a64 WatchSource:0}: Error finding container 4cdebb29db26b4cfdb04ce83a269196000deaf68ab91be2b19b9b1aa47021a64: Status 404 returned error can't find the container with id 4cdebb29db26b4cfdb04ce83a269196000deaf68ab91be2b19b9b1aa47021a64
Nov 02 03:34:19.071659 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:19.071619    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9d5625-5hfhp" event={"ID":"2114be44-517b-4282-ab24-db10a832039d","Type":"ContainerStarted","Data":"4cdebb29db26b4cfdb04ce83a269196000deaf68ab91be2b19b9b1aa47021a64"}
Nov 02 03:34:19.574863 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:19.574829    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8d9e090-928d-489e-9eb4-82ad15bb6aa5" path="/var/lib/kubelet/pods/a8d9e090-928d-489e-9eb4-82ad15bb6aa5/volumes"
Nov 02 03:34:19.575363 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:19.575342    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d695fa41-7976-4a64-b9e2-4642ab0ed15f" path="/var/lib/kubelet/pods/d695fa41-7976-4a64-b9e2-4642ab0ed15f/volumes"
Nov 02 03:34:20.080671 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:20.080636    4496 generic.go:356] "Generic (PLEG): container finished" podID="2114be44-517b-4282-ab24-db10a832039d" containerID="75ccc98797c6967cbce0093b4f2e511ce680a7634d982c4a62f4cbdda69d6963" exitCode=0
Nov 02 03:34:20.081096 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:20.080679    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9d5625-5hfhp" event={"ID":"2114be44-517b-4282-ab24-db10a832039d","Type":"ContainerDied","Data":"75ccc98797c6967cbce0093b4f2e511ce680a7634d982c4a62f4cbdda69d6963"}
Nov 02 03:34:21.254788 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:21.254762    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9d5625-5hfhp"
Nov 02 03:34:21.339506 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:21.339435    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2114be44-517b-4282-ab24-db10a832039d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2114be44-517b-4282-ab24-db10a832039d-maas-api-openshift-ca\") pod \"2114be44-517b-4282-ab24-db10a832039d\" (UID: \"2114be44-517b-4282-ab24-db10a832039d\") "
Nov 02 03:34:21.339506 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:21.339496    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2114be44-517b-4282-ab24-db10a832039d-kube-api-access-wvttn\" (UniqueName: \"kubernetes.io/projected/2114be44-517b-4282-ab24-db10a832039d-kube-api-access-wvttn\") pod \"2114be44-517b-4282-ab24-db10a832039d\" (UID: \"2114be44-517b-4282-ab24-db10a832039d\") "
Nov 02 03:34:21.339683 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:21.339572    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2114be44-517b-4282-ab24-db10a832039d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2114be44-517b-4282-ab24-db10a832039d-maas-api-xks-ca\") pod \"2114be44-517b-4282-ab24-db10a832039d\" (UID: \"2114be44-517b-4282-ab24-db10a832039d\") "
Nov 02 03:34:21.339822 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:21.339795    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2114be44-517b-4282-ab24-db10a832039d-maas-api-openshift-ca" pod "2114be44-517b-4282-ab24-db10a832039d" (UID: "2114be44-517b-4282-ab24-db10a832039d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:34:21.339986 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:21.339963    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2114be44-517b-4282-ab24-db10a832039d-maas-api-openshift-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:34:21.341653 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:21.341622    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2114be44-517b-4282-ab24-db10a832039d-maas-api-xks-ca" pod "2114be44-517b-4282-ab24-db10a832039d" (UID: "2114be44-517b-4282-ab24-db10a832039d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 03:34:21.341742 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:21.341684    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2114be44-517b-4282-ab24-db10a832039d-kube-api-access-wvttn" pod "2114be44-517b-4282-ab24-db10a832039d" (UID: "2114be44-517b-4282-ab24-db10a832039d"). InnerVolumeSpecName "kube-api-access-wvttn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:34:21.441387 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:21.441355    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvttn\" (UniqueName: \"kubernetes.io/projected/2114be44-517b-4282-ab24-db10a832039d-kube-api-access-wvttn\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:34:21.441387 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:21.441384    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2114be44-517b-4282-ab24-db10a832039d-maas-api-xks-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:34:22.097461 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:22.097419    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9d5625-5hfhp" event={"ID":"2114be44-517b-4282-ab24-db10a832039d","Type":"ContainerDied","Data":"4cdebb29db26b4cfdb04ce83a269196000deaf68ab91be2b19b9b1aa47021a64"}
Nov 02 03:34:22.097668 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:22.097500    4496 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4cdebb29db26b4cfdb04ce83a269196000deaf68ab91be2b19b9b1aa47021a64"
Nov 02 03:34:22.097668 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:22.097594    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9d5625-5hfhp"
Nov 02 03:34:26.140876 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:26.140835    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9d5625-58c6cfdff9-l8wn2"]
Nov 02 03:34:26.141824 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:26.141803    4496 cpu_manager.go:414] "containerMap: removing container" podUID="2114be44-517b-4282-ab24-db10a832039d" containerName="revoke-keys"
Nov 02 03:34:26.141875 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:26.141827    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2114be44-517b-4282-ab24-db10a832039d" containerName="revoke-keys"
Nov 02 03:34:26.141976 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:26.141965    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="2114be44-517b-4282-ab24-db10a832039d" containerName="revoke-keys"
Nov 02 03:34:26.146259 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:26.146237    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d5625-58c6cfdff9-l8wn2"
Nov 02 03:34:26.198466 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:26.198442    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9d5625-58c6cfdff9-l8wn2"]
Nov 02 03:34:26.203626 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:26.203588    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17edac58-4392-408a-bac2-10716f6cf244-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-9d5625-58c6cfdff9-l8wn2\" (UID: \"17edac58-4392-408a-bac2-10716f6cf244\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d5625-58c6cfdff9-l8wn2"
Nov 02 03:34:26.203842 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:26.203809    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bbsx\" (UniqueName: \"kubernetes.io/projected/17edac58-4392-408a-bac2-10716f6cf244-kube-api-access-5bbsx\") pod \"payload-pre-processing-e2e-ipp-b-9d5625-58c6cfdff9-l8wn2\" (UID: \"17edac58-4392-408a-bac2-10716f6cf244\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d5625-58c6cfdff9-l8wn2"
Nov 02 03:34:26.244851 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:26.244824    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-9d5625\""
Nov 02 03:34:26.244991 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:26.244938    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-9d5625-dockercfg-wfsfk\""
Nov 02 03:34:26.305701 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:26.305665    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17edac58-4392-408a-bac2-10716f6cf244-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-9d5625-58c6cfdff9-l8wn2\" (UID: \"17edac58-4392-408a-bac2-10716f6cf244\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d5625-58c6cfdff9-l8wn2"
Nov 02 03:34:26.306054 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:26.306009    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bbsx\" (UniqueName: \"kubernetes.io/projected/17edac58-4392-408a-bac2-10716f6cf244-kube-api-access-5bbsx\") pod \"payload-pre-processing-e2e-ipp-b-9d5625-58c6cfdff9-l8wn2\" (UID: \"17edac58-4392-408a-bac2-10716f6cf244\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d5625-58c6cfdff9-l8wn2"
Nov 02 03:34:26.306715 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:26.306690    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17edac58-4392-408a-bac2-10716f6cf244-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-9d5625-58c6cfdff9-l8wn2\" (UID: \"17edac58-4392-408a-bac2-10716f6cf244\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d5625-58c6cfdff9-l8wn2"
Nov 02 03:34:26.363481 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:26.363445    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bbsx\" (UniqueName: \"kubernetes.io/projected/17edac58-4392-408a-bac2-10716f6cf244-kube-api-access-5bbsx\") pod \"payload-pre-processing-e2e-ipp-b-9d5625-58c6cfdff9-l8wn2\" (UID: \"17edac58-4392-408a-bac2-10716f6cf244\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d5625-58c6cfdff9-l8wn2"
Nov 02 03:34:26.462400 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:26.462314    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d5625-58c6cfdff9-l8wn2"
Nov 02 03:34:26.845045 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:34:26.844531    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17edac58_4392_408a_bac2_10716f6cf244.slice/crio-e63c79021106dc7ba2564d216a71becba96c27c1f7915084ed937e9319db0157 WatchSource:0}: Error finding container e63c79021106dc7ba2564d216a71becba96c27c1f7915084ed937e9319db0157: Status 404 returned error can't find the container with id e63c79021106dc7ba2564d216a71becba96c27c1f7915084ed937e9319db0157
Nov 02 03:34:26.845574 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:26.845546    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9d5625-58c6cfdff9-l8wn2"]
Nov 02 03:34:26.852992 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:26.852970    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9d5625-558f54dc9d-qnzfm"]
Nov 02 03:34:26.858540 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:26.858503    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-9d5625-558f54dc9d-qnzfm"
Nov 02 03:34:26.913997 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:26.913971    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/235eff5d-7ffc-43c8-9381-b400cbbbf497-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-9d5625-558f54dc9d-qnzfm\" (UID: \"235eff5d-7ffc-43c8-9381-b400cbbbf497\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9d5625-558f54dc9d-qnzfm"
Nov 02 03:34:26.914198 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:26.914174    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gplfs\" (UniqueName: \"kubernetes.io/projected/235eff5d-7ffc-43c8-9381-b400cbbbf497-kube-api-access-gplfs\") pod \"payload-processing-e2e-ipp-b-9d5625-558f54dc9d-qnzfm\" (UID: \"235eff5d-7ffc-43c8-9381-b400cbbbf497\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9d5625-558f54dc9d-qnzfm"
Nov 02 03:34:26.954509 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:26.954471    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9d5625-558f54dc9d-qnzfm"]
Nov 02 03:34:27.014982 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:27.014952    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gplfs\" (UniqueName: \"kubernetes.io/projected/235eff5d-7ffc-43c8-9381-b400cbbbf497-kube-api-access-gplfs\") pod \"payload-processing-e2e-ipp-b-9d5625-558f54dc9d-qnzfm\" (UID: \"235eff5d-7ffc-43c8-9381-b400cbbbf497\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9d5625-558f54dc9d-qnzfm"
Nov 02 03:34:27.015155 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:27.015051    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/235eff5d-7ffc-43c8-9381-b400cbbbf497-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-9d5625-558f54dc9d-qnzfm\" (UID: \"235eff5d-7ffc-43c8-9381-b400cbbbf497\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9d5625-558f54dc9d-qnzfm"
Nov 02 03:34:27.015791 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:27.015772    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/235eff5d-7ffc-43c8-9381-b400cbbbf497-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-9d5625-558f54dc9d-qnzfm\" (UID: \"235eff5d-7ffc-43c8-9381-b400cbbbf497\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9d5625-558f54dc9d-qnzfm"
Nov 02 03:34:27.072613 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:27.072589    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gplfs\" (UniqueName: \"kubernetes.io/projected/235eff5d-7ffc-43c8-9381-b400cbbbf497-kube-api-access-gplfs\") pod \"payload-processing-e2e-ipp-b-9d5625-558f54dc9d-qnzfm\" (UID: \"235eff5d-7ffc-43c8-9381-b400cbbbf497\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9d5625-558f54dc9d-qnzfm"
Nov 02 03:34:27.133715 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:27.133629    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d5625-58c6cfdff9-l8wn2" event={"ID":"17edac58-4392-408a-bac2-10716f6cf244","Type":"ContainerStarted","Data":"4ef2199a820a06cd1d05e0ff1b5664b8e9ea04923dcb935ed65161b18aef8db4"}
Nov 02 03:34:27.133715 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:27.133686    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d5625-58c6cfdff9-l8wn2" event={"ID":"17edac58-4392-408a-bac2-10716f6cf244","Type":"ContainerStarted","Data":"e63c79021106dc7ba2564d216a71becba96c27c1f7915084ed937e9319db0157"}
Nov 02 03:34:27.133715 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:27.133708    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d5625-58c6cfdff9-l8wn2"
Nov 02 03:34:27.201135 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:27.201107    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-9d5625-558f54dc9d-qnzfm"
Nov 02 03:34:27.587127 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:27.587081    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d5625-58c6cfdff9-l8wn2" podStartSLOduration=1.587066793 podStartE2EDuration="1.587066793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:34:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:34:27.25777578 +0000 UTC m=+2807.836383161" watchObservedRunningTime="2026-11-02 03:34:27.587066793 +0000 UTC m=+2808.165674173"
Nov 02 03:34:27.589078 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:27.589050    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9d5625-558f54dc9d-qnzfm"]
Nov 02 03:34:27.589280 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:34:27.589257    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod235eff5d_7ffc_43c8_9381_b400cbbbf497.slice/crio-44d94302ed6c712402f47701d5d3b5f6c2eb42818811e0ef020a18d2228f9cfe WatchSource:0}: Error finding container 44d94302ed6c712402f47701d5d3b5f6c2eb42818811e0ef020a18d2228f9cfe: Status 404 returned error can't find the container with id 44d94302ed6c712402f47701d5d3b5f6c2eb42818811e0ef020a18d2228f9cfe
Nov 02 03:34:28.144414 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:28.144344    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-9d5625-558f54dc9d-qnzfm" event={"ID":"235eff5d-7ffc-43c8-9381-b400cbbbf497","Type":"ContainerStarted","Data":"f1e4a3236307dc3a5632e8ac133bd2b51d385b587ea0a5d090afd2d69063b40a"}
Nov 02 03:34:28.144414 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:28.144417    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-9d5625-558f54dc9d-qnzfm" event={"ID":"235eff5d-7ffc-43c8-9381-b400cbbbf497","Type":"ContainerStarted","Data":"44d94302ed6c712402f47701d5d3b5f6c2eb42818811e0ef020a18d2228f9cfe"}
Nov 02 03:34:28.144715 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:28.144602    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-9d5625-558f54dc9d-qnzfm"
Nov 02 03:34:28.264464 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:28.264405    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-9d5625-558f54dc9d-qnzfm" podStartSLOduration=2.26438717 podStartE2EDuration="2.26438717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:34:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:34:28.259456679 +0000 UTC m=+2808.838064061" watchObservedRunningTime="2026-11-02 03:34:28.26438717 +0000 UTC m=+2808.842994548"
Nov 02 03:34:33.526387 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:34:33.526349    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 03:34:38.147205 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:38.147167    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d5625-58c6cfdff9-l8wn2"
Nov 02 03:34:39.155695 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:34:39.155665    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-9d5625-558f54dc9d-qnzfm"
Nov 02 03:35:03.530463 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:35:03.530427    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 03:35:07.614121 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:07.614078    4496 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-dd87f0-54b67c4cd8-k55c8_0609027c-0577-4487-8d37-0c0316de5c5f/payload-processing/0.log"
Nov 02 03:35:07.918770 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:07.918683    4496 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-gzc2f_bd8a1b95-491e-44ea-988a-e4437c386a1a/payload-processing/0.log"
Nov 02 03:35:08.159140 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:08.159112    4496 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-gzc2f_bd8a1b95-491e-44ea-988a-e4437c386a1a/payload-processing/0.log"
Nov 02 03:35:09.662401 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:09.662365    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dd87f0-kcqqn"]
Nov 02 03:35:09.668121 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:09.668102    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dd87f0-kcqqn"
Nov 02 03:35:09.759661 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:09.759622    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ebf7e9b-bce3-4b88-9f30-1341cc3bfb7e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-dd87f0-kcqqn\" (UID: \"0ebf7e9b-bce3-4b88-9f30-1341cc3bfb7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dd87f0-kcqqn"
Nov 02 03:35:09.759851 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:09.759676    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ms8ng\" (UniqueName: \"kubernetes.io/projected/0ebf7e9b-bce3-4b88-9f30-1341cc3bfb7e-kube-api-access-ms8ng\") pod \"maas-api-revoke-keys-e2e-ipp-a-dd87f0-kcqqn\" (UID: \"0ebf7e9b-bce3-4b88-9f30-1341cc3bfb7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dd87f0-kcqqn"
Nov 02 03:35:09.759851 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:09.759798    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ebf7e9b-bce3-4b88-9f30-1341cc3bfb7e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-dd87f0-kcqqn\" (UID: \"0ebf7e9b-bce3-4b88-9f30-1341cc3bfb7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dd87f0-kcqqn"
Nov 02 03:35:09.763152 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:09.763114    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 03:35:09.771837 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:09.771776    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9z5bk\""
Nov 02 03:35:09.771837 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:09.771795    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 03:35:09.771837 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:09.771783    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 03:35:09.776191 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:09.776166    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dd87f0-kcqqn"]
Nov 02 03:35:09.861066 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:09.861030    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ebf7e9b-bce3-4b88-9f30-1341cc3bfb7e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-dd87f0-kcqqn\" (UID: \"0ebf7e9b-bce3-4b88-9f30-1341cc3bfb7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dd87f0-kcqqn"
Nov 02 03:35:09.861250 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:09.861098    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ms8ng\" (UniqueName: \"kubernetes.io/projected/0ebf7e9b-bce3-4b88-9f30-1341cc3bfb7e-kube-api-access-ms8ng\") pod \"maas-api-revoke-keys-e2e-ipp-a-dd87f0-kcqqn\" (UID: \"0ebf7e9b-bce3-4b88-9f30-1341cc3bfb7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dd87f0-kcqqn"
Nov 02 03:35:09.861250 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:09.861167    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ebf7e9b-bce3-4b88-9f30-1341cc3bfb7e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-dd87f0-kcqqn\" (UID: \"0ebf7e9b-bce3-4b88-9f30-1341cc3bfb7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dd87f0-kcqqn"
Nov 02 03:35:09.861866 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:09.861839    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ebf7e9b-bce3-4b88-9f30-1341cc3bfb7e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-dd87f0-kcqqn\" (UID: \"0ebf7e9b-bce3-4b88-9f30-1341cc3bfb7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dd87f0-kcqqn"
Nov 02 03:35:09.863514 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:09.863483    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ebf7e9b-bce3-4b88-9f30-1341cc3bfb7e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-dd87f0-kcqqn\" (UID: \"0ebf7e9b-bce3-4b88-9f30-1341cc3bfb7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dd87f0-kcqqn"
Nov 02 03:35:09.919253 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:09.919184    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ms8ng\" (UniqueName: \"kubernetes.io/projected/0ebf7e9b-bce3-4b88-9f30-1341cc3bfb7e-kube-api-access-ms8ng\") pod \"maas-api-revoke-keys-e2e-ipp-a-dd87f0-kcqqn\" (UID: \"0ebf7e9b-bce3-4b88-9f30-1341cc3bfb7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dd87f0-kcqqn"
Nov 02 03:35:09.981901 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:09.981859    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dd87f0-kcqqn"
Nov 02 03:35:10.356920 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:35:10.356883    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ebf7e9b_bce3_4b88_9f30_1341cc3bfb7e.slice/crio-5949b441b453787303bf341b3c7fd1fcbfa292cf7d858f0162ae1cb04ae4ea8c WatchSource:0}: Error finding container 5949b441b453787303bf341b3c7fd1fcbfa292cf7d858f0162ae1cb04ae4ea8c: Status 404 returned error can't find the container with id 5949b441b453787303bf341b3c7fd1fcbfa292cf7d858f0162ae1cb04ae4ea8c
Nov 02 03:35:10.360078 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:10.360051    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dd87f0-kcqqn"]
Nov 02 03:35:10.457320 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:10.457275    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dd87f0-kcqqn" event={"ID":"0ebf7e9b-bce3-4b88-9f30-1341cc3bfb7e","Type":"ContainerStarted","Data":"5949b441b453787303bf341b3c7fd1fcbfa292cf7d858f0162ae1cb04ae4ea8c"}
Nov 02 03:35:11.467652 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:11.467555    4496 generic.go:356] "Generic (PLEG): container finished" podID="0ebf7e9b-bce3-4b88-9f30-1341cc3bfb7e" containerID="dabf6b33ba89435c774afa3c9306bc2b0e7a40870c3f3a8f417b702b44160091" exitCode=0
Nov 02 03:35:11.468136 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:11.467652    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dd87f0-kcqqn" event={"ID":"0ebf7e9b-bce3-4b88-9f30-1341cc3bfb7e","Type":"ContainerDied","Data":"dabf6b33ba89435c774afa3c9306bc2b0e7a40870c3f3a8f417b702b44160091"}
Nov 02 03:35:12.642975 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:12.642937    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dd87f0-kcqqn"
Nov 02 03:35:12.694840 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:12.694812    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ebf7e9b-bce3-4b88-9f30-1341cc3bfb7e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ebf7e9b-bce3-4b88-9f30-1341cc3bfb7e-maas-api-openshift-ca\") pod \"0ebf7e9b-bce3-4b88-9f30-1341cc3bfb7e\" (UID: \"0ebf7e9b-bce3-4b88-9f30-1341cc3bfb7e\") "
Nov 02 03:35:12.695091 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:12.695070    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ebf7e9b-bce3-4b88-9f30-1341cc3bfb7e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ebf7e9b-bce3-4b88-9f30-1341cc3bfb7e-maas-api-xks-ca\") pod \"0ebf7e9b-bce3-4b88-9f30-1341cc3bfb7e\" (UID: \"0ebf7e9b-bce3-4b88-9f30-1341cc3bfb7e\") "
Nov 02 03:35:12.695158 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:12.695119    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ebf7e9b-bce3-4b88-9f30-1341cc3bfb7e-kube-api-access-ms8ng\" (UniqueName: \"kubernetes.io/projected/0ebf7e9b-bce3-4b88-9f30-1341cc3bfb7e-kube-api-access-ms8ng\") pod \"0ebf7e9b-bce3-4b88-9f30-1341cc3bfb7e\" (UID: \"0ebf7e9b-bce3-4b88-9f30-1341cc3bfb7e\") "
Nov 02 03:35:12.695255 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:12.695236    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ebf7e9b-bce3-4b88-9f30-1341cc3bfb7e-maas-api-openshift-ca" pod "0ebf7e9b-bce3-4b88-9f30-1341cc3bfb7e" (UID: "0ebf7e9b-bce3-4b88-9f30-1341cc3bfb7e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:35:12.695547 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:12.695530    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ebf7e9b-bce3-4b88-9f30-1341cc3bfb7e-maas-api-openshift-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:35:12.697011 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:12.696934    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ebf7e9b-bce3-4b88-9f30-1341cc3bfb7e-maas-api-xks-ca" pod "0ebf7e9b-bce3-4b88-9f30-1341cc3bfb7e" (UID: "0ebf7e9b-bce3-4b88-9f30-1341cc3bfb7e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 03:35:12.697156 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:12.697125    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ebf7e9b-bce3-4b88-9f30-1341cc3bfb7e-kube-api-access-ms8ng" pod "0ebf7e9b-bce3-4b88-9f30-1341cc3bfb7e" (UID: "0ebf7e9b-bce3-4b88-9f30-1341cc3bfb7e"). InnerVolumeSpecName "kube-api-access-ms8ng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:35:12.797337 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:12.797261    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ebf7e9b-bce3-4b88-9f30-1341cc3bfb7e-maas-api-xks-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:35:12.797337 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:12.797292    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ms8ng\" (UniqueName: \"kubernetes.io/projected/0ebf7e9b-bce3-4b88-9f30-1341cc3bfb7e-kube-api-access-ms8ng\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:35:13.486526 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:13.486496    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dd87f0-kcqqn"
Nov 02 03:35:13.486678 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:13.486537    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dd87f0-kcqqn" event={"ID":"0ebf7e9b-bce3-4b88-9f30-1341cc3bfb7e","Type":"ContainerDied","Data":"5949b441b453787303bf341b3c7fd1fcbfa292cf7d858f0162ae1cb04ae4ea8c"}
Nov 02 03:35:13.486678 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:13.486595    4496 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5949b441b453787303bf341b3c7fd1fcbfa292cf7d858f0162ae1cb04ae4ea8c"
Nov 02 03:35:18.665233 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:18.665192    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-csm6n"]
Nov 02 03:35:18.665664 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:18.665459    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-csm6n" podUID="2cfcd892-9dc1-4853-bb63-3b79e5c70942" containerName="discovery" containerID="cri-o://5c38de8389b0b433309dde41f9ecd53dae948d17ffaf26c51ec089d1c9158986" gracePeriod=30
Nov 02 03:35:18.956700 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:18.956357    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-csm6n"
Nov 02 03:35:19.074334 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:19.073485    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2cfcd892-9dc1-4853-bb63-3b79e5c70942-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2cfcd892-9dc1-4853-bb63-3b79e5c70942-istio-kubeconfig\") pod \"2cfcd892-9dc1-4853-bb63-3b79e5c70942\" (UID: \"2cfcd892-9dc1-4853-bb63-3b79e5c70942\") "
Nov 02 03:35:19.074334 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:19.073595    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2cfcd892-9dc1-4853-bb63-3b79e5c70942-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2cfcd892-9dc1-4853-bb63-3b79e5c70942-istio-csr-ca-configmap\") pod \"2cfcd892-9dc1-4853-bb63-3b79e5c70942\" (UID: \"2cfcd892-9dc1-4853-bb63-3b79e5c70942\") "
Nov 02 03:35:19.074334 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:19.073650    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cfcd892-9dc1-4853-bb63-3b79e5c70942-kube-api-access-2wzxs\" (UniqueName: \"kubernetes.io/projected/2cfcd892-9dc1-4853-bb63-3b79e5c70942-kube-api-access-2wzxs\") pod \"2cfcd892-9dc1-4853-bb63-3b79e5c70942\" (UID: \"2cfcd892-9dc1-4853-bb63-3b79e5c70942\") "
Nov 02 03:35:19.074334 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:19.073780    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2cfcd892-9dc1-4853-bb63-3b79e5c70942-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2cfcd892-9dc1-4853-bb63-3b79e5c70942-istio-csr-dns-cert\") pod \"2cfcd892-9dc1-4853-bb63-3b79e5c70942\" (UID: \"2cfcd892-9dc1-4853-bb63-3b79e5c70942\") "
Nov 02 03:35:19.074334 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:19.073824    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cfcd892-9dc1-4853-bb63-3b79e5c70942-istio-token\" (UniqueName: \"kubernetes.io/projected/2cfcd892-9dc1-4853-bb63-3b79e5c70942-istio-token\") pod \"2cfcd892-9dc1-4853-bb63-3b79e5c70942\" (UID: \"2cfcd892-9dc1-4853-bb63-3b79e5c70942\") "
Nov 02 03:35:19.074334 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:19.073885    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2cfcd892-9dc1-4853-bb63-3b79e5c70942-cacerts\" (UniqueName: \"kubernetes.io/secret/2cfcd892-9dc1-4853-bb63-3b79e5c70942-cacerts\") pod \"2cfcd892-9dc1-4853-bb63-3b79e5c70942\" (UID: \"2cfcd892-9dc1-4853-bb63-3b79e5c70942\") "
Nov 02 03:35:19.074334 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:19.073920    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cfcd892-9dc1-4853-bb63-3b79e5c70942-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cfcd892-9dc1-4853-bb63-3b79e5c70942-local-certs\") pod \"2cfcd892-9dc1-4853-bb63-3b79e5c70942\" (UID: \"2cfcd892-9dc1-4853-bb63-3b79e5c70942\") "
Nov 02 03:35:19.075562 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:19.075273    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cfcd892-9dc1-4853-bb63-3b79e5c70942-istio-csr-ca-configmap" pod "2cfcd892-9dc1-4853-bb63-3b79e5c70942" (UID: "2cfcd892-9dc1-4853-bb63-3b79e5c70942"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:35:19.079722 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:19.079431    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cfcd892-9dc1-4853-bb63-3b79e5c70942-istio-kubeconfig" pod "2cfcd892-9dc1-4853-bb63-3b79e5c70942" (UID: "2cfcd892-9dc1-4853-bb63-3b79e5c70942"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 03:35:19.082098 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:19.082067    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cfcd892-9dc1-4853-bb63-3b79e5c70942-istio-token" pod "2cfcd892-9dc1-4853-bb63-3b79e5c70942" (UID: "2cfcd892-9dc1-4853-bb63-3b79e5c70942"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:35:19.082497 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:19.082455    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cfcd892-9dc1-4853-bb63-3b79e5c70942-cacerts" pod "2cfcd892-9dc1-4853-bb63-3b79e5c70942" (UID: "2cfcd892-9dc1-4853-bb63-3b79e5c70942"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 03:35:19.083360 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:19.083323    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cfcd892-9dc1-4853-bb63-3b79e5c70942-istio-csr-dns-cert" pod "2cfcd892-9dc1-4853-bb63-3b79e5c70942" (UID: "2cfcd892-9dc1-4853-bb63-3b79e5c70942"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 03:35:19.083460 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:19.083365    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cfcd892-9dc1-4853-bb63-3b79e5c70942-local-certs" pod "2cfcd892-9dc1-4853-bb63-3b79e5c70942" (UID: "2cfcd892-9dc1-4853-bb63-3b79e5c70942"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:35:19.084630 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:19.084603    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cfcd892-9dc1-4853-bb63-3b79e5c70942-kube-api-access-2wzxs" pod "2cfcd892-9dc1-4853-bb63-3b79e5c70942" (UID: "2cfcd892-9dc1-4853-bb63-3b79e5c70942"). InnerVolumeSpecName "kube-api-access-2wzxs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:35:19.175961 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:19.175886    4496 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2cfcd892-9dc1-4853-bb63-3b79e5c70942-cacerts\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:35:19.175961 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:19.175915    4496 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cfcd892-9dc1-4853-bb63-3b79e5c70942-local-certs\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:35:19.175961 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:19.175924    4496 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2cfcd892-9dc1-4853-bb63-3b79e5c70942-istio-kubeconfig\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:35:19.175961 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:19.175934    4496 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2cfcd892-9dc1-4853-bb63-3b79e5c70942-istio-csr-ca-configmap\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:35:19.175961 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:19.175944    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wzxs\" (UniqueName: \"kubernetes.io/projected/2cfcd892-9dc1-4853-bb63-3b79e5c70942-kube-api-access-2wzxs\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:35:19.175961 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:19.175952    4496 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2cfcd892-9dc1-4853-bb63-3b79e5c70942-istio-csr-dns-cert\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:35:19.175961 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:19.175963    4496 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cfcd892-9dc1-4853-bb63-3b79e5c70942-istio-token\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:35:19.531444 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:19.531407    4496 generic.go:356] "Generic (PLEG): container finished" podID="2cfcd892-9dc1-4853-bb63-3b79e5c70942" containerID="5c38de8389b0b433309dde41f9ecd53dae948d17ffaf26c51ec089d1c9158986" exitCode=0
Nov 02 03:35:19.531630 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:19.531454    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-csm6n"
Nov 02 03:35:19.531630 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:19.531493    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-csm6n" event={"ID":"2cfcd892-9dc1-4853-bb63-3b79e5c70942","Type":"ContainerDied","Data":"5c38de8389b0b433309dde41f9ecd53dae948d17ffaf26c51ec089d1c9158986"}
Nov 02 03:35:19.531630 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:19.531549    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-csm6n" event={"ID":"2cfcd892-9dc1-4853-bb63-3b79e5c70942","Type":"ContainerDied","Data":"116109002c1f187a5e8e0d8720febb8e59faff1c12ce7dce33900cd5f655a7a1"}
Nov 02 03:35:19.531630 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:19.531575    4496 scope.go:122] "RemoveContainer" containerID="5c38de8389b0b433309dde41f9ecd53dae948d17ffaf26c51ec089d1c9158986"
Nov 02 03:35:19.544055 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:19.544030    4496 scope.go:122] "RemoveContainer" containerID="5c38de8389b0b433309dde41f9ecd53dae948d17ffaf26c51ec089d1c9158986"
Nov 02 03:35:19.544397 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:35:19.544372    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c38de8389b0b433309dde41f9ecd53dae948d17ffaf26c51ec089d1c9158986\": container with ID starting with 5c38de8389b0b433309dde41f9ecd53dae948d17ffaf26c51ec089d1c9158986 not found: ID does not exist" containerID="5c38de8389b0b433309dde41f9ecd53dae948d17ffaf26c51ec089d1c9158986"
Nov 02 03:35:19.544451 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:19.544412    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c38de8389b0b433309dde41f9ecd53dae948d17ffaf26c51ec089d1c9158986"} err="failed to get container status \"5c38de8389b0b433309dde41f9ecd53dae948d17ffaf26c51ec089d1c9158986\": rpc error: code = NotFound desc = could not find container \"5c38de8389b0b433309dde41f9ecd53dae948d17ffaf26c51ec089d1c9158986\": container with ID starting with 5c38de8389b0b433309dde41f9ecd53dae948d17ffaf26c51ec089d1c9158986 not found: ID does not exist"
Nov 02 03:35:19.703751 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:19.703717    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-csm6n"]
Nov 02 03:35:19.754324 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:19.754290    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-csm6n"]
Nov 02 03:35:21.574577 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:21.574538    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cfcd892-9dc1-4853-bb63-3b79e5c70942" path="/var/lib/kubelet/pods/2cfcd892-9dc1-4853-bb63-3b79e5c70942/volumes"
Nov 02 03:35:32.787443 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:32.787404    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9d5625-558f54dc9d-qnzfm"]
Nov 02 03:35:32.788284 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:32.788232    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-9d5625-558f54dc9d-qnzfm" podUID="235eff5d-7ffc-43c8-9381-b400cbbbf497" containerName="payload-processing" containerID="cri-o://f1e4a3236307dc3a5632e8ac133bd2b51d385b587ea0a5d090afd2d69063b40a" gracePeriod=30
Nov 02 03:35:32.842539 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:32.842505    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9d5625-58c6cfdff9-l8wn2"]
Nov 02 03:35:32.842786 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:32.842758    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d5625-58c6cfdff9-l8wn2" podUID="17edac58-4392-408a-bac2-10716f6cf244" containerName="payload-pre-processing" containerID="cri-o://4ef2199a820a06cd1d05e0ff1b5664b8e9ea04923dcb935ed65161b18aef8db4" gracePeriod=30
Nov 02 03:35:33.126722 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:33.126695    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-9d5625-558f54dc9d-qnzfm"
Nov 02 03:35:33.159333 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:33.159300    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d5625-58c6cfdff9-l8wn2"
Nov 02 03:35:33.240135 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:33.240104    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/235eff5d-7ffc-43c8-9381-b400cbbbf497-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/235eff5d-7ffc-43c8-9381-b400cbbbf497-plugins-config-volume\") pod \"235eff5d-7ffc-43c8-9381-b400cbbbf497\" (UID: \"235eff5d-7ffc-43c8-9381-b400cbbbf497\") "
Nov 02 03:35:33.240324 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:33.240206    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/235eff5d-7ffc-43c8-9381-b400cbbbf497-kube-api-access-gplfs\" (UniqueName: \"kubernetes.io/projected/235eff5d-7ffc-43c8-9381-b400cbbbf497-kube-api-access-gplfs\") pod \"235eff5d-7ffc-43c8-9381-b400cbbbf497\" (UID: \"235eff5d-7ffc-43c8-9381-b400cbbbf497\") "
Nov 02 03:35:33.240496 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:33.240468    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/235eff5d-7ffc-43c8-9381-b400cbbbf497-plugins-config-volume" pod "235eff5d-7ffc-43c8-9381-b400cbbbf497" (UID: "235eff5d-7ffc-43c8-9381-b400cbbbf497"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:35:33.240633 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:33.240619    4496 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/235eff5d-7ffc-43c8-9381-b400cbbbf497-plugins-config-volume\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:35:33.242823 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:33.242796    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/235eff5d-7ffc-43c8-9381-b400cbbbf497-kube-api-access-gplfs" pod "235eff5d-7ffc-43c8-9381-b400cbbbf497" (UID: "235eff5d-7ffc-43c8-9381-b400cbbbf497"). InnerVolumeSpecName "kube-api-access-gplfs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:35:33.343630 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:33.343523    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17edac58-4392-408a-bac2-10716f6cf244-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17edac58-4392-408a-bac2-10716f6cf244-plugins-config-volume\") pod \"17edac58-4392-408a-bac2-10716f6cf244\" (UID: \"17edac58-4392-408a-bac2-10716f6cf244\") "
Nov 02 03:35:33.343630 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:33.343584    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17edac58-4392-408a-bac2-10716f6cf244-kube-api-access-5bbsx\" (UniqueName: \"kubernetes.io/projected/17edac58-4392-408a-bac2-10716f6cf244-kube-api-access-5bbsx\") pod \"17edac58-4392-408a-bac2-10716f6cf244\" (UID: \"17edac58-4392-408a-bac2-10716f6cf244\") "
Nov 02 03:35:33.344099 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:33.344079    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gplfs\" (UniqueName: \"kubernetes.io/projected/235eff5d-7ffc-43c8-9381-b400cbbbf497-kube-api-access-gplfs\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:35:33.345183 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:33.345126    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17edac58-4392-408a-bac2-10716f6cf244-plugins-config-volume" pod "17edac58-4392-408a-bac2-10716f6cf244" (UID: "17edac58-4392-408a-bac2-10716f6cf244"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:35:33.347623 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:33.347600    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17edac58-4392-408a-bac2-10716f6cf244-kube-api-access-5bbsx" pod "17edac58-4392-408a-bac2-10716f6cf244" (UID: "17edac58-4392-408a-bac2-10716f6cf244"). InnerVolumeSpecName "kube-api-access-5bbsx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:35:33.445753 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:33.445720    4496 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17edac58-4392-408a-bac2-10716f6cf244-plugins-config-volume\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:35:33.445753 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:33.445753    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bbsx\" (UniqueName: \"kubernetes.io/projected/17edac58-4392-408a-bac2-10716f6cf244-kube-api-access-5bbsx\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:35:33.527291 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:35:33.527251    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 03:35:33.643783 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:33.643698    4496 generic.go:356] "Generic (PLEG): container finished" podID="235eff5d-7ffc-43c8-9381-b400cbbbf497" containerID="f1e4a3236307dc3a5632e8ac133bd2b51d385b587ea0a5d090afd2d69063b40a" exitCode=0
Nov 02 03:35:33.643783 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:33.643745    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-9d5625-558f54dc9d-qnzfm"
Nov 02 03:35:33.644057 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:33.643772    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-9d5625-558f54dc9d-qnzfm" event={"ID":"235eff5d-7ffc-43c8-9381-b400cbbbf497","Type":"ContainerDied","Data":"f1e4a3236307dc3a5632e8ac133bd2b51d385b587ea0a5d090afd2d69063b40a"}
Nov 02 03:35:33.644057 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:33.643837    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-9d5625-558f54dc9d-qnzfm" event={"ID":"235eff5d-7ffc-43c8-9381-b400cbbbf497","Type":"ContainerDied","Data":"44d94302ed6c712402f47701d5d3b5f6c2eb42818811e0ef020a18d2228f9cfe"}
Nov 02 03:35:33.644057 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:33.643867    4496 scope.go:122] "RemoveContainer" containerID="f1e4a3236307dc3a5632e8ac133bd2b51d385b587ea0a5d090afd2d69063b40a"
Nov 02 03:35:33.680876 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:33.679883    4496 generic.go:356] "Generic (PLEG): container finished" podID="17edac58-4392-408a-bac2-10716f6cf244" containerID="4ef2199a820a06cd1d05e0ff1b5664b8e9ea04923dcb935ed65161b18aef8db4" exitCode=0
Nov 02 03:35:33.680876 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:33.679931    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d5625-58c6cfdff9-l8wn2" event={"ID":"17edac58-4392-408a-bac2-10716f6cf244","Type":"ContainerDied","Data":"4ef2199a820a06cd1d05e0ff1b5664b8e9ea04923dcb935ed65161b18aef8db4"}
Nov 02 03:35:33.680876 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:33.679981    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d5625-58c6cfdff9-l8wn2" event={"ID":"17edac58-4392-408a-bac2-10716f6cf244","Type":"ContainerDied","Data":"e63c79021106dc7ba2564d216a71becba96c27c1f7915084ed937e9319db0157"}
Nov 02 03:35:33.680876 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:33.680073    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d5625-58c6cfdff9-l8wn2"
Nov 02 03:35:33.696712 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:33.696691    4496 scope.go:122] "RemoveContainer" containerID="f1e4a3236307dc3a5632e8ac133bd2b51d385b587ea0a5d090afd2d69063b40a"
Nov 02 03:35:33.697166 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:35:33.697132    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1e4a3236307dc3a5632e8ac133bd2b51d385b587ea0a5d090afd2d69063b40a\": container with ID starting with f1e4a3236307dc3a5632e8ac133bd2b51d385b587ea0a5d090afd2d69063b40a not found: ID does not exist" containerID="f1e4a3236307dc3a5632e8ac133bd2b51d385b587ea0a5d090afd2d69063b40a"
Nov 02 03:35:33.697286 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:33.697178    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1e4a3236307dc3a5632e8ac133bd2b51d385b587ea0a5d090afd2d69063b40a"} err="failed to get container status \"f1e4a3236307dc3a5632e8ac133bd2b51d385b587ea0a5d090afd2d69063b40a\": rpc error: code = NotFound desc = could not find container \"f1e4a3236307dc3a5632e8ac133bd2b51d385b587ea0a5d090afd2d69063b40a\": container with ID starting with f1e4a3236307dc3a5632e8ac133bd2b51d385b587ea0a5d090afd2d69063b40a not found: ID does not exist"
Nov 02 03:35:33.697286 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:33.697204    4496 scope.go:122] "RemoveContainer" containerID="4ef2199a820a06cd1d05e0ff1b5664b8e9ea04923dcb935ed65161b18aef8db4"
Nov 02 03:35:33.713712 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:33.713691    4496 scope.go:122] "RemoveContainer" containerID="4ef2199a820a06cd1d05e0ff1b5664b8e9ea04923dcb935ed65161b18aef8db4"
Nov 02 03:35:33.714074 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:35:33.714053    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ef2199a820a06cd1d05e0ff1b5664b8e9ea04923dcb935ed65161b18aef8db4\": container with ID starting with 4ef2199a820a06cd1d05e0ff1b5664b8e9ea04923dcb935ed65161b18aef8db4 not found: ID does not exist" containerID="4ef2199a820a06cd1d05e0ff1b5664b8e9ea04923dcb935ed65161b18aef8db4"
Nov 02 03:35:33.714160 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:33.714086    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ef2199a820a06cd1d05e0ff1b5664b8e9ea04923dcb935ed65161b18aef8db4"} err="failed to get container status \"4ef2199a820a06cd1d05e0ff1b5664b8e9ea04923dcb935ed65161b18aef8db4\": rpc error: code = NotFound desc = could not find container \"4ef2199a820a06cd1d05e0ff1b5664b8e9ea04923dcb935ed65161b18aef8db4\": container with ID starting with 4ef2199a820a06cd1d05e0ff1b5664b8e9ea04923dcb935ed65161b18aef8db4 not found: ID does not exist"
Nov 02 03:35:33.812578 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:33.812550    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9d5625-558f54dc9d-qnzfm"]
Nov 02 03:35:33.815726 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:33.815701    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9d5625-558f54dc9d-qnzfm"]
Nov 02 03:35:33.923718 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:33.923640    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9d5625-58c6cfdff9-l8wn2"]
Nov 02 03:35:33.926866 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:33.926842    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9d5625-58c6cfdff9-l8wn2"]
Nov 02 03:35:34.552869 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:34.552838    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9d5625-5hfhp"]
Nov 02 03:35:34.603067 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:34.603027    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9d5625-5hfhp"]
Nov 02 03:35:34.933857 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:34.933771    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-abc348-h65xh"]
Nov 02 03:35:34.934419 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:34.934403    4496 cpu_manager.go:414] "containerMap: removing container" podUID="235eff5d-7ffc-43c8-9381-b400cbbbf497" containerName="payload-processing"
Nov 02 03:35:34.934479 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:34.934420    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="235eff5d-7ffc-43c8-9381-b400cbbbf497" containerName="payload-processing"
Nov 02 03:35:34.934479 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:34.934441    4496 cpu_manager.go:414] "containerMap: removing container" podUID="17edac58-4392-408a-bac2-10716f6cf244" containerName="payload-pre-processing"
Nov 02 03:35:34.934479 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:34.934447    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17edac58-4392-408a-bac2-10716f6cf244" containerName="payload-pre-processing"
Nov 02 03:35:34.934479 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:34.934457    4496 cpu_manager.go:414] "containerMap: removing container" podUID="2cfcd892-9dc1-4853-bb63-3b79e5c70942" containerName="discovery"
Nov 02 03:35:34.934479 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:34.934463    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cfcd892-9dc1-4853-bb63-3b79e5c70942" containerName="discovery"
Nov 02 03:35:34.934479 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:34.934478    4496 cpu_manager.go:414] "containerMap: removing container" podUID="0ebf7e9b-bce3-4b88-9f30-1341cc3bfb7e" containerName="revoke-keys"
Nov 02 03:35:34.934670 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:34.934484    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ebf7e9b-bce3-4b88-9f30-1341cc3bfb7e" containerName="revoke-keys"
Nov 02 03:35:34.934670 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:34.934569    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="17edac58-4392-408a-bac2-10716f6cf244" containerName="payload-pre-processing"
Nov 02 03:35:34.934670 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:34.934580    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cfcd892-9dc1-4853-bb63-3b79e5c70942" containerName="discovery"
Nov 02 03:35:34.934670 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:34.934588    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ebf7e9b-bce3-4b88-9f30-1341cc3bfb7e" containerName="revoke-keys"
Nov 02 03:35:34.934670 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:34.934598    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="235eff5d-7ffc-43c8-9381-b400cbbbf497" containerName="payload-processing"
Nov 02 03:35:34.939623 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:34.939605    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-abc348-h65xh"
Nov 02 03:35:34.961242 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:34.961217    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63fa6a92-0860-40ca-98eb-df5946bfeaf4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-abc348-h65xh\" (UID: \"63fa6a92-0860-40ca-98eb-df5946bfeaf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-abc348-h65xh"
Nov 02 03:35:34.961353 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:34.961334    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpdb2\" (UniqueName: \"kubernetes.io/projected/63fa6a92-0860-40ca-98eb-df5946bfeaf4-kube-api-access-hpdb2\") pod \"maas-api-revoke-keys-e2e-ipp-a-abc348-h65xh\" (UID: \"63fa6a92-0860-40ca-98eb-df5946bfeaf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-abc348-h65xh"
Nov 02 03:35:34.961421 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:34.961406    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63fa6a92-0860-40ca-98eb-df5946bfeaf4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-abc348-h65xh\" (UID: \"63fa6a92-0860-40ca-98eb-df5946bfeaf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-abc348-h65xh"
Nov 02 03:35:34.992622 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:34.992553    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 03:35:34.992622 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:34.992573    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9z5bk\""
Nov 02 03:35:34.992622 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:34.992574    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 03:35:34.992879 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:34.992711    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 03:35:34.994910 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:34.994886    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-abc348-h65xh"]
Nov 02 03:35:35.062224 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:35.062190    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpdb2\" (UniqueName: \"kubernetes.io/projected/63fa6a92-0860-40ca-98eb-df5946bfeaf4-kube-api-access-hpdb2\") pod \"maas-api-revoke-keys-e2e-ipp-a-abc348-h65xh\" (UID: \"63fa6a92-0860-40ca-98eb-df5946bfeaf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-abc348-h65xh"
Nov 02 03:35:35.062390 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:35.062257    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63fa6a92-0860-40ca-98eb-df5946bfeaf4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-abc348-h65xh\" (UID: \"63fa6a92-0860-40ca-98eb-df5946bfeaf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-abc348-h65xh"
Nov 02 03:35:35.062439 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:35.062388    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63fa6a92-0860-40ca-98eb-df5946bfeaf4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-abc348-h65xh\" (UID: \"63fa6a92-0860-40ca-98eb-df5946bfeaf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-abc348-h65xh"
Nov 02 03:35:35.062962 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:35.062943    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63fa6a92-0860-40ca-98eb-df5946bfeaf4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-abc348-h65xh\" (UID: \"63fa6a92-0860-40ca-98eb-df5946bfeaf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-abc348-h65xh"
Nov 02 03:35:35.064598 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:35.064577    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63fa6a92-0860-40ca-98eb-df5946bfeaf4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-abc348-h65xh\" (UID: \"63fa6a92-0860-40ca-98eb-df5946bfeaf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-abc348-h65xh"
Nov 02 03:35:35.121473 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:35.121432    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpdb2\" (UniqueName: \"kubernetes.io/projected/63fa6a92-0860-40ca-98eb-df5946bfeaf4-kube-api-access-hpdb2\") pod \"maas-api-revoke-keys-e2e-ipp-a-abc348-h65xh\" (UID: \"63fa6a92-0860-40ca-98eb-df5946bfeaf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-abc348-h65xh"
Nov 02 03:35:35.253427 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:35.253397    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-abc348-h65xh"
Nov 02 03:35:35.575482 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:35.575383    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17edac58-4392-408a-bac2-10716f6cf244" path="/var/lib/kubelet/pods/17edac58-4392-408a-bac2-10716f6cf244/volumes"
Nov 02 03:35:35.575941 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:35.575917    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2114be44-517b-4282-ab24-db10a832039d" path="/var/lib/kubelet/pods/2114be44-517b-4282-ab24-db10a832039d/volumes"
Nov 02 03:35:35.576380 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:35.576360    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="235eff5d-7ffc-43c8-9381-b400cbbbf497" path="/var/lib/kubelet/pods/235eff5d-7ffc-43c8-9381-b400cbbbf497/volumes"
Nov 02 03:35:35.630162 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:35.630127    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-abc348-h65xh"]
Nov 02 03:35:35.630951 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:35:35.630921    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63fa6a92_0860_40ca_98eb_df5946bfeaf4.slice/crio-4e457b2a2a971b71843bbbef9ae5eb3e3440ad77febbe968264b30aa86b223a5 WatchSource:0}: Error finding container 4e457b2a2a971b71843bbbef9ae5eb3e3440ad77febbe968264b30aa86b223a5: Status 404 returned error can't find the container with id 4e457b2a2a971b71843bbbef9ae5eb3e3440ad77febbe968264b30aa86b223a5
Nov 02 03:35:35.702373 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:35.702337    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-abc348-h65xh" event={"ID":"63fa6a92-0860-40ca-98eb-df5946bfeaf4","Type":"ContainerStarted","Data":"4e457b2a2a971b71843bbbef9ae5eb3e3440ad77febbe968264b30aa86b223a5"}
Nov 02 03:35:36.711453 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:36.711419    4496 generic.go:356] "Generic (PLEG): container finished" podID="63fa6a92-0860-40ca-98eb-df5946bfeaf4" containerID="ddd693c34dcbe085cff011ef95c7b4c7dcda920477fb97210bd7342d9e182588" exitCode=0
Nov 02 03:35:36.711857 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:36.711506    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-abc348-h65xh" event={"ID":"63fa6a92-0860-40ca-98eb-df5946bfeaf4","Type":"ContainerDied","Data":"ddd693c34dcbe085cff011ef95c7b4c7dcda920477fb97210bd7342d9e182588"}
Nov 02 03:35:37.887874 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:37.887849    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-abc348-h65xh"
Nov 02 03:35:37.991907 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:37.991877    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63fa6a92-0860-40ca-98eb-df5946bfeaf4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63fa6a92-0860-40ca-98eb-df5946bfeaf4-maas-api-xks-ca\") pod \"63fa6a92-0860-40ca-98eb-df5946bfeaf4\" (UID: \"63fa6a92-0860-40ca-98eb-df5946bfeaf4\") "
Nov 02 03:35:37.992166 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:37.991959    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63fa6a92-0860-40ca-98eb-df5946bfeaf4-kube-api-access-hpdb2\" (UniqueName: \"kubernetes.io/projected/63fa6a92-0860-40ca-98eb-df5946bfeaf4-kube-api-access-hpdb2\") pod \"63fa6a92-0860-40ca-98eb-df5946bfeaf4\" (UID: \"63fa6a92-0860-40ca-98eb-df5946bfeaf4\") "
Nov 02 03:35:37.992166 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:37.992072    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63fa6a92-0860-40ca-98eb-df5946bfeaf4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63fa6a92-0860-40ca-98eb-df5946bfeaf4-maas-api-openshift-ca\") pod \"63fa6a92-0860-40ca-98eb-df5946bfeaf4\" (UID: \"63fa6a92-0860-40ca-98eb-df5946bfeaf4\") "
Nov 02 03:35:37.992473 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:37.992446    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63fa6a92-0860-40ca-98eb-df5946bfeaf4-maas-api-openshift-ca" pod "63fa6a92-0860-40ca-98eb-df5946bfeaf4" (UID: "63fa6a92-0860-40ca-98eb-df5946bfeaf4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:35:37.992703 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:37.992680    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63fa6a92-0860-40ca-98eb-df5946bfeaf4-maas-api-openshift-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:35:37.994149 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:37.994118    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63fa6a92-0860-40ca-98eb-df5946bfeaf4-maas-api-xks-ca" pod "63fa6a92-0860-40ca-98eb-df5946bfeaf4" (UID: "63fa6a92-0860-40ca-98eb-df5946bfeaf4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 03:35:37.994240 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:37.994148    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63fa6a92-0860-40ca-98eb-df5946bfeaf4-kube-api-access-hpdb2" pod "63fa6a92-0860-40ca-98eb-df5946bfeaf4" (UID: "63fa6a92-0860-40ca-98eb-df5946bfeaf4"). InnerVolumeSpecName "kube-api-access-hpdb2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:35:38.093594 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:38.093558    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63fa6a92-0860-40ca-98eb-df5946bfeaf4-maas-api-xks-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:35:38.093594 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:38.093590    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpdb2\" (UniqueName: \"kubernetes.io/projected/63fa6a92-0860-40ca-98eb-df5946bfeaf4-kube-api-access-hpdb2\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:35:38.728262 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:38.728213    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-abc348-h65xh" event={"ID":"63fa6a92-0860-40ca-98eb-df5946bfeaf4","Type":"ContainerDied","Data":"4e457b2a2a971b71843bbbef9ae5eb3e3440ad77febbe968264b30aa86b223a5"}
Nov 02 03:35:38.728581 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:38.728334    4496 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4e457b2a2a971b71843bbbef9ae5eb3e3440ad77febbe968264b30aa86b223a5"
Nov 02 03:35:38.728581 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:38.728435    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-abc348-h65xh"
Nov 02 03:35:42.730998 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:42.730962    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-e44998-openshift-default-6bfcf667b5-mwgmh"]
Nov 02 03:35:42.731855 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:42.731830    4496 cpu_manager.go:414] "containerMap: removing container" podUID="63fa6a92-0860-40ca-98eb-df5946bfeaf4" containerName="revoke-keys"
Nov 02 03:35:42.731965 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:42.731856    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63fa6a92-0860-40ca-98eb-df5946bfeaf4" containerName="revoke-keys"
Nov 02 03:35:42.732070 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:42.732000    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="63fa6a92-0860-40ca-98eb-df5946bfeaf4" containerName="revoke-keys"
Nov 02 03:35:42.736667 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:42.736646    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-e44998-openshift-default-6bfcf667b5-mwgmh"
Nov 02 03:35:42.839439 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:42.839397    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-e44998-openshift-default-6bfcf667b5-mwgmh"]
Nov 02 03:35:42.854638 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:42.854604    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c01165ba-6eca-4e2a-81b1-c743acb0fbd6-istiod-ca-cert\") pod \"e2e-praxis-a-e44998-openshift-default-6bfcf667b5-mwgmh\" (UID: \"c01165ba-6eca-4e2a-81b1-c743acb0fbd6\") " pod="openshift-ingress/e2e-praxis-a-e44998-openshift-default-6bfcf667b5-mwgmh"
Nov 02 03:35:42.854836 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:42.854744    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c01165ba-6eca-4e2a-81b1-c743acb0fbd6-workload-certs\") pod \"e2e-praxis-a-e44998-openshift-default-6bfcf667b5-mwgmh\" (UID: \"c01165ba-6eca-4e2a-81b1-c743acb0fbd6\") " pod="openshift-ingress/e2e-praxis-a-e44998-openshift-default-6bfcf667b5-mwgmh"
Nov 02 03:35:42.854836 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:42.854788    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c01165ba-6eca-4e2a-81b1-c743acb0fbd6-workload-socket\") pod \"e2e-praxis-a-e44998-openshift-default-6bfcf667b5-mwgmh\" (UID: \"c01165ba-6eca-4e2a-81b1-c743acb0fbd6\") " pod="openshift-ingress/e2e-praxis-a-e44998-openshift-default-6bfcf667b5-mwgmh"
Nov 02 03:35:42.854984 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:42.854859    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c01165ba-6eca-4e2a-81b1-c743acb0fbd6-istio-token\") pod \"e2e-praxis-a-e44998-openshift-default-6bfcf667b5-mwgmh\" (UID: \"c01165ba-6eca-4e2a-81b1-c743acb0fbd6\") " pod="openshift-ingress/e2e-praxis-a-e44998-openshift-default-6bfcf667b5-mwgmh"
Nov 02 03:35:42.854984 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:42.854921    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c01165ba-6eca-4e2a-81b1-c743acb0fbd6-istio-envoy\") pod \"e2e-praxis-a-e44998-openshift-default-6bfcf667b5-mwgmh\" (UID: \"c01165ba-6eca-4e2a-81b1-c743acb0fbd6\") " pod="openshift-ingress/e2e-praxis-a-e44998-openshift-default-6bfcf667b5-mwgmh"
Nov 02 03:35:42.854984 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:42.854956    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c01165ba-6eca-4e2a-81b1-c743acb0fbd6-credential-socket\") pod \"e2e-praxis-a-e44998-openshift-default-6bfcf667b5-mwgmh\" (UID: \"c01165ba-6eca-4e2a-81b1-c743acb0fbd6\") " pod="openshift-ingress/e2e-praxis-a-e44998-openshift-default-6bfcf667b5-mwgmh"
Nov 02 03:35:42.854984 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:42.854977    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7pk4\" (UniqueName: \"kubernetes.io/projected/c01165ba-6eca-4e2a-81b1-c743acb0fbd6-kube-api-access-l7pk4\") pod \"e2e-praxis-a-e44998-openshift-default-6bfcf667b5-mwgmh\" (UID: \"c01165ba-6eca-4e2a-81b1-c743acb0fbd6\") " pod="openshift-ingress/e2e-praxis-a-e44998-openshift-default-6bfcf667b5-mwgmh"
Nov 02 03:35:42.855329 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:42.854993    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c01165ba-6eca-4e2a-81b1-c743acb0fbd6-istio-data\") pod \"e2e-praxis-a-e44998-openshift-default-6bfcf667b5-mwgmh\" (UID: \"c01165ba-6eca-4e2a-81b1-c743acb0fbd6\") " pod="openshift-ingress/e2e-praxis-a-e44998-openshift-default-6bfcf667b5-mwgmh"
Nov 02 03:35:42.855329 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:42.855062    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c01165ba-6eca-4e2a-81b1-c743acb0fbd6-istio-podinfo\") pod \"e2e-praxis-a-e44998-openshift-default-6bfcf667b5-mwgmh\" (UID: \"c01165ba-6eca-4e2a-81b1-c743acb0fbd6\") " pod="openshift-ingress/e2e-praxis-a-e44998-openshift-default-6bfcf667b5-mwgmh"
Nov 02 03:35:42.864236 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:42.864204    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-e44998-openshift-default-dockercfg-rjbvx\""
Nov 02 03:35:42.956898 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:42.956859    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c01165ba-6eca-4e2a-81b1-c743acb0fbd6-credential-socket\") pod \"e2e-praxis-a-e44998-openshift-default-6bfcf667b5-mwgmh\" (UID: \"c01165ba-6eca-4e2a-81b1-c743acb0fbd6\") " pod="openshift-ingress/e2e-praxis-a-e44998-openshift-default-6bfcf667b5-mwgmh"
Nov 02 03:35:42.957146 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:42.956911    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7pk4\" (UniqueName: \"kubernetes.io/projected/c01165ba-6eca-4e2a-81b1-c743acb0fbd6-kube-api-access-l7pk4\") pod \"e2e-praxis-a-e44998-openshift-default-6bfcf667b5-mwgmh\" (UID: \"c01165ba-6eca-4e2a-81b1-c743acb0fbd6\") " pod="openshift-ingress/e2e-praxis-a-e44998-openshift-default-6bfcf667b5-mwgmh"
Nov 02 03:35:42.957146 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:42.956942    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c01165ba-6eca-4e2a-81b1-c743acb0fbd6-istio-data\") pod \"e2e-praxis-a-e44998-openshift-default-6bfcf667b5-mwgmh\" (UID: \"c01165ba-6eca-4e2a-81b1-c743acb0fbd6\") " pod="openshift-ingress/e2e-praxis-a-e44998-openshift-default-6bfcf667b5-mwgmh"
Nov 02 03:35:42.957146 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:42.957010    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c01165ba-6eca-4e2a-81b1-c743acb0fbd6-istio-podinfo\") pod \"e2e-praxis-a-e44998-openshift-default-6bfcf667b5-mwgmh\" (UID: \"c01165ba-6eca-4e2a-81b1-c743acb0fbd6\") " pod="openshift-ingress/e2e-praxis-a-e44998-openshift-default-6bfcf667b5-mwgmh"
Nov 02 03:35:42.957146 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:42.957056    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c01165ba-6eca-4e2a-81b1-c743acb0fbd6-istiod-ca-cert\") pod \"e2e-praxis-a-e44998-openshift-default-6bfcf667b5-mwgmh\" (UID: \"c01165ba-6eca-4e2a-81b1-c743acb0fbd6\") " pod="openshift-ingress/e2e-praxis-a-e44998-openshift-default-6bfcf667b5-mwgmh"
Nov 02 03:35:42.957392 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:42.957167    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c01165ba-6eca-4e2a-81b1-c743acb0fbd6-workload-certs\") pod \"e2e-praxis-a-e44998-openshift-default-6bfcf667b5-mwgmh\" (UID: \"c01165ba-6eca-4e2a-81b1-c743acb0fbd6\") " pod="openshift-ingress/e2e-praxis-a-e44998-openshift-default-6bfcf667b5-mwgmh"
Nov 02 03:35:42.957392 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:42.957200    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c01165ba-6eca-4e2a-81b1-c743acb0fbd6-workload-socket\") pod \"e2e-praxis-a-e44998-openshift-default-6bfcf667b5-mwgmh\" (UID: \"c01165ba-6eca-4e2a-81b1-c743acb0fbd6\") " pod="openshift-ingress/e2e-praxis-a-e44998-openshift-default-6bfcf667b5-mwgmh"
Nov 02 03:35:42.957392 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:42.957269    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c01165ba-6eca-4e2a-81b1-c743acb0fbd6-istio-token\") pod \"e2e-praxis-a-e44998-openshift-default-6bfcf667b5-mwgmh\" (UID: \"c01165ba-6eca-4e2a-81b1-c743acb0fbd6\") " pod="openshift-ingress/e2e-praxis-a-e44998-openshift-default-6bfcf667b5-mwgmh"
Nov 02 03:35:42.957392 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:42.957319    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c01165ba-6eca-4e2a-81b1-c743acb0fbd6-istio-envoy\") pod \"e2e-praxis-a-e44998-openshift-default-6bfcf667b5-mwgmh\" (UID: \"c01165ba-6eca-4e2a-81b1-c743acb0fbd6\") " pod="openshift-ingress/e2e-praxis-a-e44998-openshift-default-6bfcf667b5-mwgmh"
Nov 02 03:35:42.957392 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:42.957340    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c01165ba-6eca-4e2a-81b1-c743acb0fbd6-istio-data\") pod \"e2e-praxis-a-e44998-openshift-default-6bfcf667b5-mwgmh\" (UID: \"c01165ba-6eca-4e2a-81b1-c743acb0fbd6\") " pod="openshift-ingress/e2e-praxis-a-e44998-openshift-default-6bfcf667b5-mwgmh"
Nov 02 03:35:42.957716 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:42.957592    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c01165ba-6eca-4e2a-81b1-c743acb0fbd6-workload-certs\") pod \"e2e-praxis-a-e44998-openshift-default-6bfcf667b5-mwgmh\" (UID: \"c01165ba-6eca-4e2a-81b1-c743acb0fbd6\") " pod="openshift-ingress/e2e-praxis-a-e44998-openshift-default-6bfcf667b5-mwgmh"
Nov 02 03:35:42.957716 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:42.957340    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c01165ba-6eca-4e2a-81b1-c743acb0fbd6-credential-socket\") pod \"e2e-praxis-a-e44998-openshift-default-6bfcf667b5-mwgmh\" (UID: \"c01165ba-6eca-4e2a-81b1-c743acb0fbd6\") " pod="openshift-ingress/e2e-praxis-a-e44998-openshift-default-6bfcf667b5-mwgmh"
Nov 02 03:35:42.957716 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:42.957629    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c01165ba-6eca-4e2a-81b1-c743acb0fbd6-workload-socket\") pod \"e2e-praxis-a-e44998-openshift-default-6bfcf667b5-mwgmh\" (UID: \"c01165ba-6eca-4e2a-81b1-c743acb0fbd6\") " pod="openshift-ingress/e2e-praxis-a-e44998-openshift-default-6bfcf667b5-mwgmh"
Nov 02 03:35:42.958152 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:42.958122    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c01165ba-6eca-4e2a-81b1-c743acb0fbd6-istiod-ca-cert\") pod \"e2e-praxis-a-e44998-openshift-default-6bfcf667b5-mwgmh\" (UID: \"c01165ba-6eca-4e2a-81b1-c743acb0fbd6\") " pod="openshift-ingress/e2e-praxis-a-e44998-openshift-default-6bfcf667b5-mwgmh"
Nov 02 03:35:42.960426 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:42.960377    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c01165ba-6eca-4e2a-81b1-c743acb0fbd6-istio-envoy\") pod \"e2e-praxis-a-e44998-openshift-default-6bfcf667b5-mwgmh\" (UID: \"c01165ba-6eca-4e2a-81b1-c743acb0fbd6\") " pod="openshift-ingress/e2e-praxis-a-e44998-openshift-default-6bfcf667b5-mwgmh"
Nov 02 03:35:42.960926 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:42.960906    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c01165ba-6eca-4e2a-81b1-c743acb0fbd6-istio-podinfo\") pod \"e2e-praxis-a-e44998-openshift-default-6bfcf667b5-mwgmh\" (UID: \"c01165ba-6eca-4e2a-81b1-c743acb0fbd6\") " pod="openshift-ingress/e2e-praxis-a-e44998-openshift-default-6bfcf667b5-mwgmh"
Nov 02 03:35:43.015609 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:43.015535    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c01165ba-6eca-4e2a-81b1-c743acb0fbd6-istio-token\") pod \"e2e-praxis-a-e44998-openshift-default-6bfcf667b5-mwgmh\" (UID: \"c01165ba-6eca-4e2a-81b1-c743acb0fbd6\") " pod="openshift-ingress/e2e-praxis-a-e44998-openshift-default-6bfcf667b5-mwgmh"
Nov 02 03:35:43.015726 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:43.015708    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7pk4\" (UniqueName: \"kubernetes.io/projected/c01165ba-6eca-4e2a-81b1-c743acb0fbd6-kube-api-access-l7pk4\") pod \"e2e-praxis-a-e44998-openshift-default-6bfcf667b5-mwgmh\" (UID: \"c01165ba-6eca-4e2a-81b1-c743acb0fbd6\") " pod="openshift-ingress/e2e-praxis-a-e44998-openshift-default-6bfcf667b5-mwgmh"
Nov 02 03:35:43.050937 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:43.050868    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-e44998-openshift-default-6bfcf667b5-mwgmh"
Nov 02 03:35:43.481201 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:35:43.481170    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc01165ba_6eca_4e2a_81b1_c743acb0fbd6.slice/crio-413e111269e470e1c0006a4461fbe9cefc7bf7312d9c3ef63f66f440a79c8ed7 WatchSource:0}: Error finding container 413e111269e470e1c0006a4461fbe9cefc7bf7312d9c3ef63f66f440a79c8ed7: Status 404 returned error can't find the container with id 413e111269e470e1c0006a4461fbe9cefc7bf7312d9c3ef63f66f440a79c8ed7
Nov 02 03:35:43.483261 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:43.483233    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-e44998-openshift-default-6bfcf667b5-mwgmh"]
Nov 02 03:35:43.483725 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:43.483694    4496 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 03:35:43.483788 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:43.483775    4496 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 03:35:43.483843 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:43.483806    4496 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 03:35:43.764149 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:43.764057    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-e44998-openshift-default-6bfcf667b5-mwgmh" event={"ID":"c01165ba-6eca-4e2a-81b1-c743acb0fbd6","Type":"ContainerStarted","Data":"534e1c32f59035b693638a67a1c0bab60c38914d4ffe15952783556e06b7fdb4"}
Nov 02 03:35:43.764149 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:43.764111    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-e44998-openshift-default-6bfcf667b5-mwgmh" event={"ID":"c01165ba-6eca-4e2a-81b1-c743acb0fbd6","Type":"ContainerStarted","Data":"413e111269e470e1c0006a4461fbe9cefc7bf7312d9c3ef63f66f440a79c8ed7"}
Nov 02 03:35:43.870007 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:43.869976    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13a4b385-pnc2c"]
Nov 02 03:35:43.874536 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:43.874517    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13a4b385-pnc2c"
Nov 02 03:35:43.881590 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:43.881555    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-e44998-openshift-default-6bfcf667b5-mwgmh" podStartSLOduration=1.881543532 podStartE2EDuration="1.881543532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:35:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:35:43.881168173 +0000 UTC m=+2884.459775553" watchObservedRunningTime="2026-11-02 03:35:43.881543532 +0000 UTC m=+2884.460150906"
Nov 02 03:35:43.927664 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:43.927626    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 03:35:43.927664 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:43.927659    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 03:35:43.927944 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:43.927637    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 03:35:43.927944 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:43.927708    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9z5bk\""
Nov 02 03:35:43.930437 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:43.930406    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13a4b385-pnc2c"]
Nov 02 03:35:43.972230 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:43.972200    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a2685ed-24a0-4ac6-838f-d47c7441f9a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13a4b385-pnc2c\" (UID: \"9a2685ed-24a0-4ac6-838f-d47c7441f9a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13a4b385-pnc2c"
Nov 02 03:35:43.972378 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:43.972328    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a2685ed-24a0-4ac6-838f-d47c7441f9a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13a4b385-pnc2c\" (UID: \"9a2685ed-24a0-4ac6-838f-d47c7441f9a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13a4b385-pnc2c"
Nov 02 03:35:43.972378 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:43.972366    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfcsl\" (UniqueName: \"kubernetes.io/projected/9a2685ed-24a0-4ac6-838f-d47c7441f9a2-kube-api-access-sfcsl\") pod \"maas-api-revoke-sub-13a4b385-pnc2c\" (UID: \"9a2685ed-24a0-4ac6-838f-d47c7441f9a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13a4b385-pnc2c"
Nov 02 03:35:44.052179 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:44.052048    4496 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-e44998-openshift-default-6bfcf667b5-mwgmh"
Nov 02 03:35:44.058636 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:44.058604    4496 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-e44998-openshift-default-6bfcf667b5-mwgmh"
Nov 02 03:35:44.073115 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:44.073089    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a2685ed-24a0-4ac6-838f-d47c7441f9a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13a4b385-pnc2c\" (UID: \"9a2685ed-24a0-4ac6-838f-d47c7441f9a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13a4b385-pnc2c"
Nov 02 03:35:44.073249 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:44.073139    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfcsl\" (UniqueName: \"kubernetes.io/projected/9a2685ed-24a0-4ac6-838f-d47c7441f9a2-kube-api-access-sfcsl\") pod \"maas-api-revoke-sub-13a4b385-pnc2c\" (UID: \"9a2685ed-24a0-4ac6-838f-d47c7441f9a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13a4b385-pnc2c"
Nov 02 03:35:44.073292 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:44.073273    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a2685ed-24a0-4ac6-838f-d47c7441f9a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13a4b385-pnc2c\" (UID: \"9a2685ed-24a0-4ac6-838f-d47c7441f9a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13a4b385-pnc2c"
Nov 02 03:35:44.073903 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:44.073879    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a2685ed-24a0-4ac6-838f-d47c7441f9a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13a4b385-pnc2c\" (UID: \"9a2685ed-24a0-4ac6-838f-d47c7441f9a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13a4b385-pnc2c"
Nov 02 03:35:44.076121 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:44.076097    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a2685ed-24a0-4ac6-838f-d47c7441f9a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13a4b385-pnc2c\" (UID: \"9a2685ed-24a0-4ac6-838f-d47c7441f9a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13a4b385-pnc2c"
Nov 02 03:35:44.130522 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:44.130495    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfcsl\" (UniqueName: \"kubernetes.io/projected/9a2685ed-24a0-4ac6-838f-d47c7441f9a2-kube-api-access-sfcsl\") pod \"maas-api-revoke-sub-13a4b385-pnc2c\" (UID: \"9a2685ed-24a0-4ac6-838f-d47c7441f9a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13a4b385-pnc2c"
Nov 02 03:35:44.201238 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:44.201198    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13a4b385-pnc2c"
Nov 02 03:35:44.583867 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:44.583842    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13a4b385-pnc2c"]
Nov 02 03:35:44.584230 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:35:44.584206    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a2685ed_24a0_4ac6_838f_d47c7441f9a2.slice/crio-3ec1515062fce888609b1c0b6f63518a63cdf963047a34871a771f2548667434 WatchSource:0}: Error finding container 3ec1515062fce888609b1c0b6f63518a63cdf963047a34871a771f2548667434: Status 404 returned error can't find the container with id 3ec1515062fce888609b1c0b6f63518a63cdf963047a34871a771f2548667434
Nov 02 03:35:44.774451 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:44.774416    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13a4b385-pnc2c" event={"ID":"9a2685ed-24a0-4ac6-838f-d47c7441f9a2","Type":"ContainerStarted","Data":"3ec1515062fce888609b1c0b6f63518a63cdf963047a34871a771f2548667434"}
Nov 02 03:35:44.774873 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:44.774847    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-e44998-openshift-default-6bfcf667b5-mwgmh"
Nov 02 03:35:44.775870 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:44.775838    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-e44998-openshift-default-6bfcf667b5-mwgmh"
Nov 02 03:35:45.783102 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:45.783069    4496 generic.go:356] "Generic (PLEG): container finished" podID="9a2685ed-24a0-4ac6-838f-d47c7441f9a2" containerID="f00177b934ebd86616420812c5424536829b95ea4a321fae58813671a79f51d2" exitCode=0
Nov 02 03:35:45.783518 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:45.783159    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13a4b385-pnc2c" event={"ID":"9a2685ed-24a0-4ac6-838f-d47c7441f9a2","Type":"ContainerDied","Data":"f00177b934ebd86616420812c5424536829b95ea4a321fae58813671a79f51d2"}
Nov 02 03:35:46.943422 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:46.943395    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13a4b385-pnc2c"
Nov 02 03:35:47.008009 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:47.007979    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a2685ed-24a0-4ac6-838f-d47c7441f9a2-kube-api-access-sfcsl\" (UniqueName: \"kubernetes.io/projected/9a2685ed-24a0-4ac6-838f-d47c7441f9a2-kube-api-access-sfcsl\") pod \"9a2685ed-24a0-4ac6-838f-d47c7441f9a2\" (UID: \"9a2685ed-24a0-4ac6-838f-d47c7441f9a2\") "
Nov 02 03:35:47.008191 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:47.008127    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a2685ed-24a0-4ac6-838f-d47c7441f9a2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a2685ed-24a0-4ac6-838f-d47c7441f9a2-maas-api-openshift-ca\") pod \"9a2685ed-24a0-4ac6-838f-d47c7441f9a2\" (UID: \"9a2685ed-24a0-4ac6-838f-d47c7441f9a2\") "
Nov 02 03:35:47.008258 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:47.008244    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a2685ed-24a0-4ac6-838f-d47c7441f9a2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a2685ed-24a0-4ac6-838f-d47c7441f9a2-maas-api-xks-ca\") pod \"9a2685ed-24a0-4ac6-838f-d47c7441f9a2\" (UID: \"9a2685ed-24a0-4ac6-838f-d47c7441f9a2\") "
Nov 02 03:35:47.008484 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:47.008458    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a2685ed-24a0-4ac6-838f-d47c7441f9a2-maas-api-openshift-ca" pod "9a2685ed-24a0-4ac6-838f-d47c7441f9a2" (UID: "9a2685ed-24a0-4ac6-838f-d47c7441f9a2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:35:47.008600 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:47.008588    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a2685ed-24a0-4ac6-838f-d47c7441f9a2-maas-api-openshift-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:35:47.010335 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:47.010306    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a2685ed-24a0-4ac6-838f-d47c7441f9a2-kube-api-access-sfcsl" pod "9a2685ed-24a0-4ac6-838f-d47c7441f9a2" (UID: "9a2685ed-24a0-4ac6-838f-d47c7441f9a2"). InnerVolumeSpecName "kube-api-access-sfcsl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:35:47.010439 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:47.010368    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a2685ed-24a0-4ac6-838f-d47c7441f9a2-maas-api-xks-ca" pod "9a2685ed-24a0-4ac6-838f-d47c7441f9a2" (UID: "9a2685ed-24a0-4ac6-838f-d47c7441f9a2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 03:35:47.109927 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:47.109851    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a2685ed-24a0-4ac6-838f-d47c7441f9a2-maas-api-xks-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:35:47.109927 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:47.109880    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfcsl\" (UniqueName: \"kubernetes.io/projected/9a2685ed-24a0-4ac6-838f-d47c7441f9a2-kube-api-access-sfcsl\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:35:47.799880 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:47.799848    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13a4b385-pnc2c"
Nov 02 03:35:47.800066 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:47.799886    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13a4b385-pnc2c" event={"ID":"9a2685ed-24a0-4ac6-838f-d47c7441f9a2","Type":"ContainerDied","Data":"3ec1515062fce888609b1c0b6f63518a63cdf963047a34871a771f2548667434"}
Nov 02 03:35:47.800066 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:47.799935    4496 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3ec1515062fce888609b1c0b6f63518a63cdf963047a34871a771f2548667434"
Nov 02 03:35:50.367406 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:50.367361    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-e44998-764fd4f68c-5ggnl"]
Nov 02 03:35:50.368080 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:50.368061    4496 cpu_manager.go:414] "containerMap: removing container" podUID="9a2685ed-24a0-4ac6-838f-d47c7441f9a2" containerName="revoke-keys"
Nov 02 03:35:50.368080 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:50.368080    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a2685ed-24a0-4ac6-838f-d47c7441f9a2" containerName="revoke-keys"
Nov 02 03:35:50.368231 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:50.368201    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a2685ed-24a0-4ac6-838f-d47c7441f9a2" containerName="revoke-keys"
Nov 02 03:35:50.371960 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:50.371935    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e44998-764fd4f68c-5ggnl"
Nov 02 03:35:50.428402 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:50.428368    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-e44998-764fd4f68c-5ggnl"]
Nov 02 03:35:50.474726 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:50.474667    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-e44998-dockercfg-7r7m9\""
Nov 02 03:35:50.475048 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:50.474996    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-e44998\""
Nov 02 03:35:50.476342 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:50.476295    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea831751-f990-46e0-bd3d-49b395a46fdd-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-e44998-764fd4f68c-5ggnl\" (UID: \"ea831751-f990-46e0-bd3d-49b395a46fdd\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e44998-764fd4f68c-5ggnl"
Nov 02 03:35:50.476914 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:50.476892    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f88kn\" (UniqueName: \"kubernetes.io/projected/ea831751-f990-46e0-bd3d-49b395a46fdd-kube-api-access-f88kn\") pod \"payload-pre-processing-e2e-praxis-a-e44998-764fd4f68c-5ggnl\" (UID: \"ea831751-f990-46e0-bd3d-49b395a46fdd\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e44998-764fd4f68c-5ggnl"
Nov 02 03:35:50.579476 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:50.579433    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea831751-f990-46e0-bd3d-49b395a46fdd-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-e44998-764fd4f68c-5ggnl\" (UID: \"ea831751-f990-46e0-bd3d-49b395a46fdd\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e44998-764fd4f68c-5ggnl"
Nov 02 03:35:50.579476 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:50.579480    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f88kn\" (UniqueName: \"kubernetes.io/projected/ea831751-f990-46e0-bd3d-49b395a46fdd-kube-api-access-f88kn\") pod \"payload-pre-processing-e2e-praxis-a-e44998-764fd4f68c-5ggnl\" (UID: \"ea831751-f990-46e0-bd3d-49b395a46fdd\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e44998-764fd4f68c-5ggnl"
Nov 02 03:35:50.580251 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:50.580228    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea831751-f990-46e0-bd3d-49b395a46fdd-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-e44998-764fd4f68c-5ggnl\" (UID: \"ea831751-f990-46e0-bd3d-49b395a46fdd\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e44998-764fd4f68c-5ggnl"
Nov 02 03:35:50.637220 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:50.637132    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f88kn\" (UniqueName: \"kubernetes.io/projected/ea831751-f990-46e0-bd3d-49b395a46fdd-kube-api-access-f88kn\") pod \"payload-pre-processing-e2e-praxis-a-e44998-764fd4f68c-5ggnl\" (UID: \"ea831751-f990-46e0-bd3d-49b395a46fdd\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e44998-764fd4f68c-5ggnl"
Nov 02 03:35:50.687975 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:50.687935    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e44998-764fd4f68c-5ggnl"
Nov 02 03:35:51.084462 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:35:51.084426    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea831751_f990_46e0_bd3d_49b395a46fdd.slice/crio-d35ed65af162987ec1a152c8c1c14e5b3cae00dc87fb4c217bd54c579d324208 WatchSource:0}: Error finding container d35ed65af162987ec1a152c8c1c14e5b3cae00dc87fb4c217bd54c579d324208: Status 404 returned error can't find the container with id d35ed65af162987ec1a152c8c1c14e5b3cae00dc87fb4c217bd54c579d324208
Nov 02 03:35:51.099894 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:51.099864    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-e44998-764fd4f68c-5ggnl"]
Nov 02 03:35:51.374749 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:51.374645    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f1a692-zdkxn"]
Nov 02 03:35:51.380089 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:51.379988    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f1a692-zdkxn"]
Nov 02 03:35:51.575534 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:51.575499    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9049d7e3-323a-4d1d-90db-79bdbc2c6c68" path="/var/lib/kubelet/pods/9049d7e3-323a-4d1d-90db-79bdbc2c6c68/volumes"
Nov 02 03:35:51.832985 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:51.832950    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e44998-764fd4f68c-5ggnl" event={"ID":"ea831751-f990-46e0-bd3d-49b395a46fdd","Type":"ContainerStarted","Data":"604c1ff9ede27eb83012e738cfc4905a715cd0870d9773a5d25dc1acfd79e695"}
Nov 02 03:35:51.833210 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:51.833010    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e44998-764fd4f68c-5ggnl" event={"ID":"ea831751-f990-46e0-bd3d-49b395a46fdd","Type":"ContainerStarted","Data":"d35ed65af162987ec1a152c8c1c14e5b3cae00dc87fb4c217bd54c579d324208"}
Nov 02 03:35:51.833210 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:51.833055    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e44998-764fd4f68c-5ggnl"
Nov 02 03:35:51.951656 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:51.951588    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e44998-764fd4f68c-5ggnl" podStartSLOduration=1.951570318 podStartE2EDuration="1.951570318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:35:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:35:51.948168918 +0000 UTC m=+2892.526776300" watchObservedRunningTime="2026-11-02 03:35:51.951570318 +0000 UTC m=+2892.530177700"
Nov 02 03:35:58.081864 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:58.081825    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-abc348-b97879cd4-b9nkq"]
Nov 02 03:35:58.082395 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:58.082087    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-abc348-b97879cd4-b9nkq" podUID="f8237198-5111-435d-a8d8-237ab0bbc48e" containerName="payload-processing" containerID="cri-o://a00df95dee4852332e6fe55c2734e2cbe2ee04ef44aee1e01a6ace8016386136" gracePeriod=30
Nov 02 03:35:58.139390 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:58.139354    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-abc348-6fbfd887f5-4zkk2"]
Nov 02 03:35:58.139761 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:58.139729    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-abc348-6fbfd887f5-4zkk2" podUID="90ea88ed-de1d-490d-8b24-03dcdacca466" containerName="payload-pre-processing" containerID="cri-o://30c1b987bfbbc906f7d3360824c039251075387a9770d8f2bb22287e9824c8c9" gracePeriod=30
Nov 02 03:35:58.457068 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:58.457041    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-abc348-b97879cd4-b9nkq"
Nov 02 03:35:58.479146 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:58.479125    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-abc348-6fbfd887f5-4zkk2"
Nov 02 03:35:58.493546 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:58.493516    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8237198-5111-435d-a8d8-237ab0bbc48e-kube-api-access-rrf2m\" (UniqueName: \"kubernetes.io/projected/f8237198-5111-435d-a8d8-237ab0bbc48e-kube-api-access-rrf2m\") pod \"f8237198-5111-435d-a8d8-237ab0bbc48e\" (UID: \"f8237198-5111-435d-a8d8-237ab0bbc48e\") "
Nov 02 03:35:58.493653 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:58.493573    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8237198-5111-435d-a8d8-237ab0bbc48e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8237198-5111-435d-a8d8-237ab0bbc48e-plugins-config-volume\") pod \"f8237198-5111-435d-a8d8-237ab0bbc48e\" (UID: \"f8237198-5111-435d-a8d8-237ab0bbc48e\") "
Nov 02 03:35:58.493977 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:58.493956    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8237198-5111-435d-a8d8-237ab0bbc48e-plugins-config-volume" pod "f8237198-5111-435d-a8d8-237ab0bbc48e" (UID: "f8237198-5111-435d-a8d8-237ab0bbc48e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:35:58.495568 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:58.495548    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8237198-5111-435d-a8d8-237ab0bbc48e-kube-api-access-rrf2m" pod "f8237198-5111-435d-a8d8-237ab0bbc48e" (UID: "f8237198-5111-435d-a8d8-237ab0bbc48e"). InnerVolumeSpecName "kube-api-access-rrf2m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:35:58.595341 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:58.595306    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90ea88ed-de1d-490d-8b24-03dcdacca466-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90ea88ed-de1d-490d-8b24-03dcdacca466-plugins-config-volume\") pod \"90ea88ed-de1d-490d-8b24-03dcdacca466\" (UID: \"90ea88ed-de1d-490d-8b24-03dcdacca466\") "
Nov 02 03:35:58.595530 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:58.595391    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90ea88ed-de1d-490d-8b24-03dcdacca466-kube-api-access-hwft2\" (UniqueName: \"kubernetes.io/projected/90ea88ed-de1d-490d-8b24-03dcdacca466-kube-api-access-hwft2\") pod \"90ea88ed-de1d-490d-8b24-03dcdacca466\" (UID: \"90ea88ed-de1d-490d-8b24-03dcdacca466\") "
Nov 02 03:35:58.595783 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:58.595753    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90ea88ed-de1d-490d-8b24-03dcdacca466-plugins-config-volume" pod "90ea88ed-de1d-490d-8b24-03dcdacca466" (UID: "90ea88ed-de1d-490d-8b24-03dcdacca466"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:35:58.595895 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:58.595788    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrf2m\" (UniqueName: \"kubernetes.io/projected/f8237198-5111-435d-a8d8-237ab0bbc48e-kube-api-access-rrf2m\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:35:58.595895 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:58.595815    4496 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8237198-5111-435d-a8d8-237ab0bbc48e-plugins-config-volume\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:35:58.597565 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:58.597501    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90ea88ed-de1d-490d-8b24-03dcdacca466-kube-api-access-hwft2" pod "90ea88ed-de1d-490d-8b24-03dcdacca466" (UID: "90ea88ed-de1d-490d-8b24-03dcdacca466"). InnerVolumeSpecName "kube-api-access-hwft2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:35:58.697641 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:58.697551    4496 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90ea88ed-de1d-490d-8b24-03dcdacca466-plugins-config-volume\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:35:58.697641 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:58.697587    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwft2\" (UniqueName: \"kubernetes.io/projected/90ea88ed-de1d-490d-8b24-03dcdacca466-kube-api-access-hwft2\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:35:58.893005 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:58.892968    4496 generic.go:356] "Generic (PLEG): container finished" podID="f8237198-5111-435d-a8d8-237ab0bbc48e" containerID="a00df95dee4852332e6fe55c2734e2cbe2ee04ef44aee1e01a6ace8016386136" exitCode=0
Nov 02 03:35:58.893224 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:58.893038    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-abc348-b97879cd4-b9nkq"
Nov 02 03:35:58.893224 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:58.893067    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-abc348-b97879cd4-b9nkq" event={"ID":"f8237198-5111-435d-a8d8-237ab0bbc48e","Type":"ContainerDied","Data":"a00df95dee4852332e6fe55c2734e2cbe2ee04ef44aee1e01a6ace8016386136"}
Nov 02 03:35:58.893224 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:58.893126    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-abc348-b97879cd4-b9nkq" event={"ID":"f8237198-5111-435d-a8d8-237ab0bbc48e","Type":"ContainerDied","Data":"feb22109ea8c0c2c21346981b682c6814962d60a8f031ba9bd404fad590ce50a"}
Nov 02 03:35:58.893224 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:58.893145    4496 scope.go:122] "RemoveContainer" containerID="a00df95dee4852332e6fe55c2734e2cbe2ee04ef44aee1e01a6ace8016386136"
Nov 02 03:35:58.894972 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:58.894950    4496 generic.go:356] "Generic (PLEG): container finished" podID="90ea88ed-de1d-490d-8b24-03dcdacca466" containerID="30c1b987bfbbc906f7d3360824c039251075387a9770d8f2bb22287e9824c8c9" exitCode=0
Nov 02 03:35:58.895099 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:58.894998    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-abc348-6fbfd887f5-4zkk2" event={"ID":"90ea88ed-de1d-490d-8b24-03dcdacca466","Type":"ContainerDied","Data":"30c1b987bfbbc906f7d3360824c039251075387a9770d8f2bb22287e9824c8c9"}
Nov 02 03:35:58.895099 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:58.895008    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-abc348-6fbfd887f5-4zkk2"
Nov 02 03:35:58.895099 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:58.895058    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-abc348-6fbfd887f5-4zkk2" event={"ID":"90ea88ed-de1d-490d-8b24-03dcdacca466","Type":"ContainerDied","Data":"b254748b83aac2f6ee1dbc13d6329788b8a79c42c49ba6cf3589046209e7029f"}
Nov 02 03:35:58.907685 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:58.907656    4496 scope.go:122] "RemoveContainer" containerID="a00df95dee4852332e6fe55c2734e2cbe2ee04ef44aee1e01a6ace8016386136"
Nov 02 03:35:58.907962 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:35:58.907943    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a00df95dee4852332e6fe55c2734e2cbe2ee04ef44aee1e01a6ace8016386136\": container with ID starting with a00df95dee4852332e6fe55c2734e2cbe2ee04ef44aee1e01a6ace8016386136 not found: ID does not exist" containerID="a00df95dee4852332e6fe55c2734e2cbe2ee04ef44aee1e01a6ace8016386136"
Nov 02 03:35:58.908052 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:58.907969    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a00df95dee4852332e6fe55c2734e2cbe2ee04ef44aee1e01a6ace8016386136"} err="failed to get container status \"a00df95dee4852332e6fe55c2734e2cbe2ee04ef44aee1e01a6ace8016386136\": rpc error: code = NotFound desc = could not find container \"a00df95dee4852332e6fe55c2734e2cbe2ee04ef44aee1e01a6ace8016386136\": container with ID starting with a00df95dee4852332e6fe55c2734e2cbe2ee04ef44aee1e01a6ace8016386136 not found: ID does not exist"
Nov 02 03:35:58.908052 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:58.907986    4496 scope.go:122] "RemoveContainer" containerID="30c1b987bfbbc906f7d3360824c039251075387a9770d8f2bb22287e9824c8c9"
Nov 02 03:35:58.918785 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:58.918763    4496 scope.go:122] "RemoveContainer" containerID="30c1b987bfbbc906f7d3360824c039251075387a9770d8f2bb22287e9824c8c9"
Nov 02 03:35:58.919156 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:35:58.919132    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30c1b987bfbbc906f7d3360824c039251075387a9770d8f2bb22287e9824c8c9\": container with ID starting with 30c1b987bfbbc906f7d3360824c039251075387a9770d8f2bb22287e9824c8c9 not found: ID does not exist" containerID="30c1b987bfbbc906f7d3360824c039251075387a9770d8f2bb22287e9824c8c9"
Nov 02 03:35:58.919247 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:58.919162    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30c1b987bfbbc906f7d3360824c039251075387a9770d8f2bb22287e9824c8c9"} err="failed to get container status \"30c1b987bfbbc906f7d3360824c039251075387a9770d8f2bb22287e9824c8c9\": rpc error: code = NotFound desc = could not find container \"30c1b987bfbbc906f7d3360824c039251075387a9770d8f2bb22287e9824c8c9\": container with ID starting with 30c1b987bfbbc906f7d3360824c039251075387a9770d8f2bb22287e9824c8c9 not found: ID does not exist"
Nov 02 03:35:59.060750 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:59.060714    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-abc348-6fbfd887f5-4zkk2"]
Nov 02 03:35:59.064170 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:59.064147    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-abc348-6fbfd887f5-4zkk2"]
Nov 02 03:35:59.175831 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:59.175794    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-abc348-b97879cd4-b9nkq"]
Nov 02 03:35:59.179325 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:59.179297    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-abc348-b97879cd4-b9nkq"]
Nov 02 03:35:59.576436 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:59.576397    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90ea88ed-de1d-490d-8b24-03dcdacca466" path="/var/lib/kubelet/pods/90ea88ed-de1d-490d-8b24-03dcdacca466/volumes"
Nov 02 03:35:59.576981 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:35:59.576963    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8237198-5111-435d-a8d8-237ab0bbc48e" path="/var/lib/kubelet/pods/f8237198-5111-435d-a8d8-237ab0bbc48e/volumes"
Nov 02 03:36:00.048273 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:00.048231    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-abc348-h65xh"]
Nov 02 03:36:00.096755 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:00.096711    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-abc348-h65xh"]
Nov 02 03:36:01.576940 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:01.576900    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63fa6a92-0860-40ca-98eb-df5946bfeaf4" path="/var/lib/kubelet/pods/63fa6a92-0860-40ca-98eb-df5946bfeaf4/volumes"
Nov 02 03:36:02.847579 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:02.847550    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e44998-764fd4f68c-5ggnl"
Nov 02 03:36:03.526065 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:36:03.526031    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 03:36:08.357212 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:08.357180    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0809c8-877wj"]
Nov 02 03:36:08.357832 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:08.357817    4496 cpu_manager.go:414] "containerMap: removing container" podUID="90ea88ed-de1d-490d-8b24-03dcdacca466" containerName="payload-pre-processing"
Nov 02 03:36:08.357883 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:08.357833    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90ea88ed-de1d-490d-8b24-03dcdacca466" containerName="payload-pre-processing"
Nov 02 03:36:08.357883 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:08.357851    4496 cpu_manager.go:414] "containerMap: removing container" podUID="f8237198-5111-435d-a8d8-237ab0bbc48e" containerName="payload-processing"
Nov 02 03:36:08.357883 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:08.357856    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8237198-5111-435d-a8d8-237ab0bbc48e" containerName="payload-processing"
Nov 02 03:36:08.357978 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:08.357953    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="90ea88ed-de1d-490d-8b24-03dcdacca466" containerName="payload-pre-processing"
Nov 02 03:36:08.357978 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:08.357963    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8237198-5111-435d-a8d8-237ab0bbc48e" containerName="payload-processing"
Nov 02 03:36:08.362945 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:08.362926    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0809c8-877wj"
Nov 02 03:36:08.418165 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:08.418135    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0809c8-877wj"]
Nov 02 03:36:08.464668 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:08.464608    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 03:36:08.464668 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:08.464608    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 03:36:08.464865 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:08.464676    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 03:36:08.468986 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:08.468961    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9z5bk\""
Nov 02 03:36:08.472707 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:08.472685    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-0a04e0-bc9cc6559-cvsjg"]
Nov 02 03:36:08.477091 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:08.477068    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0a04e0-bc9cc6559-cvsjg"
Nov 02 03:36:08.527295 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:08.523126    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cbd8a74-902a-45cf-b431-7e98f6ccd25c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-0809c8-877wj\" (UID: \"4cbd8a74-902a-45cf-b431-7e98f6ccd25c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0809c8-877wj"
Nov 02 03:36:08.527295 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:08.523248    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxqq5\" (UniqueName: \"kubernetes.io/projected/4cbd8a74-902a-45cf-b431-7e98f6ccd25c-kube-api-access-mxqq5\") pod \"maas-api-revoke-keys-e2e-ipp-b-0809c8-877wj\" (UID: \"4cbd8a74-902a-45cf-b431-7e98f6ccd25c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0809c8-877wj"
Nov 02 03:36:08.527295 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:08.523309    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cbd8a74-902a-45cf-b431-7e98f6ccd25c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-0809c8-877wj\" (UID: \"4cbd8a74-902a-45cf-b431-7e98f6ccd25c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0809c8-877wj"
Nov 02 03:36:08.529927 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:08.529903    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-0a04e0-dockercfg-4gltc\""
Nov 02 03:36:08.530069 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:08.529970    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-0a04e0\""
Nov 02 03:36:08.530069 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:08.530001    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-0a04e0-bc9cc6559-cvsjg"]
Nov 02 03:36:08.617455 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:08.617362    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-0a04e0-5db76b8859-d6szv"]
Nov 02 03:36:08.624080 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:08.624008    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-0a04e0-5db76b8859-d6szv"
Nov 02 03:36:08.624769 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:08.624628    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9dbfe75-2071-49ea-a422-88c87d623118-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-0a04e0-bc9cc6559-cvsjg\" (UID: \"d9dbfe75-2071-49ea-a422-88c87d623118\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0a04e0-bc9cc6559-cvsjg"
Nov 02 03:36:08.624978 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:08.624956    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cbd8a74-902a-45cf-b431-7e98f6ccd25c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-0809c8-877wj\" (UID: \"4cbd8a74-902a-45cf-b431-7e98f6ccd25c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0809c8-877wj"
Nov 02 03:36:08.625101 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:08.625081    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxqq5\" (UniqueName: \"kubernetes.io/projected/4cbd8a74-902a-45cf-b431-7e98f6ccd25c-kube-api-access-mxqq5\") pod \"maas-api-revoke-keys-e2e-ipp-b-0809c8-877wj\" (UID: \"4cbd8a74-902a-45cf-b431-7e98f6ccd25c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0809c8-877wj"
Nov 02 03:36:08.625180 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:08.625143    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckmbh\" (UniqueName: \"kubernetes.io/projected/d9dbfe75-2071-49ea-a422-88c87d623118-kube-api-access-ckmbh\") pod \"payload-pre-processing-e2e-praxis-a-0a04e0-bc9cc6559-cvsjg\" (UID: \"d9dbfe75-2071-49ea-a422-88c87d623118\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0a04e0-bc9cc6559-cvsjg"
Nov 02 03:36:08.625239 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:08.625179    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cbd8a74-902a-45cf-b431-7e98f6ccd25c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-0809c8-877wj\" (UID: \"4cbd8a74-902a-45cf-b431-7e98f6ccd25c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0809c8-877wj"
Nov 02 03:36:08.626103 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:08.626074    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cbd8a74-902a-45cf-b431-7e98f6ccd25c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-0809c8-877wj\" (UID: \"4cbd8a74-902a-45cf-b431-7e98f6ccd25c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0809c8-877wj"
Nov 02 03:36:08.628052 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:08.628009    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cbd8a74-902a-45cf-b431-7e98f6ccd25c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-0809c8-877wj\" (UID: \"4cbd8a74-902a-45cf-b431-7e98f6ccd25c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0809c8-877wj"
Nov 02 03:36:08.682631 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:08.682592    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-0a04e0-5db76b8859-d6szv"]
Nov 02 03:36:08.687585 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:08.687557    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxqq5\" (UniqueName: \"kubernetes.io/projected/4cbd8a74-902a-45cf-b431-7e98f6ccd25c-kube-api-access-mxqq5\") pod \"maas-api-revoke-keys-e2e-ipp-b-0809c8-877wj\" (UID: \"4cbd8a74-902a-45cf-b431-7e98f6ccd25c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0809c8-877wj"
Nov 02 03:36:08.726333 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:08.726303    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckmbh\" (UniqueName: \"kubernetes.io/projected/d9dbfe75-2071-49ea-a422-88c87d623118-kube-api-access-ckmbh\") pod \"payload-pre-processing-e2e-praxis-a-0a04e0-bc9cc6559-cvsjg\" (UID: \"d9dbfe75-2071-49ea-a422-88c87d623118\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0a04e0-bc9cc6559-cvsjg"
Nov 02 03:36:08.726478 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:08.726460    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a3941e2-4480-4bc8-8b7a-b8f7a093a609-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-0a04e0-5db76b8859-d6szv\" (UID: \"2a3941e2-4480-4bc8-8b7a-b8f7a093a609\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-0a04e0-5db76b8859-d6szv"
Nov 02 03:36:08.726528 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:08.726498    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9dbfe75-2071-49ea-a422-88c87d623118-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-0a04e0-bc9cc6559-cvsjg\" (UID: \"d9dbfe75-2071-49ea-a422-88c87d623118\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0a04e0-bc9cc6559-cvsjg"
Nov 02 03:36:08.726587 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:08.726571    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpb2x\" (UniqueName: \"kubernetes.io/projected/2a3941e2-4480-4bc8-8b7a-b8f7a093a609-kube-api-access-lpb2x\") pod \"payload-processing-e2e-praxis-a-0a04e0-5db76b8859-d6szv\" (UID: \"2a3941e2-4480-4bc8-8b7a-b8f7a093a609\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-0a04e0-5db76b8859-d6szv"
Nov 02 03:36:08.727179 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:08.727160    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9dbfe75-2071-49ea-a422-88c87d623118-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-0a04e0-bc9cc6559-cvsjg\" (UID: \"d9dbfe75-2071-49ea-a422-88c87d623118\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0a04e0-bc9cc6559-cvsjg"
Nov 02 03:36:08.785209 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:08.785183    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckmbh\" (UniqueName: \"kubernetes.io/projected/d9dbfe75-2071-49ea-a422-88c87d623118-kube-api-access-ckmbh\") pod \"payload-pre-processing-e2e-praxis-a-0a04e0-bc9cc6559-cvsjg\" (UID: \"d9dbfe75-2071-49ea-a422-88c87d623118\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0a04e0-bc9cc6559-cvsjg"
Nov 02 03:36:08.792075 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:08.792049    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0a04e0-bc9cc6559-cvsjg"
Nov 02 03:36:08.828321 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:08.828288    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a3941e2-4480-4bc8-8b7a-b8f7a093a609-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-0a04e0-5db76b8859-d6szv\" (UID: \"2a3941e2-4480-4bc8-8b7a-b8f7a093a609\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-0a04e0-5db76b8859-d6szv"
Nov 02 03:36:08.828453 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:08.828339    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpb2x\" (UniqueName: \"kubernetes.io/projected/2a3941e2-4480-4bc8-8b7a-b8f7a093a609-kube-api-access-lpb2x\") pod \"payload-processing-e2e-praxis-a-0a04e0-5db76b8859-d6szv\" (UID: \"2a3941e2-4480-4bc8-8b7a-b8f7a093a609\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-0a04e0-5db76b8859-d6szv"
Nov 02 03:36:08.828987 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:08.828965    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a3941e2-4480-4bc8-8b7a-b8f7a093a609-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-0a04e0-5db76b8859-d6szv\" (UID: \"2a3941e2-4480-4bc8-8b7a-b8f7a093a609\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-0a04e0-5db76b8859-d6szv"
Nov 02 03:36:08.886151 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:08.886073    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpb2x\" (UniqueName: \"kubernetes.io/projected/2a3941e2-4480-4bc8-8b7a-b8f7a093a609-kube-api-access-lpb2x\") pod \"payload-processing-e2e-praxis-a-0a04e0-5db76b8859-d6szv\" (UID: \"2a3941e2-4480-4bc8-8b7a-b8f7a093a609\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-0a04e0-5db76b8859-d6szv"
Nov 02 03:36:08.957441 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:08.957411    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-0a04e0-5db76b8859-d6szv"
Nov 02 03:36:08.977163 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:08.977134    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0809c8-877wj"
Nov 02 03:36:09.171420 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:09.171395    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-0a04e0-bc9cc6559-cvsjg"]
Nov 02 03:36:09.172078 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:36:09.172049    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9dbfe75_2071_49ea_a422_88c87d623118.slice/crio-558726d96f09da68bcf828d830c80c89f754ae2fa63b92f1881f8bcda9f175c1 WatchSource:0}: Error finding container 558726d96f09da68bcf828d830c80c89f754ae2fa63b92f1881f8bcda9f175c1: Status 404 returned error can't find the container with id 558726d96f09da68bcf828d830c80c89f754ae2fa63b92f1881f8bcda9f175c1
Nov 02 03:36:09.340005 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:09.339971    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-0a04e0-5db76b8859-d6szv"]
Nov 02 03:36:09.340605 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:36:09.340578    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a3941e2_4480_4bc8_8b7a_b8f7a093a609.slice/crio-359c4effbe5a402285dc2bddb704f797048cd1af0c8cc7a295212f46fddc67f4 WatchSource:0}: Error finding container 359c4effbe5a402285dc2bddb704f797048cd1af0c8cc7a295212f46fddc67f4: Status 404 returned error can't find the container with id 359c4effbe5a402285dc2bddb704f797048cd1af0c8cc7a295212f46fddc67f4
Nov 02 03:36:09.374580 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:09.374394    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0809c8-877wj"]
Nov 02 03:36:09.375252 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:36:09.375229    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cbd8a74_902a_45cf_b431_7e98f6ccd25c.slice/crio-f32fd812390d3186b96f3499a1f2963ea13eff787a1fec2472ec8c154e7f1142 WatchSource:0}: Error finding container f32fd812390d3186b96f3499a1f2963ea13eff787a1fec2472ec8c154e7f1142: Status 404 returned error can't find the container with id f32fd812390d3186b96f3499a1f2963ea13eff787a1fec2472ec8c154e7f1142
Nov 02 03:36:09.978699 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:09.978656    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0a04e0-bc9cc6559-cvsjg" event={"ID":"d9dbfe75-2071-49ea-a422-88c87d623118","Type":"ContainerStarted","Data":"5d3de9e6cec98eec1a77be771cbdea52fb037f009cc109f12360dd633afa6261"}
Nov 02 03:36:09.978931 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:09.978758    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0a04e0-bc9cc6559-cvsjg" event={"ID":"d9dbfe75-2071-49ea-a422-88c87d623118","Type":"ContainerStarted","Data":"558726d96f09da68bcf828d830c80c89f754ae2fa63b92f1881f8bcda9f175c1"}
Nov 02 03:36:09.978931 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:09.978793    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0a04e0-bc9cc6559-cvsjg"
Nov 02 03:36:09.980437 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:09.980407    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0809c8-877wj" event={"ID":"4cbd8a74-902a-45cf-b431-7e98f6ccd25c","Type":"ContainerStarted","Data":"aa62060c88809929e9ffc9de4c20f01ba8c60d77ec1d379d19e578da792d1e94"}
Nov 02 03:36:09.980553 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:09.980461    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0809c8-877wj" event={"ID":"4cbd8a74-902a-45cf-b431-7e98f6ccd25c","Type":"ContainerStarted","Data":"f32fd812390d3186b96f3499a1f2963ea13eff787a1fec2472ec8c154e7f1142"}
Nov 02 03:36:09.982061 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:09.982038    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-0a04e0-5db76b8859-d6szv" event={"ID":"2a3941e2-4480-4bc8-8b7a-b8f7a093a609","Type":"ContainerStarted","Data":"e03e3af415607aa62cfea4a2dfb9cc842e74a558d24f550fe4e81e54093ae559"}
Nov 02 03:36:09.982154 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:09.982072    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-0a04e0-5db76b8859-d6szv" event={"ID":"2a3941e2-4480-4bc8-8b7a-b8f7a093a609","Type":"ContainerStarted","Data":"359c4effbe5a402285dc2bddb704f797048cd1af0c8cc7a295212f46fddc67f4"}
Nov 02 03:36:09.982197 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:09.982165    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-0a04e0-5db76b8859-d6szv"
Nov 02 03:36:10.096793 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:10.096746    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0a04e0-bc9cc6559-cvsjg" podStartSLOduration=2.09673244 podStartE2EDuration="2.09673244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:36:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:36:10.094187429 +0000 UTC m=+2910.672794810" watchObservedRunningTime="2026-11-02 03:36:10.09673244 +0000 UTC m=+2910.675339819"
Nov 02 03:36:10.223873 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:10.223822    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-0a04e0-5db76b8859-d6szv" podStartSLOduration=2.223804806 podStartE2EDuration="2.223804806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:36:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:36:10.210184228 +0000 UTC m=+2910.788791609" watchObservedRunningTime="2026-11-02 03:36:10.223804806 +0000 UTC m=+2910.802412188"
Nov 02 03:36:10.325935 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:10.325814    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0809c8-877wj" podStartSLOduration=2.325795171 podStartE2EDuration="2.325795171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:36:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:36:10.323826991 +0000 UTC m=+2910.902434382" watchObservedRunningTime="2026-11-02 03:36:10.325795171 +0000 UTC m=+2910.904402553"
Nov 02 03:36:10.992997 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:10.992959    4496 generic.go:356] "Generic (PLEG): container finished" podID="4cbd8a74-902a-45cf-b431-7e98f6ccd25c" containerID="aa62060c88809929e9ffc9de4c20f01ba8c60d77ec1d379d19e578da792d1e94" exitCode=0
Nov 02 03:36:10.993443 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:10.993052    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0809c8-877wj" event={"ID":"4cbd8a74-902a-45cf-b431-7e98f6ccd25c","Type":"ContainerDied","Data":"aa62060c88809929e9ffc9de4c20f01ba8c60d77ec1d379d19e578da792d1e94"}
Nov 02 03:36:12.175184 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:12.175156    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0809c8-877wj"
Nov 02 03:36:12.272082 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:12.271997    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4cbd8a74-902a-45cf-b431-7e98f6ccd25c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cbd8a74-902a-45cf-b431-7e98f6ccd25c-maas-api-xks-ca\") pod \"4cbd8a74-902a-45cf-b431-7e98f6ccd25c\" (UID: \"4cbd8a74-902a-45cf-b431-7e98f6ccd25c\") "
Nov 02 03:36:12.272221 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:12.272179    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4cbd8a74-902a-45cf-b431-7e98f6ccd25c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cbd8a74-902a-45cf-b431-7e98f6ccd25c-maas-api-openshift-ca\") pod \"4cbd8a74-902a-45cf-b431-7e98f6ccd25c\" (UID: \"4cbd8a74-902a-45cf-b431-7e98f6ccd25c\") "
Nov 02 03:36:12.272221 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:12.272214    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cbd8a74-902a-45cf-b431-7e98f6ccd25c-kube-api-access-mxqq5\" (UniqueName: \"kubernetes.io/projected/4cbd8a74-902a-45cf-b431-7e98f6ccd25c-kube-api-access-mxqq5\") pod \"4cbd8a74-902a-45cf-b431-7e98f6ccd25c\" (UID: \"4cbd8a74-902a-45cf-b431-7e98f6ccd25c\") "
Nov 02 03:36:12.272558 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:12.272534    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cbd8a74-902a-45cf-b431-7e98f6ccd25c-maas-api-openshift-ca" pod "4cbd8a74-902a-45cf-b431-7e98f6ccd25c" (UID: "4cbd8a74-902a-45cf-b431-7e98f6ccd25c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:36:12.272675 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:12.272661    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cbd8a74-902a-45cf-b431-7e98f6ccd25c-maas-api-openshift-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:36:12.274070 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:12.274039    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cbd8a74-902a-45cf-b431-7e98f6ccd25c-maas-api-xks-ca" pod "4cbd8a74-902a-45cf-b431-7e98f6ccd25c" (UID: "4cbd8a74-902a-45cf-b431-7e98f6ccd25c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 03:36:12.274277 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:12.274257    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cbd8a74-902a-45cf-b431-7e98f6ccd25c-kube-api-access-mxqq5" pod "4cbd8a74-902a-45cf-b431-7e98f6ccd25c" (UID: "4cbd8a74-902a-45cf-b431-7e98f6ccd25c"). InnerVolumeSpecName "kube-api-access-mxqq5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:36:12.373685 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:12.373645    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cbd8a74-902a-45cf-b431-7e98f6ccd25c-maas-api-xks-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:36:12.373685 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:12.373679    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxqq5\" (UniqueName: \"kubernetes.io/projected/4cbd8a74-902a-45cf-b431-7e98f6ccd25c-kube-api-access-mxqq5\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:36:13.008506 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:13.008477    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0809c8-877wj"
Nov 02 03:36:13.008711 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:13.008507    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0809c8-877wj" event={"ID":"4cbd8a74-902a-45cf-b431-7e98f6ccd25c","Type":"ContainerDied","Data":"f32fd812390d3186b96f3499a1f2963ea13eff787a1fec2472ec8c154e7f1142"}
Nov 02 03:36:13.008711 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:13.008557    4496 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f32fd812390d3186b96f3499a1f2963ea13eff787a1fec2472ec8c154e7f1142"
Nov 02 03:36:20.996505 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:20.996475    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0a04e0-bc9cc6559-cvsjg"
Nov 02 03:36:20.996995 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:20.996541    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-0a04e0-5db76b8859-d6szv"
Nov 02 03:36:33.526622 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:36:33.526589    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 03:36:41.989179 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:41.989147    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-dd87f0-54b67c4cd8-k55c8"]
Nov 02 03:36:41.989640 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:41.989379    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-dd87f0-54b67c4cd8-k55c8" podUID="0609027c-0577-4487-8d37-0c0316de5c5f" containerName="payload-processing" containerID="cri-o://febe49dc3c122f0c5002cd3aa9b752ba54aa87bff947f046c61a840931b894b7" gracePeriod=30
Nov 02 03:36:42.043352 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:42.043325    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-dd87f0-5dbfc776f5-4b62z"]
Nov 02 03:36:42.043541 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:42.043521    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dd87f0-5dbfc776f5-4b62z" podUID="20f3f370-4233-4348-b0f6-176122fc91f8" containerName="payload-pre-processing" containerID="cri-o://a836d92cac3744936b23ae2ec6037127762fa40e928d5b80876d84ccdb3c7a8b" gracePeriod=30
Nov 02 03:36:42.240371 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:42.240280    4496 generic.go:356] "Generic (PLEG): container finished" podID="20f3f370-4233-4348-b0f6-176122fc91f8" containerID="a836d92cac3744936b23ae2ec6037127762fa40e928d5b80876d84ccdb3c7a8b" exitCode=0
Nov 02 03:36:42.240570 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:42.240383    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dd87f0-5dbfc776f5-4b62z" event={"ID":"20f3f370-4233-4348-b0f6-176122fc91f8","Type":"ContainerDied","Data":"a836d92cac3744936b23ae2ec6037127762fa40e928d5b80876d84ccdb3c7a8b"}
Nov 02 03:36:42.242750 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:42.242372    4496 generic.go:356] "Generic (PLEG): container finished" podID="0609027c-0577-4487-8d37-0c0316de5c5f" containerID="febe49dc3c122f0c5002cd3aa9b752ba54aa87bff947f046c61a840931b894b7" exitCode=0
Nov 02 03:36:42.242750 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:42.242441    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-dd87f0-54b67c4cd8-k55c8" event={"ID":"0609027c-0577-4487-8d37-0c0316de5c5f","Type":"ContainerDied","Data":"febe49dc3c122f0c5002cd3aa9b752ba54aa87bff947f046c61a840931b894b7"}
Nov 02 03:36:42.325957 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:42.325929    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-dd87f0-54b67c4cd8-k55c8"
Nov 02 03:36:42.354032 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:42.353992    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dd87f0-5dbfc776f5-4b62z"
Nov 02 03:36:42.449364 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:42.449336    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0609027c-0577-4487-8d37-0c0316de5c5f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0609027c-0577-4487-8d37-0c0316de5c5f-plugins-config-volume\") pod \"0609027c-0577-4487-8d37-0c0316de5c5f\" (UID: \"0609027c-0577-4487-8d37-0c0316de5c5f\") "
Nov 02 03:36:42.449554 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:42.449448    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20f3f370-4233-4348-b0f6-176122fc91f8-kube-api-access-7v9hl\" (UniqueName: \"kubernetes.io/projected/20f3f370-4233-4348-b0f6-176122fc91f8-kube-api-access-7v9hl\") pod \"20f3f370-4233-4348-b0f6-176122fc91f8\" (UID: \"20f3f370-4233-4348-b0f6-176122fc91f8\") "
Nov 02 03:36:42.449632 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:42.449586    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20f3f370-4233-4348-b0f6-176122fc91f8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20f3f370-4233-4348-b0f6-176122fc91f8-plugins-config-volume\") pod \"20f3f370-4233-4348-b0f6-176122fc91f8\" (UID: \"20f3f370-4233-4348-b0f6-176122fc91f8\") "
Nov 02 03:36:42.449697 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:42.449668    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0609027c-0577-4487-8d37-0c0316de5c5f-kube-api-access-9jphk\" (UniqueName: \"kubernetes.io/projected/0609027c-0577-4487-8d37-0c0316de5c5f-kube-api-access-9jphk\") pod \"0609027c-0577-4487-8d37-0c0316de5c5f\" (UID: \"0609027c-0577-4487-8d37-0c0316de5c5f\") "
Nov 02 03:36:42.449761 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:42.449736    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0609027c-0577-4487-8d37-0c0316de5c5f-plugins-config-volume" pod "0609027c-0577-4487-8d37-0c0316de5c5f" (UID: "0609027c-0577-4487-8d37-0c0316de5c5f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:36:42.450078 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:42.450050    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20f3f370-4233-4348-b0f6-176122fc91f8-plugins-config-volume" pod "20f3f370-4233-4348-b0f6-176122fc91f8" (UID: "20f3f370-4233-4348-b0f6-176122fc91f8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:36:42.450248 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:42.450225    4496 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20f3f370-4233-4348-b0f6-176122fc91f8-plugins-config-volume\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:36:42.450330 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:42.450252    4496 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0609027c-0577-4487-8d37-0c0316de5c5f-plugins-config-volume\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:36:42.451845 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:42.451814    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20f3f370-4233-4348-b0f6-176122fc91f8-kube-api-access-7v9hl" pod "20f3f370-4233-4348-b0f6-176122fc91f8" (UID: "20f3f370-4233-4348-b0f6-176122fc91f8"). InnerVolumeSpecName "kube-api-access-7v9hl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:36:42.452095 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:42.452069    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0609027c-0577-4487-8d37-0c0316de5c5f-kube-api-access-9jphk" pod "0609027c-0577-4487-8d37-0c0316de5c5f" (UID: "0609027c-0577-4487-8d37-0c0316de5c5f"). InnerVolumeSpecName "kube-api-access-9jphk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:36:42.551420 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:42.551330    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jphk\" (UniqueName: \"kubernetes.io/projected/0609027c-0577-4487-8d37-0c0316de5c5f-kube-api-access-9jphk\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:36:42.551420 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:42.551359    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7v9hl\" (UniqueName: \"kubernetes.io/projected/20f3f370-4233-4348-b0f6-176122fc91f8-kube-api-access-7v9hl\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:36:43.251428 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:43.251399    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dd87f0-5dbfc776f5-4b62z"
Nov 02 03:36:43.251949 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:43.251432    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dd87f0-5dbfc776f5-4b62z" event={"ID":"20f3f370-4233-4348-b0f6-176122fc91f8","Type":"ContainerDied","Data":"33c593065195017c2f6e467b13c20dca025dbfdd132579d1590d8718562787f5"}
Nov 02 03:36:43.251949 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:43.251541    4496 scope.go:122] "RemoveContainer" containerID="a836d92cac3744936b23ae2ec6037127762fa40e928d5b80876d84ccdb3c7a8b"
Nov 02 03:36:43.253249 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:43.253105    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-dd87f0-54b67c4cd8-k55c8"
Nov 02 03:36:43.253249 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:43.253151    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-dd87f0-54b67c4cd8-k55c8" event={"ID":"0609027c-0577-4487-8d37-0c0316de5c5f","Type":"ContainerDied","Data":"8d849b8556432f23901781ccac241f3e18ffa496422a594f11d188bb2f944482"}
Nov 02 03:36:43.263404 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:43.263385    4496 scope.go:122] "RemoveContainer" containerID="febe49dc3c122f0c5002cd3aa9b752ba54aa87bff947f046c61a840931b894b7"
Nov 02 03:36:43.419689 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:43.419658    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-dd87f0-54b67c4cd8-k55c8"]
Nov 02 03:36:43.423447 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:43.423420    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-dd87f0-54b67c4cd8-k55c8"]
Nov 02 03:36:43.539206 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:43.539100    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-dd87f0-5dbfc776f5-4b62z"]
Nov 02 03:36:43.555624 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:43.555582    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-dd87f0-5dbfc776f5-4b62z"]
Nov 02 03:36:43.586862 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:43.586817    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0609027c-0577-4487-8d37-0c0316de5c5f" path="/var/lib/kubelet/pods/0609027c-0577-4487-8d37-0c0316de5c5f/volumes"
Nov 02 03:36:43.587619 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:43.587594    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20f3f370-4233-4348-b0f6-176122fc91f8" path="/var/lib/kubelet/pods/20f3f370-4233-4348-b0f6-176122fc91f8/volumes"
Nov 02 03:36:43.753990 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:43.753949    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dd87f0-kcqqn"]
Nov 02 03:36:43.757243 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:43.757214    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dd87f0-kcqqn"]
Nov 02 03:36:45.575490 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:45.575458    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ebf7e9b-bce3-4b88-9f30-1341cc3bfb7e" path="/var/lib/kubelet/pods/0ebf7e9b-bce3-4b88-9f30-1341cc3bfb7e/volumes"
Nov 02 03:36:46.062186 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:46.062147    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lpxkv"]
Nov 02 03:36:46.062411 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:46.062383    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lpxkv" podUID="f395e8e3-ec87-4a63-94b5-8b29e017c2e9" containerName="istio-proxy" containerID="cri-o://9e9567b0a912755a5e5ddc41ca9c638ebe3b98bfbcdccaa34c95aaea52404ad2" gracePeriod=30
Nov 02 03:36:46.115133 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:46.115084    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cdwxx"]
Nov 02 03:36:46.115367 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:46.115340    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cdwxx" podUID="41697786-2b0f-4cb2-8975-1942d727c987" containerName="istio-proxy" containerID="cri-o://d97efdf4aed3ec1a74c908cc6aa65855bf41c4e7978e28e395833834a40d45e9" gracePeriod=30
Nov 02 03:36:46.120436 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:46.120406    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6qhq2"]
Nov 02 03:36:46.120656 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:46.120631    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6qhq2" podUID="175952d1-62ed-4ea6-85d0-37d1a651f769" containerName="istio-proxy" containerID="cri-o://d732676fda6c0553e54381592d74a6114f6870ce3d6374678e041e1fc3cae7b1" gracePeriod=30
Nov 02 03:36:47.899975 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:47.899937    4496 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-cdwxx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 03:36:47.900399 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:47.900009    4496 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cdwxx" podUID="41697786-2b0f-4cb2-8975-1942d727c987" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 03:36:51.143340 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:36:51.143297    4496 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf395e8e3_ec87_4a63_94b5_8b29e017c2e9.slice/crio-conmon-9e9567b0a912755a5e5ddc41ca9c638ebe3b98bfbcdccaa34c95aaea52404ad2.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf395e8e3_ec87_4a63_94b5_8b29e017c2e9.slice/crio-9e9567b0a912755a5e5ddc41ca9c638ebe3b98bfbcdccaa34c95aaea52404ad2.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 03:36:51.357226 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:51.357132    4496 generic.go:356] "Generic (PLEG): container finished" podID="175952d1-62ed-4ea6-85d0-37d1a651f769" containerID="d732676fda6c0553e54381592d74a6114f6870ce3d6374678e041e1fc3cae7b1" exitCode=0
Nov 02 03:36:51.357410 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:51.357308    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6qhq2" event={"ID":"175952d1-62ed-4ea6-85d0-37d1a651f769","Type":"ContainerDied","Data":"d732676fda6c0553e54381592d74a6114f6870ce3d6374678e041e1fc3cae7b1"}
Nov 02 03:36:51.360504 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:51.360472    4496 generic.go:356] "Generic (PLEG): container finished" podID="f395e8e3-ec87-4a63-94b5-8b29e017c2e9" containerID="9e9567b0a912755a5e5ddc41ca9c638ebe3b98bfbcdccaa34c95aaea52404ad2" exitCode=0
Nov 02 03:36:51.360641 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:51.360557    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lpxkv" event={"ID":"f395e8e3-ec87-4a63-94b5-8b29e017c2e9","Type":"ContainerDied","Data":"9e9567b0a912755a5e5ddc41ca9c638ebe3b98bfbcdccaa34c95aaea52404ad2"}
Nov 02 03:36:51.362999 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:51.362977    4496 generic.go:356] "Generic (PLEG): container finished" podID="41697786-2b0f-4cb2-8975-1942d727c987" containerID="d97efdf4aed3ec1a74c908cc6aa65855bf41c4e7978e28e395833834a40d45e9" exitCode=0
Nov 02 03:36:51.363219 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:51.363052    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cdwxx" event={"ID":"41697786-2b0f-4cb2-8975-1942d727c987","Type":"ContainerDied","Data":"d97efdf4aed3ec1a74c908cc6aa65855bf41c4e7978e28e395833834a40d45e9"}
Nov 02 03:36:51.363399 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:51.363382    4496 scope.go:122] "RemoveContainer" containerID="a9dec28ff6474e708ad4695c5434851a7e15a0187897222fc106592a444533b5"
Nov 02 03:36:51.444066 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:51.441550    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lpxkv"
Nov 02 03:36:51.498552 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:51.498528    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cdwxx"
Nov 02 03:36:51.505171 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:51.505150    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6qhq2"
Nov 02 03:36:51.561814 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:51.561782    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f395e8e3-ec87-4a63-94b5-8b29e017c2e9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f395e8e3-ec87-4a63-94b5-8b29e017c2e9-workload-socket\") pod \"f395e8e3-ec87-4a63-94b5-8b29e017c2e9\" (UID: \"f395e8e3-ec87-4a63-94b5-8b29e017c2e9\") "
Nov 02 03:36:51.561990 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:51.561853    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41697786-2b0f-4cb2-8975-1942d727c987-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41697786-2b0f-4cb2-8975-1942d727c987-istio-envoy\") pod \"41697786-2b0f-4cb2-8975-1942d727c987\" (UID: \"41697786-2b0f-4cb2-8975-1942d727c987\") "
Nov 02 03:36:51.561990 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:51.561899    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/175952d1-62ed-4ea6-85d0-37d1a651f769-istio-token\" (UniqueName: \"kubernetes.io/projected/175952d1-62ed-4ea6-85d0-37d1a651f769-istio-token\") pod \"175952d1-62ed-4ea6-85d0-37d1a651f769\" (UID: \"175952d1-62ed-4ea6-85d0-37d1a651f769\") "
Nov 02 03:36:51.561990 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:51.561930    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f395e8e3-ec87-4a63-94b5-8b29e017c2e9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f395e8e3-ec87-4a63-94b5-8b29e017c2e9-istio-podinfo\") pod \"f395e8e3-ec87-4a63-94b5-8b29e017c2e9\" (UID: \"f395e8e3-ec87-4a63-94b5-8b29e017c2e9\") "
Nov 02 03:36:51.561990 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:51.561959    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f395e8e3-ec87-4a63-94b5-8b29e017c2e9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f395e8e3-ec87-4a63-94b5-8b29e017c2e9-istiod-ca-cert\") pod \"f395e8e3-ec87-4a63-94b5-8b29e017c2e9\" (UID: \"f395e8e3-ec87-4a63-94b5-8b29e017c2e9\") "
Nov 02 03:36:51.562713 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:51.562114    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f395e8e3-ec87-4a63-94b5-8b29e017c2e9-workload-socket" pod "f395e8e3-ec87-4a63-94b5-8b29e017c2e9" (UID: "f395e8e3-ec87-4a63-94b5-8b29e017c2e9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:36:51.562713 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:51.562688    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41697786-2b0f-4cb2-8975-1942d727c987-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41697786-2b0f-4cb2-8975-1942d727c987-credential-socket\") pod \"41697786-2b0f-4cb2-8975-1942d727c987\" (UID: \"41697786-2b0f-4cb2-8975-1942d727c987\") "
Nov 02 03:36:51.562879 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:51.562379    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f395e8e3-ec87-4a63-94b5-8b29e017c2e9-istiod-ca-cert" pod "f395e8e3-ec87-4a63-94b5-8b29e017c2e9" (UID: "f395e8e3-ec87-4a63-94b5-8b29e017c2e9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:36:51.562879 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:51.562847    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41697786-2b0f-4cb2-8975-1942d727c987-istio-token\" (UniqueName: \"kubernetes.io/projected/41697786-2b0f-4cb2-8975-1942d727c987-istio-token\") pod \"41697786-2b0f-4cb2-8975-1942d727c987\" (UID: \"41697786-2b0f-4cb2-8975-1942d727c987\") "
Nov 02 03:36:51.563002 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:51.562884    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f395e8e3-ec87-4a63-94b5-8b29e017c2e9-istio-token\" (UniqueName: \"kubernetes.io/projected/f395e8e3-ec87-4a63-94b5-8b29e017c2e9-istio-token\") pod \"f395e8e3-ec87-4a63-94b5-8b29e017c2e9\" (UID: \"f395e8e3-ec87-4a63-94b5-8b29e017c2e9\") "
Nov 02 03:36:51.563002 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:51.562914    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/175952d1-62ed-4ea6-85d0-37d1a651f769-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/175952d1-62ed-4ea6-85d0-37d1a651f769-istiod-ca-cert\") pod \"175952d1-62ed-4ea6-85d0-37d1a651f769\" (UID: \"175952d1-62ed-4ea6-85d0-37d1a651f769\") "
Nov 02 03:36:51.563002 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:51.562958    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41697786-2b0f-4cb2-8975-1942d727c987-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41697786-2b0f-4cb2-8975-1942d727c987-istiod-ca-cert\") pod \"41697786-2b0f-4cb2-8975-1942d727c987\" (UID: \"41697786-2b0f-4cb2-8975-1942d727c987\") "
Nov 02 03:36:51.563002 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:51.562961    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41697786-2b0f-4cb2-8975-1942d727c987-credential-socket" pod "41697786-2b0f-4cb2-8975-1942d727c987" (UID: "41697786-2b0f-4cb2-8975-1942d727c987"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:36:51.563002 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:51.562992    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41697786-2b0f-4cb2-8975-1942d727c987-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41697786-2b0f-4cb2-8975-1942d727c987-workload-socket\") pod \"41697786-2b0f-4cb2-8975-1942d727c987\" (UID: \"41697786-2b0f-4cb2-8975-1942d727c987\") "
Nov 02 03:36:51.563325 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:51.563040    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f395e8e3-ec87-4a63-94b5-8b29e017c2e9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f395e8e3-ec87-4a63-94b5-8b29e017c2e9-workload-certs\") pod \"f395e8e3-ec87-4a63-94b5-8b29e017c2e9\" (UID: \"f395e8e3-ec87-4a63-94b5-8b29e017c2e9\") "
Nov 02 03:36:51.563325 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:51.563078    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/175952d1-62ed-4ea6-85d0-37d1a651f769-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/175952d1-62ed-4ea6-85d0-37d1a651f769-istio-data\") pod \"175952d1-62ed-4ea6-85d0-37d1a651f769\" (UID: \"175952d1-62ed-4ea6-85d0-37d1a651f769\") "
Nov 02 03:36:51.563325 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:51.563150    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/175952d1-62ed-4ea6-85d0-37d1a651f769-kube-api-access-vkccw\" (UniqueName: \"kubernetes.io/projected/175952d1-62ed-4ea6-85d0-37d1a651f769-kube-api-access-vkccw\") pod \"175952d1-62ed-4ea6-85d0-37d1a651f769\" (UID: \"175952d1-62ed-4ea6-85d0-37d1a651f769\") "
Nov 02 03:36:51.563325 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:51.563196    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/175952d1-62ed-4ea6-85d0-37d1a651f769-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/175952d1-62ed-4ea6-85d0-37d1a651f769-credential-socket\") pod \"175952d1-62ed-4ea6-85d0-37d1a651f769\" (UID: \"175952d1-62ed-4ea6-85d0-37d1a651f769\") "
Nov 02 03:36:51.563325 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:51.563253    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f395e8e3-ec87-4a63-94b5-8b29e017c2e9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f395e8e3-ec87-4a63-94b5-8b29e017c2e9-credential-socket\") pod \"f395e8e3-ec87-4a63-94b5-8b29e017c2e9\" (UID: \"f395e8e3-ec87-4a63-94b5-8b29e017c2e9\") "
Nov 02 03:36:51.563325 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:51.563310    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41697786-2b0f-4cb2-8975-1942d727c987-istiod-ca-cert" pod "41697786-2b0f-4cb2-8975-1942d727c987" (UID: "41697786-2b0f-4cb2-8975-1942d727c987"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:36:51.563655 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:51.563326    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f395e8e3-ec87-4a63-94b5-8b29e017c2e9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f395e8e3-ec87-4a63-94b5-8b29e017c2e9-istio-data\") pod \"f395e8e3-ec87-4a63-94b5-8b29e017c2e9\" (UID: \"f395e8e3-ec87-4a63-94b5-8b29e017c2e9\") "
Nov 02 03:36:51.563655 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:51.563363    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41697786-2b0f-4cb2-8975-1942d727c987-kube-api-access-jvmwf\" (UniqueName: \"kubernetes.io/projected/41697786-2b0f-4cb2-8975-1942d727c987-kube-api-access-jvmwf\") pod \"41697786-2b0f-4cb2-8975-1942d727c987\" (UID: \"41697786-2b0f-4cb2-8975-1942d727c987\") "
Nov 02 03:36:51.563655 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:51.563392    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/175952d1-62ed-4ea6-85d0-37d1a651f769-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/175952d1-62ed-4ea6-85d0-37d1a651f769-workload-socket\") pod \"175952d1-62ed-4ea6-85d0-37d1a651f769\" (UID: \"175952d1-62ed-4ea6-85d0-37d1a651f769\") "
Nov 02 03:36:51.563655 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:51.563421    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/175952d1-62ed-4ea6-85d0-37d1a651f769-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/175952d1-62ed-4ea6-85d0-37d1a651f769-istio-podinfo\") pod \"175952d1-62ed-4ea6-85d0-37d1a651f769\" (UID: \"175952d1-62ed-4ea6-85d0-37d1a651f769\") "
Nov 02 03:36:51.563655 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:51.563453    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/175952d1-62ed-4ea6-85d0-37d1a651f769-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/175952d1-62ed-4ea6-85d0-37d1a651f769-workload-certs\") pod \"175952d1-62ed-4ea6-85d0-37d1a651f769\" (UID: \"175952d1-62ed-4ea6-85d0-37d1a651f769\") "
Nov 02 03:36:51.563655 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:51.563490    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/175952d1-62ed-4ea6-85d0-37d1a651f769-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/175952d1-62ed-4ea6-85d0-37d1a651f769-istio-envoy\") pod \"175952d1-62ed-4ea6-85d0-37d1a651f769\" (UID: \"175952d1-62ed-4ea6-85d0-37d1a651f769\") "
Nov 02 03:36:51.563655 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:51.563521    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41697786-2b0f-4cb2-8975-1942d727c987-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41697786-2b0f-4cb2-8975-1942d727c987-istio-data\") pod \"41697786-2b0f-4cb2-8975-1942d727c987\" (UID: \"41697786-2b0f-4cb2-8975-1942d727c987\") "
Nov 02 03:36:51.563655 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:51.563549    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f395e8e3-ec87-4a63-94b5-8b29e017c2e9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f395e8e3-ec87-4a63-94b5-8b29e017c2e9-istio-envoy\") pod \"f395e8e3-ec87-4a63-94b5-8b29e017c2e9\" (UID: \"f395e8e3-ec87-4a63-94b5-8b29e017c2e9\") "
Nov 02 03:36:51.563655 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:51.563551    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/175952d1-62ed-4ea6-85d0-37d1a651f769-istio-data" pod "175952d1-62ed-4ea6-85d0-37d1a651f769" (UID: "175952d1-62ed-4ea6-85d0-37d1a651f769"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:36:51.563655 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:51.563579    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f395e8e3-ec87-4a63-94b5-8b29e017c2e9-kube-api-access-2jldt\" (UniqueName: \"kubernetes.io/projected/f395e8e3-ec87-4a63-94b5-8b29e017c2e9-kube-api-access-2jldt\") pod \"f395e8e3-ec87-4a63-94b5-8b29e017c2e9\" (UID: \"f395e8e3-ec87-4a63-94b5-8b29e017c2e9\") "
Nov 02 03:36:51.563655 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:51.563642    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/41697786-2b0f-4cb2-8975-1942d727c987-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41697786-2b0f-4cb2-8975-1942d727c987-istio-podinfo\") pod \"41697786-2b0f-4cb2-8975-1942d727c987\" (UID: \"41697786-2b0f-4cb2-8975-1942d727c987\") "
Nov 02 03:36:51.564279 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:51.563671    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41697786-2b0f-4cb2-8975-1942d727c987-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41697786-2b0f-4cb2-8975-1942d727c987-workload-certs\") pod \"41697786-2b0f-4cb2-8975-1942d727c987\" (UID: \"41697786-2b0f-4cb2-8975-1942d727c987\") "
Nov 02 03:36:51.564279 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:51.563718    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41697786-2b0f-4cb2-8975-1942d727c987-workload-socket" pod "41697786-2b0f-4cb2-8975-1942d727c987" (UID: "41697786-2b0f-4cb2-8975-1942d727c987"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:36:51.564279 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:51.563881    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f395e8e3-ec87-4a63-94b5-8b29e017c2e9-workload-certs" pod "f395e8e3-ec87-4a63-94b5-8b29e017c2e9" (UID: "f395e8e3-ec87-4a63-94b5-8b29e017c2e9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:36:51.564436 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:51.564308    4496 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f395e8e3-ec87-4a63-94b5-8b29e017c2e9-workload-socket\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:36:51.564436 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:51.564332    4496 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f395e8e3-ec87-4a63-94b5-8b29e017c2e9-istiod-ca-cert\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:36:51.564436 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:51.564348    4496 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41697786-2b0f-4cb2-8975-1942d727c987-credential-socket\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:36:51.564436 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:51.564364    4496 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41697786-2b0f-4cb2-8975-1942d727c987-istiod-ca-cert\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:36:51.564436 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:51.564379    4496 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41697786-2b0f-4cb2-8975-1942d727c987-workload-socket\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:36:51.564436 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:51.564393    4496 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f395e8e3-ec87-4a63-94b5-8b29e017c2e9-workload-certs\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:36:51.564436 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:51.564407    4496 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/175952d1-62ed-4ea6-85d0-37d1a651f769-istio-data\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:36:51.564691 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:51.564633    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41697786-2b0f-4cb2-8975-1942d727c987-workload-certs" pod "41697786-2b0f-4cb2-8975-1942d727c987" (UID: "41697786-2b0f-4cb2-8975-1942d727c987"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:36:51.565279 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:51.565232    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f395e8e3-ec87-4a63-94b5-8b29e017c2e9-credential-socket" pod "f395e8e3-ec87-4a63-94b5-8b29e017c2e9" (UID: "f395e8e3-ec87-4a63-94b5-8b29e017c2e9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:36:51.565561 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:51.565329    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/175952d1-62ed-4ea6-85d0-37d1a651f769-workload-certs" pod "175952d1-62ed-4ea6-85d0-37d1a651f769" (UID: "175952d1-62ed-4ea6-85d0-37d1a651f769"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:36:51.567312 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:51.567284    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41697786-2b0f-4cb2-8975-1942d727c987-istio-data" pod "41697786-2b0f-4cb2-8975-1942d727c987" (UID: "41697786-2b0f-4cb2-8975-1942d727c987"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:36:51.567486 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:51.567468    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/175952d1-62ed-4ea6-85d0-37d1a651f769-credential-socket" pod "175952d1-62ed-4ea6-85d0-37d1a651f769" (UID: "175952d1-62ed-4ea6-85d0-37d1a651f769"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:36:51.569958 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:51.569780    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f395e8e3-ec87-4a63-94b5-8b29e017c2e9-kube-api-access-2jldt" pod "f395e8e3-ec87-4a63-94b5-8b29e017c2e9" (UID: "f395e8e3-ec87-4a63-94b5-8b29e017c2e9"). InnerVolumeSpecName "kube-api-access-2jldt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:36:51.569958 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:51.569788    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/175952d1-62ed-4ea6-85d0-37d1a651f769-istio-envoy" pod "175952d1-62ed-4ea6-85d0-37d1a651f769" (UID: "175952d1-62ed-4ea6-85d0-37d1a651f769"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:36:51.569958 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:51.569871    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f395e8e3-ec87-4a63-94b5-8b29e017c2e9-istio-token" pod "f395e8e3-ec87-4a63-94b5-8b29e017c2e9" (UID: "f395e8e3-ec87-4a63-94b5-8b29e017c2e9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:36:51.569958 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:51.569929    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f395e8e3-ec87-4a63-94b5-8b29e017c2e9-istio-podinfo" pod "f395e8e3-ec87-4a63-94b5-8b29e017c2e9" (UID: "f395e8e3-ec87-4a63-94b5-8b29e017c2e9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 03:36:51.570309 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:51.569979    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41697786-2b0f-4cb2-8975-1942d727c987-istio-envoy" pod "41697786-2b0f-4cb2-8975-1942d727c987" (UID: "41697786-2b0f-4cb2-8975-1942d727c987"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:36:51.570309 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:51.570073    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/175952d1-62ed-4ea6-85d0-37d1a651f769-istio-token" pod "175952d1-62ed-4ea6-85d0-37d1a651f769" (UID: "175952d1-62ed-4ea6-85d0-37d1a651f769"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:36:51.570424 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:51.570325    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/175952d1-62ed-4ea6-85d0-37d1a651f769-workload-socket" pod "175952d1-62ed-4ea6-85d0-37d1a651f769" (UID: "175952d1-62ed-4ea6-85d0-37d1a651f769"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:36:51.570537 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:51.570508    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/41697786-2b0f-4cb2-8975-1942d727c987-istio-podinfo" pod "41697786-2b0f-4cb2-8975-1942d727c987" (UID: "41697786-2b0f-4cb2-8975-1942d727c987"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 03:36:51.570881 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:51.570805    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41697786-2b0f-4cb2-8975-1942d727c987-kube-api-access-jvmwf" pod "41697786-2b0f-4cb2-8975-1942d727c987" (UID: "41697786-2b0f-4cb2-8975-1942d727c987"). InnerVolumeSpecName "kube-api-access-jvmwf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:36:51.571263 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:51.571222    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/175952d1-62ed-4ea6-85d0-37d1a651f769-istiod-ca-cert" pod "175952d1-62ed-4ea6-85d0-37d1a651f769" (UID: "175952d1-62ed-4ea6-85d0-37d1a651f769"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:36:51.571352 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:51.571280    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f395e8e3-ec87-4a63-94b5-8b29e017c2e9-istio-data" pod "f395e8e3-ec87-4a63-94b5-8b29e017c2e9" (UID: "f395e8e3-ec87-4a63-94b5-8b29e017c2e9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:36:51.571352 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:51.571328    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/175952d1-62ed-4ea6-85d0-37d1a651f769-kube-api-access-vkccw" pod "175952d1-62ed-4ea6-85d0-37d1a651f769" (UID: "175952d1-62ed-4ea6-85d0-37d1a651f769"). InnerVolumeSpecName "kube-api-access-vkccw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:36:51.572217 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:51.572197    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/175952d1-62ed-4ea6-85d0-37d1a651f769-istio-podinfo" pod "175952d1-62ed-4ea6-85d0-37d1a651f769" (UID: "175952d1-62ed-4ea6-85d0-37d1a651f769"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 03:36:51.572659 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:51.572639    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41697786-2b0f-4cb2-8975-1942d727c987-istio-token" pod "41697786-2b0f-4cb2-8975-1942d727c987" (UID: "41697786-2b0f-4cb2-8975-1942d727c987"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:36:51.573112 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:51.573088    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f395e8e3-ec87-4a63-94b5-8b29e017c2e9-istio-envoy" pod "f395e8e3-ec87-4a63-94b5-8b29e017c2e9" (UID: "f395e8e3-ec87-4a63-94b5-8b29e017c2e9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:36:51.666068 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:51.665945    4496 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41697786-2b0f-4cb2-8975-1942d727c987-istio-envoy\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:36:51.666068 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:51.665986    4496 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/175952d1-62ed-4ea6-85d0-37d1a651f769-istio-token\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:36:51.666068 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:51.666002    4496 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f395e8e3-ec87-4a63-94b5-8b29e017c2e9-istio-podinfo\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:36:51.666068 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:51.666011    4496 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41697786-2b0f-4cb2-8975-1942d727c987-istio-token\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:36:51.666068 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:51.666031    4496 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f395e8e3-ec87-4a63-94b5-8b29e017c2e9-istio-token\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:36:51.666068 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:51.666039    4496 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/175952d1-62ed-4ea6-85d0-37d1a651f769-istiod-ca-cert\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:36:51.666068 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:51.666048    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkccw\" (UniqueName: \"kubernetes.io/projected/175952d1-62ed-4ea6-85d0-37d1a651f769-kube-api-access-vkccw\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:36:51.666068 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:51.666057    4496 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/175952d1-62ed-4ea6-85d0-37d1a651f769-credential-socket\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:36:51.666068 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:51.666067    4496 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f395e8e3-ec87-4a63-94b5-8b29e017c2e9-credential-socket\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:36:51.666068 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:51.666076    4496 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f395e8e3-ec87-4a63-94b5-8b29e017c2e9-istio-data\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:36:51.666504 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:51.666085    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvmwf\" (UniqueName: \"kubernetes.io/projected/41697786-2b0f-4cb2-8975-1942d727c987-kube-api-access-jvmwf\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:36:51.666504 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:51.666093    4496 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/175952d1-62ed-4ea6-85d0-37d1a651f769-workload-socket\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:36:51.666504 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:51.666102    4496 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/175952d1-62ed-4ea6-85d0-37d1a651f769-istio-podinfo\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:36:51.666504 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:51.666110    4496 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/175952d1-62ed-4ea6-85d0-37d1a651f769-workload-certs\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:36:51.666504 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:51.666118    4496 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/175952d1-62ed-4ea6-85d0-37d1a651f769-istio-envoy\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:36:51.666504 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:51.666126    4496 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41697786-2b0f-4cb2-8975-1942d727c987-istio-data\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:36:51.666504 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:51.666134    4496 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f395e8e3-ec87-4a63-94b5-8b29e017c2e9-istio-envoy\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:36:51.666504 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:51.666142    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jldt\" (UniqueName: \"kubernetes.io/projected/f395e8e3-ec87-4a63-94b5-8b29e017c2e9-kube-api-access-2jldt\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:36:51.666504 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:51.666150    4496 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41697786-2b0f-4cb2-8975-1942d727c987-istio-podinfo\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:36:51.666504 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:51.666159    4496 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41697786-2b0f-4cb2-8975-1942d727c987-workload-certs\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:36:52.375854 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:52.375753    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lpxkv" event={"ID":"f395e8e3-ec87-4a63-94b5-8b29e017c2e9","Type":"ContainerDied","Data":"7340e0387975abf27063e436e13360754c3ed87f1264900c2d3e677bf239b12d"}
Nov 02 03:36:52.375854 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:52.375846    4496 scope.go:122] "RemoveContainer" containerID="9e9567b0a912755a5e5ddc41ca9c638ebe3b98bfbcdccaa34c95aaea52404ad2"
Nov 02 03:36:52.376417 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:52.375767    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lpxkv"
Nov 02 03:36:52.378492 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:52.377984    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cdwxx" event={"ID":"41697786-2b0f-4cb2-8975-1942d727c987","Type":"ContainerDied","Data":"7736aa78ae9e6073eaa844613d93a720975a8667bff0f081c8cafabf87e0ac5b"}
Nov 02 03:36:52.378492 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:52.378097    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cdwxx"
Nov 02 03:36:52.381076 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:52.381053    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6qhq2" event={"ID":"175952d1-62ed-4ea6-85d0-37d1a651f769","Type":"ContainerDied","Data":"3370b34b28d613dfe4ec647b3cbeff989dff57991ae0057b26bffbec6dff5dff"}
Nov 02 03:36:52.381251 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:52.381070    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6qhq2"
Nov 02 03:36:52.389199 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:52.389182    4496 scope.go:122] "RemoveContainer" containerID="d97efdf4aed3ec1a74c908cc6aa65855bf41c4e7978e28e395833834a40d45e9"
Nov 02 03:36:52.398459 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:52.398441    4496 scope.go:122] "RemoveContainer" containerID="d732676fda6c0553e54381592d74a6114f6870ce3d6374678e041e1fc3cae7b1"
Nov 02 03:36:52.483073 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:52.483040    4496 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-6qhq2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 03:36:52.483231 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:52.483118    4496 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6qhq2" podUID="175952d1-62ed-4ea6-85d0-37d1a651f769" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 03:36:52.594919 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:52.594882    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6qhq2"]
Nov 02 03:36:52.645812 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:52.645719    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6qhq2"]
Nov 02 03:36:52.709222 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:52.709187    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cdwxx"]
Nov 02 03:36:52.712178 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:52.712155    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cdwxx"]
Nov 02 03:36:52.820427 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:52.820383    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lpxkv"]
Nov 02 03:36:52.823367 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:52.823345    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-lpxkv"]
Nov 02 03:36:53.575804 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:53.575762    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="175952d1-62ed-4ea6-85d0-37d1a651f769" path="/var/lib/kubelet/pods/175952d1-62ed-4ea6-85d0-37d1a651f769/volumes"
Nov 02 03:36:53.576328 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:53.576306    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41697786-2b0f-4cb2-8975-1942d727c987" path="/var/lib/kubelet/pods/41697786-2b0f-4cb2-8975-1942d727c987/volumes"
Nov 02 03:36:53.576790 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:36:53.576773    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f395e8e3-ec87-4a63-94b5-8b29e017c2e9" path="/var/lib/kubelet/pods/f395e8e3-ec87-4a63-94b5-8b29e017c2e9/volumes"
Nov 02 03:37:03.526779 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:37:03.526738    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 03:37:04.407938 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:04.407894    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-0a04e0-785685977-slvzx"]
Nov 02 03:37:04.408514 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:04.408498    4496 cpu_manager.go:414] "containerMap: removing container" podUID="175952d1-62ed-4ea6-85d0-37d1a651f769" containerName="istio-proxy"
Nov 02 03:37:04.408566 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:04.408517    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="175952d1-62ed-4ea6-85d0-37d1a651f769" containerName="istio-proxy"
Nov 02 03:37:04.408566 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:04.408533    4496 cpu_manager.go:414] "containerMap: removing container" podUID="0609027c-0577-4487-8d37-0c0316de5c5f" containerName="payload-processing"
Nov 02 03:37:04.408566 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:04.408538    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0609027c-0577-4487-8d37-0c0316de5c5f" containerName="payload-processing"
Nov 02 03:37:04.408566 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:04.408549    4496 cpu_manager.go:414] "containerMap: removing container" podUID="41697786-2b0f-4cb2-8975-1942d727c987" containerName="istio-proxy"
Nov 02 03:37:04.408566 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:04.408555    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41697786-2b0f-4cb2-8975-1942d727c987" containerName="istio-proxy"
Nov 02 03:37:04.408566 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:04.408563    4496 cpu_manager.go:414] "containerMap: removing container" podUID="41697786-2b0f-4cb2-8975-1942d727c987" containerName="istio-proxy"
Nov 02 03:37:04.408566 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:04.408568    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41697786-2b0f-4cb2-8975-1942d727c987" containerName="istio-proxy"
Nov 02 03:37:04.408789 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:04.408591    4496 cpu_manager.go:414] "containerMap: removing container" podUID="4cbd8a74-902a-45cf-b431-7e98f6ccd25c" containerName="revoke-keys"
Nov 02 03:37:04.408789 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:04.408596    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cbd8a74-902a-45cf-b431-7e98f6ccd25c" containerName="revoke-keys"
Nov 02 03:37:04.408789 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:04.408609    4496 cpu_manager.go:414] "containerMap: removing container" podUID="20f3f370-4233-4348-b0f6-176122fc91f8" containerName="payload-pre-processing"
Nov 02 03:37:04.408789 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:04.408614    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20f3f370-4233-4348-b0f6-176122fc91f8" containerName="payload-pre-processing"
Nov 02 03:37:04.408789 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:04.408625    4496 cpu_manager.go:414] "containerMap: removing container" podUID="f395e8e3-ec87-4a63-94b5-8b29e017c2e9" containerName="istio-proxy"
Nov 02 03:37:04.408789 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:04.408629    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f395e8e3-ec87-4a63-94b5-8b29e017c2e9" containerName="istio-proxy"
Nov 02 03:37:04.408789 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:04.408709    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="f395e8e3-ec87-4a63-94b5-8b29e017c2e9" containerName="istio-proxy"
Nov 02 03:37:04.408789 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:04.408719    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="175952d1-62ed-4ea6-85d0-37d1a651f769" containerName="istio-proxy"
Nov 02 03:37:04.408789 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:04.408726    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="0609027c-0577-4487-8d37-0c0316de5c5f" containerName="payload-processing"
Nov 02 03:37:04.408789 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:04.408734    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="41697786-2b0f-4cb2-8975-1942d727c987" containerName="istio-proxy"
Nov 02 03:37:04.408789 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:04.408740    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="41697786-2b0f-4cb2-8975-1942d727c987" containerName="istio-proxy"
Nov 02 03:37:04.408789 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:04.408749    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cbd8a74-902a-45cf-b431-7e98f6ccd25c" containerName="revoke-keys"
Nov 02 03:37:04.408789 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:04.408755    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="20f3f370-4233-4348-b0f6-176122fc91f8" containerName="payload-pre-processing"
Nov 02 03:37:04.414026 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:04.413997    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0a04e0-785685977-slvzx"
Nov 02 03:37:04.468937 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:04.468907    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-0a04e0-785685977-slvzx"]
Nov 02 03:37:04.515130 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:04.515092    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 03:37:04.515339 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:04.515312    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 03:37:04.517100 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:04.517064    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95349ebe-7d57-45bf-8143-c82c14bd43fa-maas-api-tls\") pod \"maas-api-e2e-praxis-a-0a04e0-785685977-slvzx\" (UID: \"95349ebe-7d57-45bf-8143-c82c14bd43fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0a04e0-785685977-slvzx"
Nov 02 03:37:04.517285 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:04.517260    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c62hm\" (UniqueName: \"kubernetes.io/projected/95349ebe-7d57-45bf-8143-c82c14bd43fa-kube-api-access-c62hm\") pod \"maas-api-e2e-praxis-a-0a04e0-785685977-slvzx\" (UID: \"95349ebe-7d57-45bf-8143-c82c14bd43fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0a04e0-785685977-slvzx"
Nov 02 03:37:04.517599 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:04.517555    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95349ebe-7d57-45bf-8143-c82c14bd43fa-metrics-certs\") pod \"maas-api-e2e-praxis-a-0a04e0-785685977-slvzx\" (UID: \"95349ebe-7d57-45bf-8143-c82c14bd43fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0a04e0-785685977-slvzx"
Nov 02 03:37:04.517743 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:04.517726    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95349ebe-7d57-45bf-8143-c82c14bd43fa-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-0a04e0-785685977-slvzx\" (UID: \"95349ebe-7d57-45bf-8143-c82c14bd43fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0a04e0-785685977-slvzx"
Nov 02 03:37:04.518961 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:04.518936    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-0a04e0-serving-cert\""
Nov 02 03:37:04.519069 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:04.518981    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-pz7wg\""
Nov 02 03:37:04.519069 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:04.518984    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 02 03:37:04.619334 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:04.619300    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95349ebe-7d57-45bf-8143-c82c14bd43fa-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-0a04e0-785685977-slvzx\" (UID: \"95349ebe-7d57-45bf-8143-c82c14bd43fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0a04e0-785685977-slvzx"
Nov 02 03:37:04.619746 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:04.619342    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95349ebe-7d57-45bf-8143-c82c14bd43fa-maas-api-tls\") pod \"maas-api-e2e-praxis-a-0a04e0-785685977-slvzx\" (UID: \"95349ebe-7d57-45bf-8143-c82c14bd43fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0a04e0-785685977-slvzx"
Nov 02 03:37:04.619746 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:04.619444    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c62hm\" (UniqueName: \"kubernetes.io/projected/95349ebe-7d57-45bf-8143-c82c14bd43fa-kube-api-access-c62hm\") pod \"maas-api-e2e-praxis-a-0a04e0-785685977-slvzx\" (UID: \"95349ebe-7d57-45bf-8143-c82c14bd43fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0a04e0-785685977-slvzx"
Nov 02 03:37:04.619746 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:04.619470    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95349ebe-7d57-45bf-8143-c82c14bd43fa-metrics-certs\") pod \"maas-api-e2e-praxis-a-0a04e0-785685977-slvzx\" (UID: \"95349ebe-7d57-45bf-8143-c82c14bd43fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0a04e0-785685977-slvzx"
Nov 02 03:37:04.619746 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:37:04.619524    4496 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-0a04e0-serving-cert: secret "maas-api-e2e-praxis-a-0a04e0-serving-cert" not found
Nov 02 03:37:04.619746 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:37:04.619602    4496 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95349ebe-7d57-45bf-8143-c82c14bd43fa-maas-api-tls podName:95349ebe-7d57-45bf-8143-c82c14bd43fa nodeName:}" failed. No retries permitted until 2026-11-02 03:37:05.119581672 +0000 UTC m=+2965.698189053 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/95349ebe-7d57-45bf-8143-c82c14bd43fa-maas-api-tls") pod "maas-api-e2e-praxis-a-0a04e0-785685977-slvzx" (UID: "95349ebe-7d57-45bf-8143-c82c14bd43fa") : secret "maas-api-e2e-praxis-a-0a04e0-serving-cert" not found
Nov 02 03:37:04.619979 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:04.619950    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95349ebe-7d57-45bf-8143-c82c14bd43fa-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-0a04e0-785685977-slvzx\" (UID: \"95349ebe-7d57-45bf-8143-c82c14bd43fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0a04e0-785685977-slvzx"
Nov 02 03:37:04.622007 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:04.621982    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95349ebe-7d57-45bf-8143-c82c14bd43fa-metrics-certs\") pod \"maas-api-e2e-praxis-a-0a04e0-785685977-slvzx\" (UID: \"95349ebe-7d57-45bf-8143-c82c14bd43fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0a04e0-785685977-slvzx"
Nov 02 03:37:04.677824 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:04.677745    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c62hm\" (UniqueName: \"kubernetes.io/projected/95349ebe-7d57-45bf-8143-c82c14bd43fa-kube-api-access-c62hm\") pod \"maas-api-e2e-praxis-a-0a04e0-785685977-slvzx\" (UID: \"95349ebe-7d57-45bf-8143-c82c14bd43fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0a04e0-785685977-slvzx"
Nov 02 03:37:05.125477 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:05.125438    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95349ebe-7d57-45bf-8143-c82c14bd43fa-maas-api-tls\") pod \"maas-api-e2e-praxis-a-0a04e0-785685977-slvzx\" (UID: \"95349ebe-7d57-45bf-8143-c82c14bd43fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0a04e0-785685977-slvzx"
Nov 02 03:37:05.128038 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:05.128000    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95349ebe-7d57-45bf-8143-c82c14bd43fa-maas-api-tls\") pod \"maas-api-e2e-praxis-a-0a04e0-785685977-slvzx\" (UID: \"95349ebe-7d57-45bf-8143-c82c14bd43fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0a04e0-785685977-slvzx"
Nov 02 03:37:05.330753 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:05.330723    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0a04e0-785685977-slvzx"
Nov 02 03:37:05.710155 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:05.710126    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-0a04e0-785685977-slvzx"]
Nov 02 03:37:05.711106 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:37:05.711082    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95349ebe_7d57_45bf_8143_c82c14bd43fa.slice/crio-75fd863f25e17679ab40ac6659a63603e6e3ebb06eb7fc3c3878feaa8f6b0221 WatchSource:0}: Error finding container 75fd863f25e17679ab40ac6659a63603e6e3ebb06eb7fc3c3878feaa8f6b0221: Status 404 returned error can't find the container with id 75fd863f25e17679ab40ac6659a63603e6e3ebb06eb7fc3c3878feaa8f6b0221
Nov 02 03:37:05.712602 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:05.712579    4496 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 03:37:06.493835 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:06.493798    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0a04e0-785685977-slvzx" event={"ID":"95349ebe-7d57-45bf-8143-c82c14bd43fa","Type":"ContainerStarted","Data":"75fd863f25e17679ab40ac6659a63603e6e3ebb06eb7fc3c3878feaa8f6b0221"}
Nov 02 03:37:10.529428 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:10.529384    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0a04e0-785685977-slvzx" event={"ID":"95349ebe-7d57-45bf-8143-c82c14bd43fa","Type":"ContainerStarted","Data":"db0133280d738cfb5f11e6c30657fa7be80b8758c3c76fb9fa1a43a3896c0c3c"}
Nov 02 03:37:10.529817 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:10.529462    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0a04e0-785685977-slvzx"
Nov 02 03:37:10.650377 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:10.650325    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0a04e0-785685977-slvzx" podStartSLOduration=2.662852903 podStartE2EDuration="6.650311316s" totalImagesPullingTime="3.987458413s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:37:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:37:10.647186995 +0000 UTC m=+2971.225794374" watchObservedRunningTime="2026-11-02 03:37:10.650311316 +0000 UTC m=+2971.228918695"
Nov 02 03:37:15.835174 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:15.835134    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0809c8-877wj"]
Nov 02 03:37:15.884881 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:15.884849    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0809c8-877wj"]
Nov 02 03:37:16.543090 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:16.543057    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0a04e0-785685977-slvzx"
Nov 02 03:37:17.574625 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:17.574589    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cbd8a74-902a-45cf-b431-7e98f6ccd25c" path="/var/lib/kubelet/pods/4cbd8a74-902a-45cf-b431-7e98f6ccd25c/volumes"
Nov 02 03:37:28.702987 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:28.702954    4496 generic.go:356] "Generic (PLEG): container finished" podID="c01165ba-6eca-4e2a-81b1-c743acb0fbd6" containerID="534e1c32f59035b693638a67a1c0bab60c38914d4ffe15952783556e06b7fdb4" exitCode=0
Nov 02 03:37:28.703434 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:28.703051    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-e44998-openshift-default-6bfcf667b5-mwgmh" event={"ID":"c01165ba-6eca-4e2a-81b1-c743acb0fbd6","Type":"ContainerDied","Data":"534e1c32f59035b693638a67a1c0bab60c38914d4ffe15952783556e06b7fdb4"}
Nov 02 03:37:28.703485 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:28.703465    4496 scope.go:122] "RemoveContainer" containerID="534e1c32f59035b693638a67a1c0bab60c38914d4ffe15952783556e06b7fdb4"
Nov 02 03:37:28.704926 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:28.704899    4496 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 03:37:28.704992 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:28.704965    4496 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 03:37:28.705062 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:28.704995    4496 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 03:37:29.712256 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:29.712222    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-e44998-openshift-default-6bfcf667b5-mwgmh" event={"ID":"c01165ba-6eca-4e2a-81b1-c743acb0fbd6","Type":"ContainerStarted","Data":"f9bbcbb98fc8a0dd9df9bc6b8731d1e9fb4c9452a7fd70684ab0e3f4f4cb0f36"}
Nov 02 03:37:29.774897 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:29.774854    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-e44998-openshift-default-6bfcf667b5-mwgmh"
Nov 02 03:37:30.056181 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:30.056082    4496 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-e44998-openshift-default-6bfcf667b5-mwgmh"
Nov 02 03:37:30.059313 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:30.059271    4496 patch_prober.go:29] interesting pod/e2e-praxis-a-e44998-openshift-default-6bfcf667b5-mwgmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body=
Nov 02 03:37:30.059486 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:30.059354    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-e44998-openshift-default-6bfcf667b5-mwgmh" podUID="c01165ba-6eca-4e2a-81b1-c743acb0fbd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused"
Nov 02 03:37:31.057040 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:31.052374    4496 patch_prober.go:29] interesting pod/e2e-praxis-a-e44998-openshift-default-6bfcf667b5-mwgmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body=
Nov 02 03:37:31.057040 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:31.052441    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-e44998-openshift-default-6bfcf667b5-mwgmh" podUID="c01165ba-6eca-4e2a-81b1-c743acb0fbd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused"
Nov 02 03:37:32.056038 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:32.055342    4496 patch_prober.go:29] interesting pod/e2e-praxis-a-e44998-openshift-default-6bfcf667b5-mwgmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body=
Nov 02 03:37:32.056038 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:32.055406    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-e44998-openshift-default-6bfcf667b5-mwgmh" podUID="c01165ba-6eca-4e2a-81b1-c743acb0fbd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused"
Nov 02 03:37:32.861083 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:32.860232    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-167a01-88474547-k8xhk"]
Nov 02 03:37:32.868255 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:32.868234    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-167a01-88474547-k8xhk"
Nov 02 03:37:32.897461 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:32.897428    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-167a01-88474547-k8xhk"]
Nov 02 03:37:32.910672 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:32.910639    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdwpg\" (UniqueName: \"kubernetes.io/projected/c0e67481-3663-4f05-aff8-3a38bc7cc8c5-kube-api-access-mdwpg\") pod \"payload-pre-processing-e2e-praxis-a-167a01-88474547-k8xhk\" (UID: \"c0e67481-3663-4f05-aff8-3a38bc7cc8c5\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-167a01-88474547-k8xhk"
Nov 02 03:37:32.910834 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:32.910807    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0e67481-3663-4f05-aff8-3a38bc7cc8c5-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-167a01-88474547-k8xhk\" (UID: \"c0e67481-3663-4f05-aff8-3a38bc7cc8c5\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-167a01-88474547-k8xhk"
Nov 02 03:37:32.951035 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:32.944296    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-167a01\""
Nov 02 03:37:32.951035 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:32.944438    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-167a01-dockercfg-ccvkc\""
Nov 02 03:37:32.978789 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:32.978757    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-167a01-7b99c886bc-k8rsp"]
Nov 02 03:37:32.983403 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:32.983383    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-167a01-7b99c886bc-k8rsp"
Nov 02 03:37:33.012080 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:33.012052    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vcll\" (UniqueName: \"kubernetes.io/projected/3524ec44-5fdd-4669-b169-99b1d1f4f353-kube-api-access-9vcll\") pod \"payload-processing-e2e-praxis-a-167a01-7b99c886bc-k8rsp\" (UID: \"3524ec44-5fdd-4669-b169-99b1d1f4f353\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-167a01-7b99c886bc-k8rsp"
Nov 02 03:37:33.012215 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:33.012114    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdwpg\" (UniqueName: \"kubernetes.io/projected/c0e67481-3663-4f05-aff8-3a38bc7cc8c5-kube-api-access-mdwpg\") pod \"payload-pre-processing-e2e-praxis-a-167a01-88474547-k8xhk\" (UID: \"c0e67481-3663-4f05-aff8-3a38bc7cc8c5\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-167a01-88474547-k8xhk"
Nov 02 03:37:33.012268 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:33.012216    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3524ec44-5fdd-4669-b169-99b1d1f4f353-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-167a01-7b99c886bc-k8rsp\" (UID: \"3524ec44-5fdd-4669-b169-99b1d1f4f353\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-167a01-7b99c886bc-k8rsp"
Nov 02 03:37:33.012341 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:33.012320    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0e67481-3663-4f05-aff8-3a38bc7cc8c5-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-167a01-88474547-k8xhk\" (UID: \"c0e67481-3663-4f05-aff8-3a38bc7cc8c5\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-167a01-88474547-k8xhk"
Nov 02 03:37:33.012985 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:33.012968    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0e67481-3663-4f05-aff8-3a38bc7cc8c5-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-167a01-88474547-k8xhk\" (UID: \"c0e67481-3663-4f05-aff8-3a38bc7cc8c5\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-167a01-88474547-k8xhk"
Nov 02 03:37:33.032277 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:33.032253    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-167a01-7b99c886bc-k8rsp"]
Nov 02 03:37:33.064416 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:33.064379    4496 patch_prober.go:29] interesting pod/e2e-praxis-a-e44998-openshift-default-6bfcf667b5-mwgmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body=
Nov 02 03:37:33.064697 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:33.064666    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-e44998-openshift-default-6bfcf667b5-mwgmh" podUID="c01165ba-6eca-4e2a-81b1-c743acb0fbd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused"
Nov 02 03:37:33.070075 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:33.069623    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdwpg\" (UniqueName: \"kubernetes.io/projected/c0e67481-3663-4f05-aff8-3a38bc7cc8c5-kube-api-access-mdwpg\") pod \"payload-pre-processing-e2e-praxis-a-167a01-88474547-k8xhk\" (UID: \"c0e67481-3663-4f05-aff8-3a38bc7cc8c5\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-167a01-88474547-k8xhk"
Nov 02 03:37:33.113337 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:33.113260    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3524ec44-5fdd-4669-b169-99b1d1f4f353-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-167a01-7b99c886bc-k8rsp\" (UID: \"3524ec44-5fdd-4669-b169-99b1d1f4f353\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-167a01-7b99c886bc-k8rsp"
Nov 02 03:37:33.113500 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:33.113383    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vcll\" (UniqueName: \"kubernetes.io/projected/3524ec44-5fdd-4669-b169-99b1d1f4f353-kube-api-access-9vcll\") pod \"payload-processing-e2e-praxis-a-167a01-7b99c886bc-k8rsp\" (UID: \"3524ec44-5fdd-4669-b169-99b1d1f4f353\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-167a01-7b99c886bc-k8rsp"
Nov 02 03:37:33.114044 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:33.113989    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3524ec44-5fdd-4669-b169-99b1d1f4f353-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-167a01-7b99c886bc-k8rsp\" (UID: \"3524ec44-5fdd-4669-b169-99b1d1f4f353\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-167a01-7b99c886bc-k8rsp"
Nov 02 03:37:33.171321 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:33.171279    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vcll\" (UniqueName: \"kubernetes.io/projected/3524ec44-5fdd-4669-b169-99b1d1f4f353-kube-api-access-9vcll\") pod \"payload-processing-e2e-praxis-a-167a01-7b99c886bc-k8rsp\" (UID: \"3524ec44-5fdd-4669-b169-99b1d1f4f353\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-167a01-7b99c886bc-k8rsp"
Nov 02 03:37:33.183238 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:33.183208    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-167a01-88474547-k8xhk"
Nov 02 03:37:33.299528 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:33.299501    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-167a01-7b99c886bc-k8rsp"
Nov 02 03:37:33.533001 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:37:33.532959    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 03:37:33.589058 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:33.589032    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-167a01-88474547-k8xhk"]
Nov 02 03:37:33.589679 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:37:33.589650    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0e67481_3663_4f05_aff8_3a38bc7cc8c5.slice/crio-e970409f741661308f3bdb7189181ab55257d4d42ec4609c6d1acef93a1d2180 WatchSource:0}: Error finding container e970409f741661308f3bdb7189181ab55257d4d42ec4609c6d1acef93a1d2180: Status 404 returned error can't find the container with id e970409f741661308f3bdb7189181ab55257d4d42ec4609c6d1acef93a1d2180
Nov 02 03:37:33.720584 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:33.720552    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-167a01-7b99c886bc-k8rsp"]
Nov 02 03:37:33.720964 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:37:33.720942    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3524ec44_5fdd_4669_b169_99b1d1f4f353.slice/crio-f6aa835a9ec4029af1133199c27ba2bd1bee831e2d8d90b4c4b7e603faf861f9 WatchSource:0}: Error finding container f6aa835a9ec4029af1133199c27ba2bd1bee831e2d8d90b4c4b7e603faf861f9: Status 404 returned error can't find the container with id f6aa835a9ec4029af1133199c27ba2bd1bee831e2d8d90b4c4b7e603faf861f9
Nov 02 03:37:33.785883 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:33.785848    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-167a01-7b99c886bc-k8rsp" event={"ID":"3524ec44-5fdd-4669-b169-99b1d1f4f353","Type":"ContainerStarted","Data":"f6aa835a9ec4029af1133199c27ba2bd1bee831e2d8d90b4c4b7e603faf861f9"}
Nov 02 03:37:33.787803 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:33.787774    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-167a01-88474547-k8xhk" event={"ID":"c0e67481-3663-4f05-aff8-3a38bc7cc8c5","Type":"ContainerStarted","Data":"829189e483e18ce71d90b877a03c32142ba96a4d6f89aa8e2db51aac759f9869"}
Nov 02 03:37:33.787917 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:33.787831    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-167a01-88474547-k8xhk" event={"ID":"c0e67481-3663-4f05-aff8-3a38bc7cc8c5","Type":"ContainerStarted","Data":"e970409f741661308f3bdb7189181ab55257d4d42ec4609c6d1acef93a1d2180"}
Nov 02 03:37:33.787917 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:33.787889    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-167a01-88474547-k8xhk"
Nov 02 03:37:33.904159 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:33.903642    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-167a01-88474547-k8xhk" podStartSLOduration=1.903624454 podStartE2EDuration="1.903624454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:37:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:37:33.902383405 +0000 UTC m=+2994.480990785" watchObservedRunningTime="2026-11-02 03:37:33.903624454 +0000 UTC m=+2994.482231835"
Nov 02 03:37:34.058606 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:34.058554    4496 patch_prober.go:29] interesting pod/e2e-praxis-a-e44998-openshift-default-6bfcf667b5-mwgmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body=
Nov 02 03:37:34.058831 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:34.058617    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-e44998-openshift-default-6bfcf667b5-mwgmh" podUID="c01165ba-6eca-4e2a-81b1-c743acb0fbd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused"
Nov 02 03:37:34.795767 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:34.795729    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-167a01-7b99c886bc-k8rsp" event={"ID":"3524ec44-5fdd-4669-b169-99b1d1f4f353","Type":"ContainerStarted","Data":"eb25bae3e2c93486a2bd913199cda2e4639d25363952aa4935ba874e6b5cd38b"}
Nov 02 03:37:34.795957 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:34.795809    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-167a01-7b99c886bc-k8rsp"
Nov 02 03:37:34.916087 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:34.916025    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-167a01-7b99c886bc-k8rsp" podStartSLOduration=2.915996871 podStartE2EDuration="2.915996871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:37:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:37:34.914425442 +0000 UTC m=+2995.493032823" watchObservedRunningTime="2026-11-02 03:37:34.915996871 +0000 UTC m=+2995.494604244"
Nov 02 03:37:35.066105 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:35.065970    4496 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-e44998-openshift-default-6bfcf667b5-mwgmh"
Nov 02 03:37:35.067187 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:35.067162    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-e44998-openshift-default-6bfcf667b5-mwgmh"
Nov 02 03:37:35.791911 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:35.791877    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d94cc3-874bc47d6-hs5kd"]
Nov 02 03:37:35.798600 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:35.798581    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d94cc3-874bc47d6-hs5kd"
Nov 02 03:37:35.851710 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:35.851667    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-d94cc3-serving-cert\""
Nov 02 03:37:35.853555 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:35.853527    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d94cc3-874bc47d6-hs5kd"]
Nov 02 03:37:35.873195 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:35.873171    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32681665-956b-4346-b466-f7257f9ba39f-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-d94cc3-874bc47d6-hs5kd\" (UID: \"32681665-956b-4346-b466-f7257f9ba39f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d94cc3-874bc47d6-hs5kd"
Nov 02 03:37:35.873313 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:35.873276    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4zcb\" (UniqueName: \"kubernetes.io/projected/32681665-956b-4346-b466-f7257f9ba39f-kube-api-access-s4zcb\") pod \"maas-api-e2e-shared-b-w2-d94cc3-874bc47d6-hs5kd\" (UID: \"32681665-956b-4346-b466-f7257f9ba39f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d94cc3-874bc47d6-hs5kd"
Nov 02 03:37:35.873519 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:35.873501    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32681665-956b-4346-b466-f7257f9ba39f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-d94cc3-874bc47d6-hs5kd\" (UID: \"32681665-956b-4346-b466-f7257f9ba39f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d94cc3-874bc47d6-hs5kd"
Nov 02 03:37:35.873707 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:35.873685    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32681665-956b-4346-b466-f7257f9ba39f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-d94cc3-874bc47d6-hs5kd\" (UID: \"32681665-956b-4346-b466-f7257f9ba39f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d94cc3-874bc47d6-hs5kd"
Nov 02 03:37:35.975403 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:35.975364    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32681665-956b-4346-b466-f7257f9ba39f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-d94cc3-874bc47d6-hs5kd\" (UID: \"32681665-956b-4346-b466-f7257f9ba39f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d94cc3-874bc47d6-hs5kd"
Nov 02 03:37:35.975869 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:35.975455    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32681665-956b-4346-b466-f7257f9ba39f-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-d94cc3-874bc47d6-hs5kd\" (UID: \"32681665-956b-4346-b466-f7257f9ba39f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d94cc3-874bc47d6-hs5kd"
Nov 02 03:37:35.975869 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:35.975512    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4zcb\" (UniqueName: \"kubernetes.io/projected/32681665-956b-4346-b466-f7257f9ba39f-kube-api-access-s4zcb\") pod \"maas-api-e2e-shared-b-w2-d94cc3-874bc47d6-hs5kd\" (UID: \"32681665-956b-4346-b466-f7257f9ba39f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d94cc3-874bc47d6-hs5kd"
Nov 02 03:37:35.975869 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:35.975610    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32681665-956b-4346-b466-f7257f9ba39f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-d94cc3-874bc47d6-hs5kd\" (UID: \"32681665-956b-4346-b466-f7257f9ba39f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d94cc3-874bc47d6-hs5kd"
Nov 02 03:37:35.976140 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:35.976118    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32681665-956b-4346-b466-f7257f9ba39f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-d94cc3-874bc47d6-hs5kd\" (UID: \"32681665-956b-4346-b466-f7257f9ba39f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d94cc3-874bc47d6-hs5kd"
Nov 02 03:37:35.978086 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:35.978062    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32681665-956b-4346-b466-f7257f9ba39f-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-d94cc3-874bc47d6-hs5kd\" (UID: \"32681665-956b-4346-b466-f7257f9ba39f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d94cc3-874bc47d6-hs5kd"
Nov 02 03:37:35.978195 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:35.978099    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32681665-956b-4346-b466-f7257f9ba39f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-d94cc3-874bc47d6-hs5kd\" (UID: \"32681665-956b-4346-b466-f7257f9ba39f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d94cc3-874bc47d6-hs5kd"
Nov 02 03:37:36.032949 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:36.032921    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4zcb\" (UniqueName: \"kubernetes.io/projected/32681665-956b-4346-b466-f7257f9ba39f-kube-api-access-s4zcb\") pod \"maas-api-e2e-shared-b-w2-d94cc3-874bc47d6-hs5kd\" (UID: \"32681665-956b-4346-b466-f7257f9ba39f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d94cc3-874bc47d6-hs5kd"
Nov 02 03:37:36.114130 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:36.114054    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d94cc3-874bc47d6-hs5kd"
Nov 02 03:37:36.501437 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:36.501406    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d94cc3-874bc47d6-hs5kd"]
Nov 02 03:37:36.502280 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:37:36.502253    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32681665_956b_4346_b466_f7257f9ba39f.slice/crio-22e23b6eae0dff7374e01a356010a1f0ea74139068cd8ab3dc1a64adf04c9ab3 WatchSource:0}: Error finding container 22e23b6eae0dff7374e01a356010a1f0ea74139068cd8ab3dc1a64adf04c9ab3: Status 404 returned error can't find the container with id 22e23b6eae0dff7374e01a356010a1f0ea74139068cd8ab3dc1a64adf04c9ab3
Nov 02 03:37:36.823610 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:36.823522    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d94cc3-874bc47d6-hs5kd" event={"ID":"32681665-956b-4346-b466-f7257f9ba39f","Type":"ContainerStarted","Data":"22e23b6eae0dff7374e01a356010a1f0ea74139068cd8ab3dc1a64adf04c9ab3"}
Nov 02 03:37:37.718477 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:37.718399    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-018375-7vfzp"]
Nov 02 03:37:37.722732 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:37.722714    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-018375-7vfzp"
Nov 02 03:37:37.778387 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:37.778359    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-018375-7vfzp"]
Nov 02 03:37:37.794384 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:37.794357    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e60d498-1189-41c5-9548-243f2d01a40e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-018375-7vfzp\" (UID: \"5e60d498-1189-41c5-9548-243f2d01a40e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-018375-7vfzp"
Nov 02 03:37:37.794554 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:37.794414    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96s2s\" (UniqueName: \"kubernetes.io/projected/5e60d498-1189-41c5-9548-243f2d01a40e-kube-api-access-96s2s\") pod \"maas-api-revoke-keys-e2e-ipp-b-018375-7vfzp\" (UID: \"5e60d498-1189-41c5-9548-243f2d01a40e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-018375-7vfzp"
Nov 02 03:37:37.794554 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:37.794492    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e60d498-1189-41c5-9548-243f2d01a40e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-018375-7vfzp\" (UID: \"5e60d498-1189-41c5-9548-243f2d01a40e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-018375-7vfzp"
Nov 02 03:37:37.825279 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:37.825253    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 03:37:37.825279 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:37.825261    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9z5bk\""
Nov 02 03:37:37.831951 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:37.831924    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d94cc3-874bc47d6-hs5kd" event={"ID":"32681665-956b-4346-b466-f7257f9ba39f","Type":"ContainerStarted","Data":"84e512aca4cee294e5d1ef6e7141592687fb9ff6b4adcf1f8af4434e0e9772d8"}
Nov 02 03:37:37.832090 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:37.832066    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d94cc3-874bc47d6-hs5kd"
Nov 02 03:37:37.895140 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:37.895110    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e60d498-1189-41c5-9548-243f2d01a40e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-018375-7vfzp\" (UID: \"5e60d498-1189-41c5-9548-243f2d01a40e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-018375-7vfzp"
Nov 02 03:37:37.895300 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:37.895189    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96s2s\" (UniqueName: \"kubernetes.io/projected/5e60d498-1189-41c5-9548-243f2d01a40e-kube-api-access-96s2s\") pod \"maas-api-revoke-keys-e2e-ipp-b-018375-7vfzp\" (UID: \"5e60d498-1189-41c5-9548-243f2d01a40e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-018375-7vfzp"
Nov 02 03:37:37.895300 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:37.895222    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e60d498-1189-41c5-9548-243f2d01a40e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-018375-7vfzp\" (UID: \"5e60d498-1189-41c5-9548-243f2d01a40e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-018375-7vfzp"
Nov 02 03:37:37.896059 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:37.896035    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e60d498-1189-41c5-9548-243f2d01a40e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-018375-7vfzp\" (UID: \"5e60d498-1189-41c5-9548-243f2d01a40e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-018375-7vfzp"
Nov 02 03:37:37.897697 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:37.897675    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e60d498-1189-41c5-9548-243f2d01a40e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-018375-7vfzp\" (UID: \"5e60d498-1189-41c5-9548-243f2d01a40e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-018375-7vfzp"
Nov 02 03:37:37.952796 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:37.952745    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d94cc3-874bc47d6-hs5kd" podStartSLOduration=2.104976332 podStartE2EDuration="2.952731603s" totalImagesPullingTime="847.755271ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:37:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:37:37.950776312 +0000 UTC m=+2998.529383707" watchObservedRunningTime="2026-11-02 03:37:37.952731603 +0000 UTC m=+2998.531338983"
Nov 02 03:37:37.955290 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:37.955266    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96s2s\" (UniqueName: \"kubernetes.io/projected/5e60d498-1189-41c5-9548-243f2d01a40e-kube-api-access-96s2s\") pod \"maas-api-revoke-keys-e2e-ipp-b-018375-7vfzp\" (UID: \"5e60d498-1189-41c5-9548-243f2d01a40e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-018375-7vfzp"
Nov 02 03:37:38.035804 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:38.035770    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-018375-7vfzp"
Nov 02 03:37:38.414318 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:37:38.414286    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e60d498_1189_41c5_9548_243f2d01a40e.slice/crio-2007fd75f386df8cde0ba3b1e4a9c2ea62e0f34d5ea69f5b950cb8083c340c8c WatchSource:0}: Error finding container 2007fd75f386df8cde0ba3b1e4a9c2ea62e0f34d5ea69f5b950cb8083c340c8c: Status 404 returned error can't find the container with id 2007fd75f386df8cde0ba3b1e4a9c2ea62e0f34d5ea69f5b950cb8083c340c8c
Nov 02 03:37:38.418087 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:38.418060    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-018375-7vfzp"]
Nov 02 03:37:38.844160 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:38.844108    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-018375-7vfzp" event={"ID":"5e60d498-1189-41c5-9548-243f2d01a40e","Type":"ContainerStarted","Data":"2007fd75f386df8cde0ba3b1e4a9c2ea62e0f34d5ea69f5b950cb8083c340c8c"}
Nov 02 03:37:39.853168 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:39.853076    4496 generic.go:356] "Generic (PLEG): container finished" podID="5e60d498-1189-41c5-9548-243f2d01a40e" containerID="e4e5e80a07d121ebbf791a58e464dd7e84a91c7395fdc447c3cd12492f308045" exitCode=0
Nov 02 03:37:39.853583 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:39.853163    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-018375-7vfzp" event={"ID":"5e60d498-1189-41c5-9548-243f2d01a40e","Type":"ContainerDied","Data":"e4e5e80a07d121ebbf791a58e464dd7e84a91c7395fdc447c3cd12492f308045"}
Nov 02 03:37:40.213197 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:40.213112    4496 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hww8q_5c797b08-a314-44ed-8026-8a0ce1f6c36f/ovn-acl-logging/0.log"
Nov 02 03:37:40.221223 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:40.221201    4496 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hww8q_5c797b08-a314-44ed-8026-8a0ce1f6c36f/ovn-acl-logging/0.log"
Nov 02 03:37:41.028418 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:41.028384    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-018375-7vfzp"
Nov 02 03:37:41.132392 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:41.132294    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e60d498-1189-41c5-9548-243f2d01a40e-kube-api-access-96s2s\" (UniqueName: \"kubernetes.io/projected/5e60d498-1189-41c5-9548-243f2d01a40e-kube-api-access-96s2s\") pod \"5e60d498-1189-41c5-9548-243f2d01a40e\" (UID: \"5e60d498-1189-41c5-9548-243f2d01a40e\") "
Nov 02 03:37:41.132563 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:41.132412    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e60d498-1189-41c5-9548-243f2d01a40e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e60d498-1189-41c5-9548-243f2d01a40e-maas-api-openshift-ca\") pod \"5e60d498-1189-41c5-9548-243f2d01a40e\" (UID: \"5e60d498-1189-41c5-9548-243f2d01a40e\") "
Nov 02 03:37:41.132563 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:41.132526    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e60d498-1189-41c5-9548-243f2d01a40e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e60d498-1189-41c5-9548-243f2d01a40e-maas-api-xks-ca\") pod \"5e60d498-1189-41c5-9548-243f2d01a40e\" (UID: \"5e60d498-1189-41c5-9548-243f2d01a40e\") "
Nov 02 03:37:41.132760 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:41.132736    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e60d498-1189-41c5-9548-243f2d01a40e-maas-api-openshift-ca" pod "5e60d498-1189-41c5-9548-243f2d01a40e" (UID: "5e60d498-1189-41c5-9548-243f2d01a40e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:37:41.132878 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:41.132860    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e60d498-1189-41c5-9548-243f2d01a40e-maas-api-openshift-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:37:41.134469 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:41.134444    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e60d498-1189-41c5-9548-243f2d01a40e-maas-api-xks-ca" pod "5e60d498-1189-41c5-9548-243f2d01a40e" (UID: "5e60d498-1189-41c5-9548-243f2d01a40e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 03:37:41.134880 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:41.134863    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e60d498-1189-41c5-9548-243f2d01a40e-kube-api-access-96s2s" pod "5e60d498-1189-41c5-9548-243f2d01a40e" (UID: "5e60d498-1189-41c5-9548-243f2d01a40e"). InnerVolumeSpecName "kube-api-access-96s2s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:37:41.234158 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:41.234125    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e60d498-1189-41c5-9548-243f2d01a40e-maas-api-xks-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:37:41.234333 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:41.234175    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96s2s\" (UniqueName: \"kubernetes.io/projected/5e60d498-1189-41c5-9548-243f2d01a40e-kube-api-access-96s2s\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:37:41.679656 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:41.679625    4496 scope.go:122] "RemoveContainer" containerID="b3c851a75a5cecf757f3f524b73730afa22f48e8df07fab14743883a0b03a4a4"
Nov 02 03:37:41.690289 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:41.690221    4496 scope.go:122] "RemoveContainer" containerID="aac907948cbf751d304b8be7b502ff49f94fe9c1c22fa067ca2255155dbb3efa"
Nov 02 03:37:41.869754 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:41.869672    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-018375-7vfzp"
Nov 02 03:37:41.869754 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:41.869697    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-018375-7vfzp" event={"ID":"5e60d498-1189-41c5-9548-243f2d01a40e","Type":"ContainerDied","Data":"2007fd75f386df8cde0ba3b1e4a9c2ea62e0f34d5ea69f5b950cb8083c340c8c"}
Nov 02 03:37:41.869936 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:41.869755    4496 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2007fd75f386df8cde0ba3b1e4a9c2ea62e0f34d5ea69f5b950cb8083c340c8c"
Nov 02 03:37:43.850150 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:43.850116    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d94cc3-874bc47d6-hs5kd"
Nov 02 03:37:44.799702 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:44.799675    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-167a01-88474547-k8xhk"
Nov 02 03:37:45.808588 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:45.808556    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-167a01-7b99c886bc-k8rsp"
Nov 02 03:37:48.147462 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:48.147421    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29ea31d1-clkzv"]
Nov 02 03:37:48.148073 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:48.148056    4496 cpu_manager.go:414] "containerMap: removing container" podUID="5e60d498-1189-41c5-9548-243f2d01a40e" containerName="revoke-keys"
Nov 02 03:37:48.148073 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:48.148072    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e60d498-1189-41c5-9548-243f2d01a40e" containerName="revoke-keys"
Nov 02 03:37:48.148195 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:48.148179    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e60d498-1189-41c5-9548-243f2d01a40e" containerName="revoke-keys"
Nov 02 03:37:48.151570 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:48.151554    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29ea31d1-clkzv"
Nov 02 03:37:48.207484 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:48.207459    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29ea31d1-clkzv"]
Nov 02 03:37:48.254060 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:48.253997    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 03:37:48.254218 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:48.254064    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9z5bk\""
Nov 02 03:37:48.315936 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:48.315902    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ab84d1d-22c4-40a8-b0d5-9e78a4f8923a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29ea31d1-clkzv\" (UID: \"7ab84d1d-22c4-40a8-b0d5-9e78a4f8923a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29ea31d1-clkzv"
Nov 02 03:37:48.316155 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:48.316089    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpk5l\" (UniqueName: \"kubernetes.io/projected/7ab84d1d-22c4-40a8-b0d5-9e78a4f8923a-kube-api-access-hpk5l\") pod \"maas-api-revoke-sub-29ea31d1-clkzv\" (UID: \"7ab84d1d-22c4-40a8-b0d5-9e78a4f8923a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29ea31d1-clkzv"
Nov 02 03:37:48.316218 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:48.316191    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ab84d1d-22c4-40a8-b0d5-9e78a4f8923a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29ea31d1-clkzv\" (UID: \"7ab84d1d-22c4-40a8-b0d5-9e78a4f8923a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29ea31d1-clkzv"
Nov 02 03:37:48.417186 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:48.417092    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ab84d1d-22c4-40a8-b0d5-9e78a4f8923a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29ea31d1-clkzv\" (UID: \"7ab84d1d-22c4-40a8-b0d5-9e78a4f8923a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29ea31d1-clkzv"
Nov 02 03:37:48.417354 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:48.417205    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpk5l\" (UniqueName: \"kubernetes.io/projected/7ab84d1d-22c4-40a8-b0d5-9e78a4f8923a-kube-api-access-hpk5l\") pod \"maas-api-revoke-sub-29ea31d1-clkzv\" (UID: \"7ab84d1d-22c4-40a8-b0d5-9e78a4f8923a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29ea31d1-clkzv"
Nov 02 03:37:48.417354 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:48.417262    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ab84d1d-22c4-40a8-b0d5-9e78a4f8923a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29ea31d1-clkzv\" (UID: \"7ab84d1d-22c4-40a8-b0d5-9e78a4f8923a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29ea31d1-clkzv"
Nov 02 03:37:48.417814 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:48.417781    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ab84d1d-22c4-40a8-b0d5-9e78a4f8923a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29ea31d1-clkzv\" (UID: \"7ab84d1d-22c4-40a8-b0d5-9e78a4f8923a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29ea31d1-clkzv"
Nov 02 03:37:48.419727 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:48.419704    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ab84d1d-22c4-40a8-b0d5-9e78a4f8923a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29ea31d1-clkzv\" (UID: \"7ab84d1d-22c4-40a8-b0d5-9e78a4f8923a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29ea31d1-clkzv"
Nov 02 03:37:48.474826 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:48.474793    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpk5l\" (UniqueName: \"kubernetes.io/projected/7ab84d1d-22c4-40a8-b0d5-9e78a4f8923a-kube-api-access-hpk5l\") pod \"maas-api-revoke-sub-29ea31d1-clkzv\" (UID: \"7ab84d1d-22c4-40a8-b0d5-9e78a4f8923a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29ea31d1-clkzv"
Nov 02 03:37:48.765456 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:48.765418    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29ea31d1-clkzv"
Nov 02 03:37:49.170286 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:49.170246    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29ea31d1-clkzv"]
Nov 02 03:37:49.171305 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:37:49.171276    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ab84d1d_22c4_40a8_b0d5_9e78a4f8923a.slice/crio-d41f2bba541590a717818d561cc124815af3976b389a8fb9dc5970103f796663 WatchSource:0}: Error finding container d41f2bba541590a717818d561cc124815af3976b389a8fb9dc5970103f796663: Status 404 returned error can't find the container with id d41f2bba541590a717818d561cc124815af3976b389a8fb9dc5970103f796663
Nov 02 03:37:49.931731 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:49.931690    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29ea31d1-clkzv" event={"ID":"7ab84d1d-22c4-40a8-b0d5-9e78a4f8923a","Type":"ContainerStarted","Data":"42da0b367f9a66076cf07827b82ce898f019f3ec6f0131275503cbe3cc9b05d7"}
Nov 02 03:37:49.931948 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:49.931759    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29ea31d1-clkzv" event={"ID":"7ab84d1d-22c4-40a8-b0d5-9e78a4f8923a","Type":"ContainerStarted","Data":"d41f2bba541590a717818d561cc124815af3976b389a8fb9dc5970103f796663"}
Nov 02 03:37:50.048434 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:50.048382    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29ea31d1-clkzv" podStartSLOduration=2.048368047 podStartE2EDuration="2.048368047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:37:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:37:50.045300549 +0000 UTC m=+3010.623907932" watchObservedRunningTime="2026-11-02 03:37:50.048368047 +0000 UTC m=+3010.626975426"
Nov 02 03:37:50.941608 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:50.941568    4496 generic.go:356] "Generic (PLEG): container finished" podID="7ab84d1d-22c4-40a8-b0d5-9e78a4f8923a" containerID="42da0b367f9a66076cf07827b82ce898f019f3ec6f0131275503cbe3cc9b05d7" exitCode=0
Nov 02 03:37:50.942077 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:50.941663    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29ea31d1-clkzv" event={"ID":"7ab84d1d-22c4-40a8-b0d5-9e78a4f8923a","Type":"ContainerDied","Data":"42da0b367f9a66076cf07827b82ce898f019f3ec6f0131275503cbe3cc9b05d7"}
Nov 02 03:37:51.531937 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:51.531902    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-018375-6495ddb8c5-r44mj"]
Nov 02 03:37:51.532150 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:51.532105    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-018375-6495ddb8c5-r44mj" podUID="cdbc2cc9-af01-41e7-b360-4ab8b73a113c" containerName="payload-pre-processing" containerID="cri-o://e2fbaed9c5a3b19b864f4625edcf13d59a9ab2849c33476ffac18f2e6ed9c730" gracePeriod=30
Nov 02 03:37:51.810387 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:51.810362    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-018375-6495ddb8c5-r44mj"
Nov 02 03:37:51.855830 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:51.855796    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdbc2cc9-af01-41e7-b360-4ab8b73a113c-kube-api-access-9nqtt\" (UniqueName: \"kubernetes.io/projected/cdbc2cc9-af01-41e7-b360-4ab8b73a113c-kube-api-access-9nqtt\") pod \"cdbc2cc9-af01-41e7-b360-4ab8b73a113c\" (UID: \"cdbc2cc9-af01-41e7-b360-4ab8b73a113c\") "
Nov 02 03:37:51.856009 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:51.855978    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cdbc2cc9-af01-41e7-b360-4ab8b73a113c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdbc2cc9-af01-41e7-b360-4ab8b73a113c-plugins-config-volume\") pod \"cdbc2cc9-af01-41e7-b360-4ab8b73a113c\" (UID: \"cdbc2cc9-af01-41e7-b360-4ab8b73a113c\") "
Nov 02 03:37:51.856450 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:51.856419    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdbc2cc9-af01-41e7-b360-4ab8b73a113c-plugins-config-volume" pod "cdbc2cc9-af01-41e7-b360-4ab8b73a113c" (UID: "cdbc2cc9-af01-41e7-b360-4ab8b73a113c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:37:51.856635 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:51.856610    4496 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdbc2cc9-af01-41e7-b360-4ab8b73a113c-plugins-config-volume\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:37:51.858369 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:51.858343    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdbc2cc9-af01-41e7-b360-4ab8b73a113c-kube-api-access-9nqtt" pod "cdbc2cc9-af01-41e7-b360-4ab8b73a113c" (UID: "cdbc2cc9-af01-41e7-b360-4ab8b73a113c"). InnerVolumeSpecName "kube-api-access-9nqtt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:37:51.952044 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:51.951978    4496 generic.go:356] "Generic (PLEG): container finished" podID="cdbc2cc9-af01-41e7-b360-4ab8b73a113c" containerID="e2fbaed9c5a3b19b864f4625edcf13d59a9ab2849c33476ffac18f2e6ed9c730" exitCode=0
Nov 02 03:37:51.952044 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:51.951997    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-018375-6495ddb8c5-r44mj"
Nov 02 03:37:51.952561 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:51.952060    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-018375-6495ddb8c5-r44mj" event={"ID":"cdbc2cc9-af01-41e7-b360-4ab8b73a113c","Type":"ContainerDied","Data":"e2fbaed9c5a3b19b864f4625edcf13d59a9ab2849c33476ffac18f2e6ed9c730"}
Nov 02 03:37:51.952561 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:51.952131    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-018375-6495ddb8c5-r44mj" event={"ID":"cdbc2cc9-af01-41e7-b360-4ab8b73a113c","Type":"ContainerDied","Data":"299ace3957518c11988b52e3eaef9bffabfe01c223f67df3ec613a68364fd680"}
Nov 02 03:37:51.952561 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:51.952160    4496 scope.go:122] "RemoveContainer" containerID="e2fbaed9c5a3b19b864f4625edcf13d59a9ab2849c33476ffac18f2e6ed9c730"
Nov 02 03:37:51.957795 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:51.957756    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nqtt\" (UniqueName: \"kubernetes.io/projected/cdbc2cc9-af01-41e7-b360-4ab8b73a113c-kube-api-access-9nqtt\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:37:51.967561 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:51.967538    4496 scope.go:122] "RemoveContainer" containerID="e2fbaed9c5a3b19b864f4625edcf13d59a9ab2849c33476ffac18f2e6ed9c730"
Nov 02 03:37:51.967882 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:37:51.967855    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2fbaed9c5a3b19b864f4625edcf13d59a9ab2849c33476ffac18f2e6ed9c730\": container with ID starting with e2fbaed9c5a3b19b864f4625edcf13d59a9ab2849c33476ffac18f2e6ed9c730 not found: ID does not exist" containerID="e2fbaed9c5a3b19b864f4625edcf13d59a9ab2849c33476ffac18f2e6ed9c730"
Nov 02 03:37:51.967958 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:51.967886    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2fbaed9c5a3b19b864f4625edcf13d59a9ab2849c33476ffac18f2e6ed9c730"} err="failed to get container status \"e2fbaed9c5a3b19b864f4625edcf13d59a9ab2849c33476ffac18f2e6ed9c730\": rpc error: code = NotFound desc = could not find container \"e2fbaed9c5a3b19b864f4625edcf13d59a9ab2849c33476ffac18f2e6ed9c730\": container with ID starting with e2fbaed9c5a3b19b864f4625edcf13d59a9ab2849c33476ffac18f2e6ed9c730 not found: ID does not exist"
Nov 02 03:37:52.119883 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:52.119856    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29ea31d1-clkzv"
Nov 02 03:37:52.122041 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:52.121988    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-018375-6495ddb8c5-r44mj"]
Nov 02 03:37:52.127128 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:52.127107    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-018375-6495ddb8c5-r44mj"]
Nov 02 03:37:52.159466 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:52.159437    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ab84d1d-22c4-40a8-b0d5-9e78a4f8923a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ab84d1d-22c4-40a8-b0d5-9e78a4f8923a-maas-api-xks-ca\") pod \"7ab84d1d-22c4-40a8-b0d5-9e78a4f8923a\" (UID: \"7ab84d1d-22c4-40a8-b0d5-9e78a4f8923a\") "
Nov 02 03:37:52.159694 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:52.159668    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ab84d1d-22c4-40a8-b0d5-9e78a4f8923a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ab84d1d-22c4-40a8-b0d5-9e78a4f8923a-maas-api-openshift-ca\") pod \"7ab84d1d-22c4-40a8-b0d5-9e78a4f8923a\" (UID: \"7ab84d1d-22c4-40a8-b0d5-9e78a4f8923a\") "
Nov 02 03:37:52.159805 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:52.159788    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ab84d1d-22c4-40a8-b0d5-9e78a4f8923a-kube-api-access-hpk5l\" (UniqueName: \"kubernetes.io/projected/7ab84d1d-22c4-40a8-b0d5-9e78a4f8923a-kube-api-access-hpk5l\") pod \"7ab84d1d-22c4-40a8-b0d5-9e78a4f8923a\" (UID: \"7ab84d1d-22c4-40a8-b0d5-9e78a4f8923a\") "
Nov 02 03:37:52.160039 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:52.160004    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ab84d1d-22c4-40a8-b0d5-9e78a4f8923a-maas-api-openshift-ca" pod "7ab84d1d-22c4-40a8-b0d5-9e78a4f8923a" (UID: "7ab84d1d-22c4-40a8-b0d5-9e78a4f8923a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:37:52.160287 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:52.160266    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ab84d1d-22c4-40a8-b0d5-9e78a4f8923a-maas-api-openshift-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:37:52.161487 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:52.161465    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ab84d1d-22c4-40a8-b0d5-9e78a4f8923a-maas-api-xks-ca" pod "7ab84d1d-22c4-40a8-b0d5-9e78a4f8923a" (UID: "7ab84d1d-22c4-40a8-b0d5-9e78a4f8923a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 03:37:52.161743 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:52.161723    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ab84d1d-22c4-40a8-b0d5-9e78a4f8923a-kube-api-access-hpk5l" pod "7ab84d1d-22c4-40a8-b0d5-9e78a4f8923a" (UID: "7ab84d1d-22c4-40a8-b0d5-9e78a4f8923a"). InnerVolumeSpecName "kube-api-access-hpk5l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:37:52.262069 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:52.262034    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ab84d1d-22c4-40a8-b0d5-9e78a4f8923a-maas-api-xks-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:37:52.262069 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:52.262067    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpk5l\" (UniqueName: \"kubernetes.io/projected/7ab84d1d-22c4-40a8-b0d5-9e78a4f8923a-kube-api-access-hpk5l\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:37:52.963879 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:52.963792    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29ea31d1-clkzv" event={"ID":"7ab84d1d-22c4-40a8-b0d5-9e78a4f8923a","Type":"ContainerDied","Data":"d41f2bba541590a717818d561cc124815af3976b389a8fb9dc5970103f796663"}
Nov 02 03:37:52.963879 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:52.963811    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29ea31d1-clkzv"
Nov 02 03:37:52.963879 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:52.963847    4496 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d41f2bba541590a717818d561cc124815af3976b389a8fb9dc5970103f796663"
Nov 02 03:37:53.312923 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:53.312890    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-018375-7vfzp"]
Nov 02 03:37:53.317860 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:53.317828    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-018375-7vfzp"]
Nov 02 03:37:53.574615 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:53.574531    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e60d498-1189-41c5-9548-243f2d01a40e" path="/var/lib/kubelet/pods/5e60d498-1189-41c5-9548-243f2d01a40e/volumes"
Nov 02 03:37:53.575081 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:53.575059    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdbc2cc9-af01-41e7-b360-4ab8b73a113c" path="/var/lib/kubelet/pods/cdbc2cc9-af01-41e7-b360-4ab8b73a113c/volumes"
Nov 02 03:37:58.888072 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:58.887989    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d98689e2-ztq7z"]
Nov 02 03:37:58.888958 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:58.888937    4496 cpu_manager.go:414] "containerMap: removing container" podUID="7ab84d1d-22c4-40a8-b0d5-9e78a4f8923a" containerName="revoke-keys"
Nov 02 03:37:58.888999 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:58.888963    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ab84d1d-22c4-40a8-b0d5-9e78a4f8923a" containerName="revoke-keys"
Nov 02 03:37:58.888999 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:58.888994    4496 cpu_manager.go:414] "containerMap: removing container" podUID="cdbc2cc9-af01-41e7-b360-4ab8b73a113c" containerName="payload-pre-processing"
Nov 02 03:37:58.889091 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:58.889003    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdbc2cc9-af01-41e7-b360-4ab8b73a113c" containerName="payload-pre-processing"
Nov 02 03:37:58.889186 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:58.889174    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ab84d1d-22c4-40a8-b0d5-9e78a4f8923a" containerName="revoke-keys"
Nov 02 03:37:58.889219 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:58.889197    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdbc2cc9-af01-41e7-b360-4ab8b73a113c" containerName="payload-pre-processing"
Nov 02 03:37:58.894451 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:58.894430    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d98689e2-ztq7z"
Nov 02 03:37:58.940883 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:58.940854    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ls66\" (UniqueName: \"kubernetes.io/projected/e768b97b-caa5-43c5-9db4-156a02a156a4-kube-api-access-5ls66\") pod \"maas-api-revoke-sub-d98689e2-ztq7z\" (UID: \"e768b97b-caa5-43c5-9db4-156a02a156a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d98689e2-ztq7z"
Nov 02 03:37:58.941071 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:58.940922    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e768b97b-caa5-43c5-9db4-156a02a156a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d98689e2-ztq7z\" (UID: \"e768b97b-caa5-43c5-9db4-156a02a156a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d98689e2-ztq7z"
Nov 02 03:37:58.941157 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:58.941055    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e768b97b-caa5-43c5-9db4-156a02a156a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d98689e2-ztq7z\" (UID: \"e768b97b-caa5-43c5-9db4-156a02a156a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d98689e2-ztq7z"
Nov 02 03:37:58.945630 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:58.945603    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d98689e2-ztq7z"]
Nov 02 03:37:58.991983 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:58.991934    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 03:37:58.992183 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:58.991985    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9z5bk\""
Nov 02 03:37:59.042068 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:59.042000    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ls66\" (UniqueName: \"kubernetes.io/projected/e768b97b-caa5-43c5-9db4-156a02a156a4-kube-api-access-5ls66\") pod \"maas-api-revoke-sub-d98689e2-ztq7z\" (UID: \"e768b97b-caa5-43c5-9db4-156a02a156a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d98689e2-ztq7z"
Nov 02 03:37:59.042227 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:59.042100    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e768b97b-caa5-43c5-9db4-156a02a156a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d98689e2-ztq7z\" (UID: \"e768b97b-caa5-43c5-9db4-156a02a156a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d98689e2-ztq7z"
Nov 02 03:37:59.042227 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:59.042129    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e768b97b-caa5-43c5-9db4-156a02a156a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d98689e2-ztq7z\" (UID: \"e768b97b-caa5-43c5-9db4-156a02a156a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d98689e2-ztq7z"
Nov 02 03:37:59.042770 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:59.042745    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e768b97b-caa5-43c5-9db4-156a02a156a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d98689e2-ztq7z\" (UID: \"e768b97b-caa5-43c5-9db4-156a02a156a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d98689e2-ztq7z"
Nov 02 03:37:59.044495 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:59.044472    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e768b97b-caa5-43c5-9db4-156a02a156a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d98689e2-ztq7z\" (UID: \"e768b97b-caa5-43c5-9db4-156a02a156a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d98689e2-ztq7z"
Nov 02 03:37:59.099176 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:59.099149    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ls66\" (UniqueName: \"kubernetes.io/projected/e768b97b-caa5-43c5-9db4-156a02a156a4-kube-api-access-5ls66\") pod \"maas-api-revoke-sub-d98689e2-ztq7z\" (UID: \"e768b97b-caa5-43c5-9db4-156a02a156a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d98689e2-ztq7z"
Nov 02 03:37:59.208629 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:59.208542    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d98689e2-ztq7z"
Nov 02 03:37:59.587683 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:37:59.587649    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode768b97b_caa5_43c5_9db4_156a02a156a4.slice/crio-f6fc995815612f25d3d4863c6f293bfad3026590a391101ed685d9c0d53e66ae WatchSource:0}: Error finding container f6fc995815612f25d3d4863c6f293bfad3026590a391101ed685d9c0d53e66ae: Status 404 returned error can't find the container with id f6fc995815612f25d3d4863c6f293bfad3026590a391101ed685d9c0d53e66ae
Nov 02 03:37:59.587842 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:59.587689    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d98689e2-ztq7z"]
Nov 02 03:37:59.843854 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:59.843772    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-018375-openshift-default-7fbc48965-7mlcx"]
Nov 02 03:37:59.844064 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:37:59.844030    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-018375-openshift-default-7fbc48965-7mlcx" podUID="ecfffbc9-4809-49a3-aed3-35b771fcb388" containerName="istio-proxy" containerID="cri-o://5f93f8ac2177bb1a83fc1c46c39136cefcc673e66b6d3c1a766de5c75cd65a70" gracePeriod=30
Nov 02 03:38:00.032405 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:00.032363    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d98689e2-ztq7z" event={"ID":"e768b97b-caa5-43c5-9db4-156a02a156a4","Type":"ContainerStarted","Data":"66861e8fe715e9eaa2899376320afe16e33c3b947bdf886aa8124486ef78362e"}
Nov 02 03:38:00.032822 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:00.032448    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d98689e2-ztq7z" event={"ID":"e768b97b-caa5-43c5-9db4-156a02a156a4","Type":"ContainerStarted","Data":"f6fc995815612f25d3d4863c6f293bfad3026590a391101ed685d9c0d53e66ae"}
Nov 02 03:38:00.149894 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:00.149775    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d98689e2-ztq7z" podStartSLOduration=2.149762703 podStartE2EDuration="2.149762703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:37:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:38:00.148809675 +0000 UTC m=+3020.727417054" watchObservedRunningTime="2026-11-02 03:38:00.149762703 +0000 UTC m=+3020.728370083"
Nov 02 03:38:01.044796 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:01.044760    4496 generic.go:356] "Generic (PLEG): container finished" podID="e768b97b-caa5-43c5-9db4-156a02a156a4" containerID="66861e8fe715e9eaa2899376320afe16e33c3b947bdf886aa8124486ef78362e" exitCode=0
Nov 02 03:38:01.045224 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:01.044840    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d98689e2-ztq7z" event={"ID":"e768b97b-caa5-43c5-9db4-156a02a156a4","Type":"ContainerDied","Data":"66861e8fe715e9eaa2899376320afe16e33c3b947bdf886aa8124486ef78362e"}
Nov 02 03:38:02.217288 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:02.217255    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d98689e2-ztq7z"
Nov 02 03:38:02.281635 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:02.281599    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e768b97b-caa5-43c5-9db4-156a02a156a4-kube-api-access-5ls66\" (UniqueName: \"kubernetes.io/projected/e768b97b-caa5-43c5-9db4-156a02a156a4-kube-api-access-5ls66\") pod \"e768b97b-caa5-43c5-9db4-156a02a156a4\" (UID: \"e768b97b-caa5-43c5-9db4-156a02a156a4\") "
Nov 02 03:38:02.281840 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:02.281665    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e768b97b-caa5-43c5-9db4-156a02a156a4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e768b97b-caa5-43c5-9db4-156a02a156a4-maas-api-openshift-ca\") pod \"e768b97b-caa5-43c5-9db4-156a02a156a4\" (UID: \"e768b97b-caa5-43c5-9db4-156a02a156a4\") "
Nov 02 03:38:02.281840 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:02.281766    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e768b97b-caa5-43c5-9db4-156a02a156a4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e768b97b-caa5-43c5-9db4-156a02a156a4-maas-api-xks-ca\") pod \"e768b97b-caa5-43c5-9db4-156a02a156a4\" (UID: \"e768b97b-caa5-43c5-9db4-156a02a156a4\") "
Nov 02 03:38:02.283686 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:02.283635    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e768b97b-caa5-43c5-9db4-156a02a156a4-maas-api-openshift-ca" pod "e768b97b-caa5-43c5-9db4-156a02a156a4" (UID: "e768b97b-caa5-43c5-9db4-156a02a156a4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:38:02.285011 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:02.284984    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e768b97b-caa5-43c5-9db4-156a02a156a4-maas-api-xks-ca" pod "e768b97b-caa5-43c5-9db4-156a02a156a4" (UID: "e768b97b-caa5-43c5-9db4-156a02a156a4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 03:38:02.285305 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:02.285276    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e768b97b-caa5-43c5-9db4-156a02a156a4-kube-api-access-5ls66" pod "e768b97b-caa5-43c5-9db4-156a02a156a4" (UID: "e768b97b-caa5-43c5-9db4-156a02a156a4"). InnerVolumeSpecName "kube-api-access-5ls66". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:38:02.383986 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:02.383890    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e768b97b-caa5-43c5-9db4-156a02a156a4-maas-api-xks-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:38:02.383986 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:02.383922    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ls66\" (UniqueName: \"kubernetes.io/projected/e768b97b-caa5-43c5-9db4-156a02a156a4-kube-api-access-5ls66\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:38:02.383986 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:02.383932    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e768b97b-caa5-43c5-9db4-156a02a156a4-maas-api-openshift-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:38:03.065200 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:03.065170    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d98689e2-ztq7z"
Nov 02 03:38:03.065392 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:03.065196    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d98689e2-ztq7z" event={"ID":"e768b97b-caa5-43c5-9db4-156a02a156a4","Type":"ContainerDied","Data":"f6fc995815612f25d3d4863c6f293bfad3026590a391101ed685d9c0d53e66ae"}
Nov 02 03:38:03.065392 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:03.065257    4496 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f6fc995815612f25d3d4863c6f293bfad3026590a391101ed685d9c0d53e66ae"
Nov 02 03:38:03.528342 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:38:03.528299    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 03:38:03.670876 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:03.670840    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-3d563c-64d4df97cf-v5sg9"]
Nov 02 03:38:03.671797 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:03.671775    4496 cpu_manager.go:414] "containerMap: removing container" podUID="e768b97b-caa5-43c5-9db4-156a02a156a4" containerName="revoke-keys"
Nov 02 03:38:03.671867 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:03.671800    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e768b97b-caa5-43c5-9db4-156a02a156a4" containerName="revoke-keys"
Nov 02 03:38:03.671986 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:03.671975    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="e768b97b-caa5-43c5-9db4-156a02a156a4" containerName="revoke-keys"
Nov 02 03:38:03.676279 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:03.676257    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3d563c-64d4df97cf-v5sg9"
Nov 02 03:38:03.729496 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:03.729457    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-3d563c\""
Nov 02 03:38:03.756065 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:03.756036    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-3d563c-64d4df97cf-v5sg9"]
Nov 02 03:38:03.801799 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:03.801703    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9afd306f-bfae-4fe0-b61f-317e997876a0-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-3d563c-64d4df97cf-v5sg9\" (UID: \"9afd306f-bfae-4fe0-b61f-317e997876a0\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3d563c-64d4df97cf-v5sg9"
Nov 02 03:38:03.802172 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:03.802142    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jdzn\" (UniqueName: \"kubernetes.io/projected/9afd306f-bfae-4fe0-b61f-317e997876a0-kube-api-access-4jdzn\") pod \"payload-pre-processing-e2e-praxis-b-3d563c-64d4df97cf-v5sg9\" (UID: \"9afd306f-bfae-4fe0-b61f-317e997876a0\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3d563c-64d4df97cf-v5sg9"
Nov 02 03:38:03.820088 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:03.820053    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-3d563c-79fbbc9545-sv8h8"]
Nov 02 03:38:03.824588 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:03.824562    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-3d563c-79fbbc9545-sv8h8"
Nov 02 03:38:03.880373 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:03.880328    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-3d563c-dockercfg-pmpk4\""
Nov 02 03:38:03.882271 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:03.882246    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-3d563c-79fbbc9545-sv8h8"]
Nov 02 03:38:03.903117 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:03.903087    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/319aacd9-1b90-48cf-8b87-04561e177fe5-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-3d563c-79fbbc9545-sv8h8\" (UID: \"319aacd9-1b90-48cf-8b87-04561e177fe5\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-3d563c-79fbbc9545-sv8h8"
Nov 02 03:38:03.903248 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:03.903188    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jdzn\" (UniqueName: \"kubernetes.io/projected/9afd306f-bfae-4fe0-b61f-317e997876a0-kube-api-access-4jdzn\") pod \"payload-pre-processing-e2e-praxis-b-3d563c-64d4df97cf-v5sg9\" (UID: \"9afd306f-bfae-4fe0-b61f-317e997876a0\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3d563c-64d4df97cf-v5sg9"
Nov 02 03:38:03.903302 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:03.903259    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lg2qc\" (UniqueName: \"kubernetes.io/projected/319aacd9-1b90-48cf-8b87-04561e177fe5-kube-api-access-lg2qc\") pod \"payload-processing-e2e-praxis-b-3d563c-79fbbc9545-sv8h8\" (UID: \"319aacd9-1b90-48cf-8b87-04561e177fe5\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-3d563c-79fbbc9545-sv8h8"
Nov 02 03:38:03.903446 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:03.903422    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9afd306f-bfae-4fe0-b61f-317e997876a0-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-3d563c-64d4df97cf-v5sg9\" (UID: \"9afd306f-bfae-4fe0-b61f-317e997876a0\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3d563c-64d4df97cf-v5sg9"
Nov 02 03:38:03.904341 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:03.904319    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9afd306f-bfae-4fe0-b61f-317e997876a0-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-3d563c-64d4df97cf-v5sg9\" (UID: \"9afd306f-bfae-4fe0-b61f-317e997876a0\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3d563c-64d4df97cf-v5sg9"
Nov 02 03:38:03.960808 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:03.960780    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jdzn\" (UniqueName: \"kubernetes.io/projected/9afd306f-bfae-4fe0-b61f-317e997876a0-kube-api-access-4jdzn\") pod \"payload-pre-processing-e2e-praxis-b-3d563c-64d4df97cf-v5sg9\" (UID: \"9afd306f-bfae-4fe0-b61f-317e997876a0\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3d563c-64d4df97cf-v5sg9"
Nov 02 03:38:03.992305 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:03.992274    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3d563c-64d4df97cf-v5sg9"
Nov 02 03:38:04.005037 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:04.004998    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/319aacd9-1b90-48cf-8b87-04561e177fe5-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-3d563c-79fbbc9545-sv8h8\" (UID: \"319aacd9-1b90-48cf-8b87-04561e177fe5\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-3d563c-79fbbc9545-sv8h8"
Nov 02 03:38:04.005141 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:04.005096    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lg2qc\" (UniqueName: \"kubernetes.io/projected/319aacd9-1b90-48cf-8b87-04561e177fe5-kube-api-access-lg2qc\") pod \"payload-processing-e2e-praxis-b-3d563c-79fbbc9545-sv8h8\" (UID: \"319aacd9-1b90-48cf-8b87-04561e177fe5\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-3d563c-79fbbc9545-sv8h8"
Nov 02 03:38:04.005895 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:04.005867    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/319aacd9-1b90-48cf-8b87-04561e177fe5-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-3d563c-79fbbc9545-sv8h8\" (UID: \"319aacd9-1b90-48cf-8b87-04561e177fe5\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-3d563c-79fbbc9545-sv8h8"
Nov 02 03:38:04.071069 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:04.070951    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lg2qc\" (UniqueName: \"kubernetes.io/projected/319aacd9-1b90-48cf-8b87-04561e177fe5-kube-api-access-lg2qc\") pod \"payload-processing-e2e-praxis-b-3d563c-79fbbc9545-sv8h8\" (UID: \"319aacd9-1b90-48cf-8b87-04561e177fe5\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-3d563c-79fbbc9545-sv8h8"
Nov 02 03:38:04.140781 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:04.140743    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-3d563c-79fbbc9545-sv8h8"
Nov 02 03:38:04.373391 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:04.373359    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-3d563c-64d4df97cf-v5sg9"]
Nov 02 03:38:04.374518 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:38:04.374485    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9afd306f_bfae_4fe0_b61f_317e997876a0.slice/crio-30fb2d0de74dcb53d29f5f5c91039379a51e18a3000199fcadfbd0dac2f872bf WatchSource:0}: Error finding container 30fb2d0de74dcb53d29f5f5c91039379a51e18a3000199fcadfbd0dac2f872bf: Status 404 returned error can't find the container with id 30fb2d0de74dcb53d29f5f5c91039379a51e18a3000199fcadfbd0dac2f872bf
Nov 02 03:38:04.462940 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:04.462908    4496 patch_prober.go:29] interesting pod/e2e-ipp-b-018375-openshift-default-7fbc48965-7mlcx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 03:38:04.463159 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:04.463128    4496 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-018375-openshift-default-7fbc48965-7mlcx" podUID="ecfffbc9-4809-49a3-aed3-35b771fcb388" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 03:38:04.520776 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:04.520744    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-3d563c-79fbbc9545-sv8h8"]
Nov 02 03:38:04.520983 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:38:04.520960    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod319aacd9_1b90_48cf_8b87_04561e177fe5.slice/crio-08b3c406d71bab99d2e7ac2c6beb97d9b0b0ea81a4e1f64d96be9336e7346004 WatchSource:0}: Error finding container 08b3c406d71bab99d2e7ac2c6beb97d9b0b0ea81a4e1f64d96be9336e7346004: Status 404 returned error can't find the container with id 08b3c406d71bab99d2e7ac2c6beb97d9b0b0ea81a4e1f64d96be9336e7346004
Nov 02 03:38:05.092107 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:05.092066    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3d563c-64d4df97cf-v5sg9" event={"ID":"9afd306f-bfae-4fe0-b61f-317e997876a0","Type":"ContainerStarted","Data":"38df86d12c20044180ff5932ff08edfa6b81d2288e1cdacd9a29fb4e09a07b5d"}
Nov 02 03:38:05.092575 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:05.092135    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3d563c-64d4df97cf-v5sg9" event={"ID":"9afd306f-bfae-4fe0-b61f-317e997876a0","Type":"ContainerStarted","Data":"30fb2d0de74dcb53d29f5f5c91039379a51e18a3000199fcadfbd0dac2f872bf"}
Nov 02 03:38:05.092575 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:05.092159    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3d563c-64d4df97cf-v5sg9"
Nov 02 03:38:05.094208 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:05.094101    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-3d563c-79fbbc9545-sv8h8" event={"ID":"319aacd9-1b90-48cf-8b87-04561e177fe5","Type":"ContainerStarted","Data":"82d601d0f4ee56e0be2b9341923b80946f68549c6214bd4380dfc105fb741a97"}
Nov 02 03:38:05.094208 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:05.094153    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-3d563c-79fbbc9545-sv8h8" event={"ID":"319aacd9-1b90-48cf-8b87-04561e177fe5","Type":"ContainerStarted","Data":"08b3c406d71bab99d2e7ac2c6beb97d9b0b0ea81a4e1f64d96be9336e7346004"}
Nov 02 03:38:05.094208 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:05.094208    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-3d563c-79fbbc9545-sv8h8"
Nov 02 03:38:05.096573 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:05.096546    4496 generic.go:356] "Generic (PLEG): container finished" podID="ecfffbc9-4809-49a3-aed3-35b771fcb388" containerID="5f93f8ac2177bb1a83fc1c46c39136cefcc673e66b6d3c1a766de5c75cd65a70" exitCode=0
Nov 02 03:38:05.096704 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:05.096598    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-018375-openshift-default-7fbc48965-7mlcx" event={"ID":"ecfffbc9-4809-49a3-aed3-35b771fcb388","Type":"ContainerDied","Data":"5f93f8ac2177bb1a83fc1c46c39136cefcc673e66b6d3c1a766de5c75cd65a70"}
Nov 02 03:38:05.096704 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:05.096646    4496 scope.go:122] "RemoveContainer" containerID="04e1a6d2ce978d286dad971f200a968a4b40fbf2ffdd3299875e06eafd008708"
Nov 02 03:38:05.134476 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:05.134452    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-018375-openshift-default-7fbc48965-7mlcx"
Nov 02 03:38:05.207331 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:05.207284    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3d563c-64d4df97cf-v5sg9" podStartSLOduration=2.207270922 podStartE2EDuration="2.207270922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:38:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:38:05.206864117 +0000 UTC m=+3025.785471498" watchObservedRunningTime="2026-11-02 03:38:05.207270922 +0000 UTC m=+3025.785878296"
Nov 02 03:38:05.221870 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:05.221839    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ecfffbc9-4809-49a3-aed3-35b771fcb388-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecfffbc9-4809-49a3-aed3-35b771fcb388-istio-podinfo\") pod \"ecfffbc9-4809-49a3-aed3-35b771fcb388\" (UID: \"ecfffbc9-4809-49a3-aed3-35b771fcb388\") "
Nov 02 03:38:05.222008 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:05.221936    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecfffbc9-4809-49a3-aed3-35b771fcb388-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecfffbc9-4809-49a3-aed3-35b771fcb388-workload-certs\") pod \"ecfffbc9-4809-49a3-aed3-35b771fcb388\" (UID: \"ecfffbc9-4809-49a3-aed3-35b771fcb388\") "
Nov 02 03:38:05.222008 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:05.221971    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecfffbc9-4809-49a3-aed3-35b771fcb388-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecfffbc9-4809-49a3-aed3-35b771fcb388-istio-envoy\") pod \"ecfffbc9-4809-49a3-aed3-35b771fcb388\" (UID: \"ecfffbc9-4809-49a3-aed3-35b771fcb388\") "
Nov 02 03:38:05.222130 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:05.222085    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecfffbc9-4809-49a3-aed3-35b771fcb388-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecfffbc9-4809-49a3-aed3-35b771fcb388-credential-socket\") pod \"ecfffbc9-4809-49a3-aed3-35b771fcb388\" (UID: \"ecfffbc9-4809-49a3-aed3-35b771fcb388\") "
Nov 02 03:38:05.222184 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:05.222127    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecfffbc9-4809-49a3-aed3-35b771fcb388-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecfffbc9-4809-49a3-aed3-35b771fcb388-workload-socket\") pod \"ecfffbc9-4809-49a3-aed3-35b771fcb388\" (UID: \"ecfffbc9-4809-49a3-aed3-35b771fcb388\") "
Nov 02 03:38:05.222240 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:05.222185    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecfffbc9-4809-49a3-aed3-35b771fcb388-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecfffbc9-4809-49a3-aed3-35b771fcb388-istiod-ca-cert\") pod \"ecfffbc9-4809-49a3-aed3-35b771fcb388\" (UID: \"ecfffbc9-4809-49a3-aed3-35b771fcb388\") "
Nov 02 03:38:05.222240 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:05.222222    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecfffbc9-4809-49a3-aed3-35b771fcb388-kube-api-access-2sjp2\" (UniqueName: \"kubernetes.io/projected/ecfffbc9-4809-49a3-aed3-35b771fcb388-kube-api-access-2sjp2\") pod \"ecfffbc9-4809-49a3-aed3-35b771fcb388\" (UID: \"ecfffbc9-4809-49a3-aed3-35b771fcb388\") "
Nov 02 03:38:05.222343 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:05.222261    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecfffbc9-4809-49a3-aed3-35b771fcb388-istio-token\" (UniqueName: \"kubernetes.io/projected/ecfffbc9-4809-49a3-aed3-35b771fcb388-istio-token\") pod \"ecfffbc9-4809-49a3-aed3-35b771fcb388\" (UID: \"ecfffbc9-4809-49a3-aed3-35b771fcb388\") "
Nov 02 03:38:05.222343 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:05.222308    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecfffbc9-4809-49a3-aed3-35b771fcb388-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecfffbc9-4809-49a3-aed3-35b771fcb388-istio-data\") pod \"ecfffbc9-4809-49a3-aed3-35b771fcb388\" (UID: \"ecfffbc9-4809-49a3-aed3-35b771fcb388\") "
Nov 02 03:38:05.222343 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:05.222313    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecfffbc9-4809-49a3-aed3-35b771fcb388-workload-certs" pod "ecfffbc9-4809-49a3-aed3-35b771fcb388" (UID: "ecfffbc9-4809-49a3-aed3-35b771fcb388"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:38:05.222510 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:05.222341    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecfffbc9-4809-49a3-aed3-35b771fcb388-credential-socket" pod "ecfffbc9-4809-49a3-aed3-35b771fcb388" (UID: "ecfffbc9-4809-49a3-aed3-35b771fcb388"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:38:05.222652 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:05.222626    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecfffbc9-4809-49a3-aed3-35b771fcb388-workload-socket" pod "ecfffbc9-4809-49a3-aed3-35b771fcb388" (UID: "ecfffbc9-4809-49a3-aed3-35b771fcb388"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:38:05.222830 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:05.222802    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecfffbc9-4809-49a3-aed3-35b771fcb388-istio-data" pod "ecfffbc9-4809-49a3-aed3-35b771fcb388" (UID: "ecfffbc9-4809-49a3-aed3-35b771fcb388"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:38:05.223079 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:05.223055    4496 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecfffbc9-4809-49a3-aed3-35b771fcb388-workload-certs\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:38:05.223169 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:05.223087    4496 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecfffbc9-4809-49a3-aed3-35b771fcb388-credential-socket\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:38:05.223169 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:05.223104    4496 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecfffbc9-4809-49a3-aed3-35b771fcb388-workload-socket\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:38:05.223169 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:05.223120    4496 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecfffbc9-4809-49a3-aed3-35b771fcb388-istio-data\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:38:05.223346 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:05.223320    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecfffbc9-4809-49a3-aed3-35b771fcb388-istiod-ca-cert" pod "ecfffbc9-4809-49a3-aed3-35b771fcb388" (UID: "ecfffbc9-4809-49a3-aed3-35b771fcb388"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:38:05.224705 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:05.224678    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecfffbc9-4809-49a3-aed3-35b771fcb388-kube-api-access-2sjp2" pod "ecfffbc9-4809-49a3-aed3-35b771fcb388" (UID: "ecfffbc9-4809-49a3-aed3-35b771fcb388"). InnerVolumeSpecName "kube-api-access-2sjp2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:38:05.224784 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:05.224701    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ecfffbc9-4809-49a3-aed3-35b771fcb388-istio-podinfo" pod "ecfffbc9-4809-49a3-aed3-35b771fcb388" (UID: "ecfffbc9-4809-49a3-aed3-35b771fcb388"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 03:38:05.225062 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:05.225028    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecfffbc9-4809-49a3-aed3-35b771fcb388-istio-envoy" pod "ecfffbc9-4809-49a3-aed3-35b771fcb388" (UID: "ecfffbc9-4809-49a3-aed3-35b771fcb388"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:38:05.225166 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:05.225153    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecfffbc9-4809-49a3-aed3-35b771fcb388-istio-token" pod "ecfffbc9-4809-49a3-aed3-35b771fcb388" (UID: "ecfffbc9-4809-49a3-aed3-35b771fcb388"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:38:05.323939 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:05.323875    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-3d563c-79fbbc9545-sv8h8" podStartSLOduration=2.323861885 podStartE2EDuration="2.323861885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:38:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:38:05.321633431 +0000 UTC m=+3025.900240811" watchObservedRunningTime="2026-11-02 03:38:05.323861885 +0000 UTC m=+3025.902469264"
Nov 02 03:38:05.324643 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:05.324618    4496 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecfffbc9-4809-49a3-aed3-35b771fcb388-istio-podinfo\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:38:05.324720 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:05.324645    4496 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecfffbc9-4809-49a3-aed3-35b771fcb388-istio-envoy\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:38:05.324720 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:05.324656    4496 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecfffbc9-4809-49a3-aed3-35b771fcb388-istiod-ca-cert\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:38:05.324720 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:05.324665    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2sjp2\" (UniqueName: \"kubernetes.io/projected/ecfffbc9-4809-49a3-aed3-35b771fcb388-kube-api-access-2sjp2\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:38:05.324720 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:05.324674    4496 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecfffbc9-4809-49a3-aed3-35b771fcb388-istio-token\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:38:06.106689 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:06.106652    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-018375-openshift-default-7fbc48965-7mlcx"
Nov 02 03:38:06.107250 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:06.106691    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-018375-openshift-default-7fbc48965-7mlcx" event={"ID":"ecfffbc9-4809-49a3-aed3-35b771fcb388","Type":"ContainerDied","Data":"c1c011a71df6103706a22e81e242eecda6f6b5709cca49085455f0b34ab03557"}
Nov 02 03:38:06.107250 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:06.106747    4496 scope.go:122] "RemoveContainer" containerID="5f93f8ac2177bb1a83fc1c46c39136cefcc673e66b6d3c1a766de5c75cd65a70"
Nov 02 03:38:06.340364 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:06.340327    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-018375-openshift-default-7fbc48965-7mlcx"]
Nov 02 03:38:06.345503 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:06.345470    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-018375-openshift-default-7fbc48965-7mlcx"]
Nov 02 03:38:07.583690 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:07.583656    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecfffbc9-4809-49a3-aed3-35b771fcb388" path="/var/lib/kubelet/pods/ecfffbc9-4809-49a3-aed3-35b771fcb388/volumes"
Nov 02 03:38:10.782482 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:10.782448    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-2912e6-66887f8d9f-fmjvw"]
Nov 02 03:38:10.783089 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:10.783074    4496 cpu_manager.go:414] "containerMap: removing container" podUID="ecfffbc9-4809-49a3-aed3-35b771fcb388" containerName="istio-proxy"
Nov 02 03:38:10.783089 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:10.783090    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecfffbc9-4809-49a3-aed3-35b771fcb388" containerName="istio-proxy"
Nov 02 03:38:10.783187 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:10.783125    4496 cpu_manager.go:414] "containerMap: removing container" podUID="ecfffbc9-4809-49a3-aed3-35b771fcb388" containerName="istio-proxy"
Nov 02 03:38:10.783187 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:10.783130    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecfffbc9-4809-49a3-aed3-35b771fcb388" containerName="istio-proxy"
Nov 02 03:38:10.783265 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:10.783218    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecfffbc9-4809-49a3-aed3-35b771fcb388" containerName="istio-proxy"
Nov 02 03:38:10.783265 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:10.783228    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecfffbc9-4809-49a3-aed3-35b771fcb388" containerName="istio-proxy"
Nov 02 03:38:10.788397 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:10.788373    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2912e6-66887f8d9f-fmjvw"
Nov 02 03:38:10.843629 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:10.843597    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-2912e6-66887f8d9f-fmjvw"]
Nov 02 03:38:10.888978 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:10.888940    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-2912e6\""
Nov 02 03:38:10.889187 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:10.889082    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-2912e6-dockercfg-jgnxc\""
Nov 02 03:38:10.892050 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:10.891992    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfnlw\" (UniqueName: \"kubernetes.io/projected/5e4bc2e9-17cc-4574-adbc-febeb3977823-kube-api-access-qfnlw\") pod \"payload-pre-processing-e2e-praxis-a-2912e6-66887f8d9f-fmjvw\" (UID: \"5e4bc2e9-17cc-4574-adbc-febeb3977823\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2912e6-66887f8d9f-fmjvw"
Nov 02 03:38:10.892232 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:10.892208    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e4bc2e9-17cc-4574-adbc-febeb3977823-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-2912e6-66887f8d9f-fmjvw\" (UID: \"5e4bc2e9-17cc-4574-adbc-febeb3977823\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2912e6-66887f8d9f-fmjvw"
Nov 02 03:38:10.945251 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:10.945217    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-2912e6-5468f5d46f-czlhp"]
Nov 02 03:38:10.950399 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:10.950375    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-2912e6-5468f5d46f-czlhp"
Nov 02 03:38:10.993597 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:10.993566    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfnlw\" (UniqueName: \"kubernetes.io/projected/5e4bc2e9-17cc-4574-adbc-febeb3977823-kube-api-access-qfnlw\") pod \"payload-pre-processing-e2e-praxis-a-2912e6-66887f8d9f-fmjvw\" (UID: \"5e4bc2e9-17cc-4574-adbc-febeb3977823\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2912e6-66887f8d9f-fmjvw"
Nov 02 03:38:10.994919 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:10.994895    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e4bc2e9-17cc-4574-adbc-febeb3977823-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-2912e6-66887f8d9f-fmjvw\" (UID: \"5e4bc2e9-17cc-4574-adbc-febeb3977823\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2912e6-66887f8d9f-fmjvw"
Nov 02 03:38:10.995830 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:10.995805    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e4bc2e9-17cc-4574-adbc-febeb3977823-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-2912e6-66887f8d9f-fmjvw\" (UID: \"5e4bc2e9-17cc-4574-adbc-febeb3977823\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2912e6-66887f8d9f-fmjvw"
Nov 02 03:38:10.997368 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:10.997345    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-2912e6-5468f5d46f-czlhp"]
Nov 02 03:38:11.053375 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:11.053304    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfnlw\" (UniqueName: \"kubernetes.io/projected/5e4bc2e9-17cc-4574-adbc-febeb3977823-kube-api-access-qfnlw\") pod \"payload-pre-processing-e2e-praxis-a-2912e6-66887f8d9f-fmjvw\" (UID: \"5e4bc2e9-17cc-4574-adbc-febeb3977823\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2912e6-66887f8d9f-fmjvw"
Nov 02 03:38:11.096210 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:11.096179    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pshx\" (UniqueName: \"kubernetes.io/projected/3cf215ca-ff27-4a44-a3a7-7ce21695a0d5-kube-api-access-9pshx\") pod \"payload-processing-e2e-praxis-a-2912e6-5468f5d46f-czlhp\" (UID: \"3cf215ca-ff27-4a44-a3a7-7ce21695a0d5\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-2912e6-5468f5d46f-czlhp"
Nov 02 03:38:11.096353 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:11.096239    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cf215ca-ff27-4a44-a3a7-7ce21695a0d5-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-2912e6-5468f5d46f-czlhp\" (UID: \"3cf215ca-ff27-4a44-a3a7-7ce21695a0d5\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-2912e6-5468f5d46f-czlhp"
Nov 02 03:38:11.106913 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:11.106887    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2912e6-66887f8d9f-fmjvw"
Nov 02 03:38:11.197775 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:11.197742    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cf215ca-ff27-4a44-a3a7-7ce21695a0d5-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-2912e6-5468f5d46f-czlhp\" (UID: \"3cf215ca-ff27-4a44-a3a7-7ce21695a0d5\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-2912e6-5468f5d46f-czlhp"
Nov 02 03:38:11.197936 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:11.197920    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pshx\" (UniqueName: \"kubernetes.io/projected/3cf215ca-ff27-4a44-a3a7-7ce21695a0d5-kube-api-access-9pshx\") pod \"payload-processing-e2e-praxis-a-2912e6-5468f5d46f-czlhp\" (UID: \"3cf215ca-ff27-4a44-a3a7-7ce21695a0d5\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-2912e6-5468f5d46f-czlhp"
Nov 02 03:38:11.198491 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:11.198473    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cf215ca-ff27-4a44-a3a7-7ce21695a0d5-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-2912e6-5468f5d46f-czlhp\" (UID: \"3cf215ca-ff27-4a44-a3a7-7ce21695a0d5\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-2912e6-5468f5d46f-czlhp"
Nov 02 03:38:11.259539 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:11.259504    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pshx\" (UniqueName: \"kubernetes.io/projected/3cf215ca-ff27-4a44-a3a7-7ce21695a0d5-kube-api-access-9pshx\") pod \"payload-processing-e2e-praxis-a-2912e6-5468f5d46f-czlhp\" (UID: \"3cf215ca-ff27-4a44-a3a7-7ce21695a0d5\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-2912e6-5468f5d46f-czlhp"
Nov 02 03:38:11.267213 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:11.267193    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-2912e6-5468f5d46f-czlhp"
Nov 02 03:38:11.489640 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:11.489607    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-2912e6-66887f8d9f-fmjvw"]
Nov 02 03:38:11.489753 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:38:11.489730    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e4bc2e9_17cc_4574_adbc_febeb3977823.slice/crio-347379b6204450c5558a67d1208d94a0e90d6fbd17fbeabbb524e8a460e1b828 WatchSource:0}: Error finding container 347379b6204450c5558a67d1208d94a0e90d6fbd17fbeabbb524e8a460e1b828: Status 404 returned error can't find the container with id 347379b6204450c5558a67d1208d94a0e90d6fbd17fbeabbb524e8a460e1b828
Nov 02 03:38:11.657811 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:11.657780    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-2912e6-5468f5d46f-czlhp"]
Nov 02 03:38:11.658439 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:38:11.658411    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cf215ca_ff27_4a44_a3a7_7ce21695a0d5.slice/crio-22b5e301809e9ca8eda75a6677518a8ba8a2ffcb7c82f5b672b249ddc8738441 WatchSource:0}: Error finding container 22b5e301809e9ca8eda75a6677518a8ba8a2ffcb7c82f5b672b249ddc8738441: Status 404 returned error can't find the container with id 22b5e301809e9ca8eda75a6677518a8ba8a2ffcb7c82f5b672b249ddc8738441
Nov 02 03:38:12.154362 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:12.154317    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2912e6-66887f8d9f-fmjvw" event={"ID":"5e4bc2e9-17cc-4574-adbc-febeb3977823","Type":"ContainerStarted","Data":"edd8d80d28c66dc24f384e57b649722f54dfad34e5b8bcce807ea6073b1671c0"}
Nov 02 03:38:12.154813 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:12.154394    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2912e6-66887f8d9f-fmjvw" event={"ID":"5e4bc2e9-17cc-4574-adbc-febeb3977823","Type":"ContainerStarted","Data":"347379b6204450c5558a67d1208d94a0e90d6fbd17fbeabbb524e8a460e1b828"}
Nov 02 03:38:12.154813 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:12.154561    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2912e6-66887f8d9f-fmjvw"
Nov 02 03:38:12.156376 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:12.156346    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-2912e6-5468f5d46f-czlhp" event={"ID":"3cf215ca-ff27-4a44-a3a7-7ce21695a0d5","Type":"ContainerStarted","Data":"d6aad296c29cf6935d401db1cc093edbbb6cfe14422b0a35e7ebf93ade4d2571"}
Nov 02 03:38:12.156518 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:12.156401    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-2912e6-5468f5d46f-czlhp" event={"ID":"3cf215ca-ff27-4a44-a3a7-7ce21695a0d5","Type":"ContainerStarted","Data":"22b5e301809e9ca8eda75a6677518a8ba8a2ffcb7c82f5b672b249ddc8738441"}
Nov 02 03:38:12.156518 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:12.156433    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-2912e6-5468f5d46f-czlhp"
Nov 02 03:38:12.269417 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:12.269365    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2912e6-66887f8d9f-fmjvw" podStartSLOduration=2.269351367 podStartE2EDuration="2.269351367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:38:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:38:12.268304933 +0000 UTC m=+3032.846912314" watchObservedRunningTime="2026-11-02 03:38:12.269351367 +0000 UTC m=+3032.847958747"
Nov 02 03:38:12.384620 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:12.384562    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-2912e6-5468f5d46f-czlhp" podStartSLOduration=2.384546932 podStartE2EDuration="2.384546932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:38:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:38:12.383453873 +0000 UTC m=+3032.962061256" watchObservedRunningTime="2026-11-02 03:38:12.384546932 +0000 UTC m=+3032.963154312"
Nov 02 03:38:14.650356 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:14.650321    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-4c3ea5/tenantcfg-model-4c3ea5-kserve-77d88775bf-gsngn"]
Nov 02 03:38:16.109913 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:16.109883    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-3d563c-79fbbc9545-sv8h8"
Nov 02 03:38:16.110330 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:16.109957    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3d563c-64d4df97cf-v5sg9"
Nov 02 03:38:17.764226 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:17.764192    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-cb56e9-d444d6cbc-vvc97"]
Nov 02 03:38:17.768836 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:17.768814    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cb56e9-d444d6cbc-vvc97"
Nov 02 03:38:17.821918 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:17.821889    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-cb56e9-dockercfg-qnh57\""
Nov 02 03:38:17.822151 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:17.821942    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-cb56e9\""
Nov 02 03:38:17.823722 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:17.823696    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-cb56e9-d444d6cbc-vvc97"]
Nov 02 03:38:17.873867 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:17.873809    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf4aaa9a-1039-4f41-ad04-2ef9c946fe4f-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-cb56e9-d444d6cbc-vvc97\" (UID: \"cf4aaa9a-1039-4f41-ad04-2ef9c946fe4f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cb56e9-d444d6cbc-vvc97"
Nov 02 03:38:17.874091 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:17.874031    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkg44\" (UniqueName: \"kubernetes.io/projected/cf4aaa9a-1039-4f41-ad04-2ef9c946fe4f-kube-api-access-lkg44\") pod \"payload-pre-processing-e2e-praxis-b-cb56e9-d444d6cbc-vvc97\" (UID: \"cf4aaa9a-1039-4f41-ad04-2ef9c946fe4f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cb56e9-d444d6cbc-vvc97"
Nov 02 03:38:17.925293 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:17.925258    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-cb56e9-545bcc7cc7-85plw"]
Nov 02 03:38:17.929625 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:17.929607    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-cb56e9-545bcc7cc7-85plw"
Nov 02 03:38:17.975378 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:17.975343    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf4aaa9a-1039-4f41-ad04-2ef9c946fe4f-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-cb56e9-d444d6cbc-vvc97\" (UID: \"cf4aaa9a-1039-4f41-ad04-2ef9c946fe4f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cb56e9-d444d6cbc-vvc97"
Nov 02 03:38:17.975540 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:17.975427    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkg44\" (UniqueName: \"kubernetes.io/projected/cf4aaa9a-1039-4f41-ad04-2ef9c946fe4f-kube-api-access-lkg44\") pod \"payload-pre-processing-e2e-praxis-b-cb56e9-d444d6cbc-vvc97\" (UID: \"cf4aaa9a-1039-4f41-ad04-2ef9c946fe4f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cb56e9-d444d6cbc-vvc97"
Nov 02 03:38:17.976001 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:17.975981    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf4aaa9a-1039-4f41-ad04-2ef9c946fe4f-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-cb56e9-d444d6cbc-vvc97\" (UID: \"cf4aaa9a-1039-4f41-ad04-2ef9c946fe4f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cb56e9-d444d6cbc-vvc97"
Nov 02 03:38:17.985921 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:17.985889    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-cb56e9-545bcc7cc7-85plw"]
Nov 02 03:38:18.033604 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:18.033531    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkg44\" (UniqueName: \"kubernetes.io/projected/cf4aaa9a-1039-4f41-ad04-2ef9c946fe4f-kube-api-access-lkg44\") pod \"payload-pre-processing-e2e-praxis-b-cb56e9-d444d6cbc-vvc97\" (UID: \"cf4aaa9a-1039-4f41-ad04-2ef9c946fe4f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cb56e9-d444d6cbc-vvc97"
Nov 02 03:38:18.076860 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:18.076822    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mwgs\" (UniqueName: \"kubernetes.io/projected/1975c9eb-9a68-4d15-9515-8a03d8e63fdd-kube-api-access-8mwgs\") pod \"payload-processing-e2e-praxis-b-cb56e9-545bcc7cc7-85plw\" (UID: \"1975c9eb-9a68-4d15-9515-8a03d8e63fdd\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-cb56e9-545bcc7cc7-85plw"
Nov 02 03:38:18.077035 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:18.076869    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1975c9eb-9a68-4d15-9515-8a03d8e63fdd-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-cb56e9-545bcc7cc7-85plw\" (UID: \"1975c9eb-9a68-4d15-9515-8a03d8e63fdd\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-cb56e9-545bcc7cc7-85plw"
Nov 02 03:38:18.083719 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:18.083698    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cb56e9-d444d6cbc-vvc97"
Nov 02 03:38:18.178335 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:18.178304    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mwgs\" (UniqueName: \"kubernetes.io/projected/1975c9eb-9a68-4d15-9515-8a03d8e63fdd-kube-api-access-8mwgs\") pod \"payload-processing-e2e-praxis-b-cb56e9-545bcc7cc7-85plw\" (UID: \"1975c9eb-9a68-4d15-9515-8a03d8e63fdd\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-cb56e9-545bcc7cc7-85plw"
Nov 02 03:38:18.178532 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:18.178344    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1975c9eb-9a68-4d15-9515-8a03d8e63fdd-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-cb56e9-545bcc7cc7-85plw\" (UID: \"1975c9eb-9a68-4d15-9515-8a03d8e63fdd\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-cb56e9-545bcc7cc7-85plw"
Nov 02 03:38:18.179223 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:18.179199    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1975c9eb-9a68-4d15-9515-8a03d8e63fdd-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-cb56e9-545bcc7cc7-85plw\" (UID: \"1975c9eb-9a68-4d15-9515-8a03d8e63fdd\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-cb56e9-545bcc7cc7-85plw"
Nov 02 03:38:18.235589 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:18.235552    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mwgs\" (UniqueName: \"kubernetes.io/projected/1975c9eb-9a68-4d15-9515-8a03d8e63fdd-kube-api-access-8mwgs\") pod \"payload-processing-e2e-praxis-b-cb56e9-545bcc7cc7-85plw\" (UID: \"1975c9eb-9a68-4d15-9515-8a03d8e63fdd\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-cb56e9-545bcc7cc7-85plw"
Nov 02 03:38:18.243432 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:18.243399    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-cb56e9-545bcc7cc7-85plw"
Nov 02 03:38:18.462502 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:18.462471    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-cb56e9-d444d6cbc-vvc97"]
Nov 02 03:38:18.463989 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:38:18.463969    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf4aaa9a_1039_4f41_ad04_2ef9c946fe4f.slice/crio-1cc9a3a257d2523ebc4321c87607e859353fba89915dab69aed708be7d95a0ac WatchSource:0}: Error finding container 1cc9a3a257d2523ebc4321c87607e859353fba89915dab69aed708be7d95a0ac: Status 404 returned error can't find the container with id 1cc9a3a257d2523ebc4321c87607e859353fba89915dab69aed708be7d95a0ac
Nov 02 03:38:18.622331 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:18.622304    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-cb56e9-545bcc7cc7-85plw"]
Nov 02 03:38:18.623838 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:38:18.623798    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1975c9eb_9a68_4d15_9515_8a03d8e63fdd.slice/crio-1d347c379b21a201a66e57568c3001d2e98cde8533e746dd46a756380c24259f WatchSource:0}: Error finding container 1d347c379b21a201a66e57568c3001d2e98cde8533e746dd46a756380c24259f: Status 404 returned error can't find the container with id 1d347c379b21a201a66e57568c3001d2e98cde8533e746dd46a756380c24259f
Nov 02 03:38:19.214412 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:19.214374    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cb56e9-d444d6cbc-vvc97" event={"ID":"cf4aaa9a-1039-4f41-ad04-2ef9c946fe4f","Type":"ContainerStarted","Data":"54a286e31d5f8304e4a572bbaac5bd993bf22affe04d6f42ff1b55794a1d0ef3"}
Nov 02 03:38:19.214893 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:19.214629    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cb56e9-d444d6cbc-vvc97"
Nov 02 03:38:19.214893 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:19.214678    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cb56e9-d444d6cbc-vvc97" event={"ID":"cf4aaa9a-1039-4f41-ad04-2ef9c946fe4f","Type":"ContainerStarted","Data":"1cc9a3a257d2523ebc4321c87607e859353fba89915dab69aed708be7d95a0ac"}
Nov 02 03:38:19.216326 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:19.216299    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-cb56e9-545bcc7cc7-85plw" event={"ID":"1975c9eb-9a68-4d15-9515-8a03d8e63fdd","Type":"ContainerStarted","Data":"0489f9db4d35bdc8e040da7a42fae9e618cbd1ecb09ce29396c868f78188bfa3"}
Nov 02 03:38:19.216435 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:19.216347    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-cb56e9-545bcc7cc7-85plw" event={"ID":"1975c9eb-9a68-4d15-9515-8a03d8e63fdd","Type":"ContainerStarted","Data":"1d347c379b21a201a66e57568c3001d2e98cde8533e746dd46a756380c24259f"}
Nov 02 03:38:19.216435 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:19.216413    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-cb56e9-545bcc7cc7-85plw"
Nov 02 03:38:19.328452 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:19.328403    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cb56e9-d444d6cbc-vvc97" podStartSLOduration=2.328390622 podStartE2EDuration="2.328390622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:38:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:38:19.327778882 +0000 UTC m=+3039.906386272" watchObservedRunningTime="2026-11-02 03:38:19.328390622 +0000 UTC m=+3039.906998001"
Nov 02 03:38:19.446956 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:19.446904    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-cb56e9-545bcc7cc7-85plw" podStartSLOduration=2.446889775 podStartE2EDuration="2.446889775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:38:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:38:19.441718779 +0000 UTC m=+3040.020326166" watchObservedRunningTime="2026-11-02 03:38:19.446889775 +0000 UTC m=+3040.025497158"
Nov 02 03:38:23.169868 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:23.169838    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2912e6-66887f8d9f-fmjvw"
Nov 02 03:38:23.170301 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:23.169908    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-2912e6-5468f5d46f-czlhp"
Nov 02 03:38:24.088362 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:24.088327    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-167a01-7dbfb6897f-6xfws"]
Nov 02 03:38:24.092845 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:24.092825    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-167a01-7dbfb6897f-6xfws"
Nov 02 03:38:24.146377 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:24.146332    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a600497-7759-4a18-adcd-29c344cba773-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-167a01-7dbfb6897f-6xfws\" (UID: \"4a600497-7759-4a18-adcd-29c344cba773\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-167a01-7dbfb6897f-6xfws"
Nov 02 03:38:24.146537 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:24.146382    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nrj8\" (UniqueName: \"kubernetes.io/projected/4a600497-7759-4a18-adcd-29c344cba773-kube-api-access-6nrj8\") pod \"maas-api-e2e-praxis-a-167a01-7dbfb6897f-6xfws\" (UID: \"4a600497-7759-4a18-adcd-29c344cba773\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-167a01-7dbfb6897f-6xfws"
Nov 02 03:38:24.146537 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:24.146425    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a600497-7759-4a18-adcd-29c344cba773-metrics-certs\") pod \"maas-api-e2e-praxis-a-167a01-7dbfb6897f-6xfws\" (UID: \"4a600497-7759-4a18-adcd-29c344cba773\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-167a01-7dbfb6897f-6xfws"
Nov 02 03:38:24.146662 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:24.146536    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a600497-7759-4a18-adcd-29c344cba773-maas-api-tls\") pod \"maas-api-e2e-praxis-a-167a01-7dbfb6897f-6xfws\" (UID: \"4a600497-7759-4a18-adcd-29c344cba773\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-167a01-7dbfb6897f-6xfws"
Nov 02 03:38:24.149023 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:24.148989    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-167a01-7dbfb6897f-6xfws"]
Nov 02 03:38:24.195540 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:24.195512    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-167a01-serving-cert\""
Nov 02 03:38:24.247577 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:24.247543    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a600497-7759-4a18-adcd-29c344cba773-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-167a01-7dbfb6897f-6xfws\" (UID: \"4a600497-7759-4a18-adcd-29c344cba773\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-167a01-7dbfb6897f-6xfws"
Nov 02 03:38:24.247901 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:24.247588    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nrj8\" (UniqueName: \"kubernetes.io/projected/4a600497-7759-4a18-adcd-29c344cba773-kube-api-access-6nrj8\") pod \"maas-api-e2e-praxis-a-167a01-7dbfb6897f-6xfws\" (UID: \"4a600497-7759-4a18-adcd-29c344cba773\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-167a01-7dbfb6897f-6xfws"
Nov 02 03:38:24.247901 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:24.247626    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a600497-7759-4a18-adcd-29c344cba773-metrics-certs\") pod \"maas-api-e2e-praxis-a-167a01-7dbfb6897f-6xfws\" (UID: \"4a600497-7759-4a18-adcd-29c344cba773\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-167a01-7dbfb6897f-6xfws"
Nov 02 03:38:24.247901 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:24.247728    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a600497-7759-4a18-adcd-29c344cba773-maas-api-tls\") pod \"maas-api-e2e-praxis-a-167a01-7dbfb6897f-6xfws\" (UID: \"4a600497-7759-4a18-adcd-29c344cba773\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-167a01-7dbfb6897f-6xfws"
Nov 02 03:38:24.247901 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:38:24.247882    4496 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-167a01-serving-cert: secret "maas-api-e2e-praxis-a-167a01-serving-cert" not found
Nov 02 03:38:24.248176 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:38:24.247953    4496 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a600497-7759-4a18-adcd-29c344cba773-maas-api-tls podName:4a600497-7759-4a18-adcd-29c344cba773 nodeName:}" failed. No retries permitted until 2026-11-02 03:38:24.747929348 +0000 UTC m=+3045.326536709 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4a600497-7759-4a18-adcd-29c344cba773-maas-api-tls") pod "maas-api-e2e-praxis-a-167a01-7dbfb6897f-6xfws" (UID: "4a600497-7759-4a18-adcd-29c344cba773") : secret "maas-api-e2e-praxis-a-167a01-serving-cert" not found
Nov 02 03:38:24.248763 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:24.248727    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a600497-7759-4a18-adcd-29c344cba773-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-167a01-7dbfb6897f-6xfws\" (UID: \"4a600497-7759-4a18-adcd-29c344cba773\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-167a01-7dbfb6897f-6xfws"
Nov 02 03:38:24.250402 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:24.250381    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a600497-7759-4a18-adcd-29c344cba773-metrics-certs\") pod \"maas-api-e2e-praxis-a-167a01-7dbfb6897f-6xfws\" (UID: \"4a600497-7759-4a18-adcd-29c344cba773\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-167a01-7dbfb6897f-6xfws"
Nov 02 03:38:24.308342 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:24.308314    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nrj8\" (UniqueName: \"kubernetes.io/projected/4a600497-7759-4a18-adcd-29c344cba773-kube-api-access-6nrj8\") pod \"maas-api-e2e-praxis-a-167a01-7dbfb6897f-6xfws\" (UID: \"4a600497-7759-4a18-adcd-29c344cba773\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-167a01-7dbfb6897f-6xfws"
Nov 02 03:38:24.753911 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:24.753872    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a600497-7759-4a18-adcd-29c344cba773-maas-api-tls\") pod \"maas-api-e2e-praxis-a-167a01-7dbfb6897f-6xfws\" (UID: \"4a600497-7759-4a18-adcd-29c344cba773\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-167a01-7dbfb6897f-6xfws"
Nov 02 03:38:24.756300 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:24.756274    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a600497-7759-4a18-adcd-29c344cba773-maas-api-tls\") pod \"maas-api-e2e-praxis-a-167a01-7dbfb6897f-6xfws\" (UID: \"4a600497-7759-4a18-adcd-29c344cba773\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-167a01-7dbfb6897f-6xfws"
Nov 02 03:38:25.007862 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:25.007782    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-167a01-7dbfb6897f-6xfws"
Nov 02 03:38:25.387772 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:25.387739    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-167a01-7dbfb6897f-6xfws"]
Nov 02 03:38:25.388486 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:38:25.388461    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a600497_7759_4a18_adcd_29c344cba773.slice/crio-9ad85a55d194c4e3b2c208e0e4fa8d355c1e2e9083cedd61313dc3b671264865 WatchSource:0}: Error finding container 9ad85a55d194c4e3b2c208e0e4fa8d355c1e2e9083cedd61313dc3b671264865: Status 404 returned error can't find the container with id 9ad85a55d194c4e3b2c208e0e4fa8d355c1e2e9083cedd61313dc3b671264865
Nov 02 03:38:26.281158 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:26.281121    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-167a01-7dbfb6897f-6xfws" event={"ID":"4a600497-7759-4a18-adcd-29c344cba773","Type":"ContainerStarted","Data":"9ad85a55d194c4e3b2c208e0e4fa8d355c1e2e9083cedd61313dc3b671264865"}
Nov 02 03:38:27.290700 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:27.290663    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-167a01-7dbfb6897f-6xfws" event={"ID":"4a600497-7759-4a18-adcd-29c344cba773","Type":"ContainerStarted","Data":"95210ae1082b473faf043c5cf8b6c56cff9195c681e746140b7dde1432bc1794"}
Nov 02 03:38:27.291116 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:27.290887    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-167a01-7dbfb6897f-6xfws"
Nov 02 03:38:27.409048 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:27.408982    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-167a01-7dbfb6897f-6xfws" podStartSLOduration=2.5468891940000002 podStartE2EDuration="3.408967824s" totalImagesPullingTime="862.07863ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:38:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:38:27.40700641 +0000 UTC m=+3047.985613801" watchObservedRunningTime="2026-11-02 03:38:27.408967824 +0000 UTC m=+3047.987575203"
Nov 02 03:38:29.671317 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:29.671274    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-4c3ea5/tenantcfg-model-4c3ea5-kserve-77d88775bf-gsngn" podUID="c08a1ba9-e651-4714-b84d-9b8161f6ae8f" containerName="main" containerID="cri-o://2cd43900264fe04ae9022fbbb3e46bc3c626dc980336a09be1adbcb51086a9cc" gracePeriod=45
Nov 02 03:38:29.948903 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:29.948873    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-4c3ea5/tenantcfg-model-4c3ea5-kserve-77d88775bf-gsngn"
Nov 02 03:38:30.016528 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:30.016494    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c08a1ba9-e651-4714-b84d-9b8161f6ae8f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c08a1ba9-e651-4714-b84d-9b8161f6ae8f-model-cache\") pod \"c08a1ba9-e651-4714-b84d-9b8161f6ae8f\" (UID: \"c08a1ba9-e651-4714-b84d-9b8161f6ae8f\") "
Nov 02 03:38:30.016720 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:30.016554    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c08a1ba9-e651-4714-b84d-9b8161f6ae8f-kube-api-access-kpcwd\" (UniqueName: \"kubernetes.io/projected/c08a1ba9-e651-4714-b84d-9b8161f6ae8f-kube-api-access-kpcwd\") pod \"c08a1ba9-e651-4714-b84d-9b8161f6ae8f\" (UID: \"c08a1ba9-e651-4714-b84d-9b8161f6ae8f\") "
Nov 02 03:38:30.016720 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:30.016577    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c08a1ba9-e651-4714-b84d-9b8161f6ae8f-home\" (UniqueName: \"kubernetes.io/empty-dir/c08a1ba9-e651-4714-b84d-9b8161f6ae8f-home\") pod \"c08a1ba9-e651-4714-b84d-9b8161f6ae8f\" (UID: \"c08a1ba9-e651-4714-b84d-9b8161f6ae8f\") "
Nov 02 03:38:30.016720 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:30.016635    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c08a1ba9-e651-4714-b84d-9b8161f6ae8f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c08a1ba9-e651-4714-b84d-9b8161f6ae8f-dshm\") pod \"c08a1ba9-e651-4714-b84d-9b8161f6ae8f\" (UID: \"c08a1ba9-e651-4714-b84d-9b8161f6ae8f\") "
Nov 02 03:38:30.016720 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:30.016719    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c08a1ba9-e651-4714-b84d-9b8161f6ae8f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c08a1ba9-e651-4714-b84d-9b8161f6ae8f-tmp-dir\") pod \"c08a1ba9-e651-4714-b84d-9b8161f6ae8f\" (UID: \"c08a1ba9-e651-4714-b84d-9b8161f6ae8f\") "
Nov 02 03:38:30.016968 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:30.016757    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c08a1ba9-e651-4714-b84d-9b8161f6ae8f-tls-certs\" (UniqueName: \"kubernetes.io/secret/c08a1ba9-e651-4714-b84d-9b8161f6ae8f-tls-certs\") pod \"c08a1ba9-e651-4714-b84d-9b8161f6ae8f\" (UID: \"c08a1ba9-e651-4714-b84d-9b8161f6ae8f\") "
Nov 02 03:38:30.016968 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:30.016853    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c08a1ba9-e651-4714-b84d-9b8161f6ae8f-model-cache" pod "c08a1ba9-e651-4714-b84d-9b8161f6ae8f" (UID: "c08a1ba9-e651-4714-b84d-9b8161f6ae8f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:38:30.016968 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:30.016890    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c08a1ba9-e651-4714-b84d-9b8161f6ae8f-home" pod "c08a1ba9-e651-4714-b84d-9b8161f6ae8f" (UID: "c08a1ba9-e651-4714-b84d-9b8161f6ae8f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:38:30.017169 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:30.016998    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c08a1ba9-e651-4714-b84d-9b8161f6ae8f-tmp-dir" pod "c08a1ba9-e651-4714-b84d-9b8161f6ae8f" (UID: "c08a1ba9-e651-4714-b84d-9b8161f6ae8f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:38:30.017321 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:30.017301    4496 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c08a1ba9-e651-4714-b84d-9b8161f6ae8f-home\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:38:30.017389 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:30.017325    4496 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c08a1ba9-e651-4714-b84d-9b8161f6ae8f-tmp-dir\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:38:30.017389 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:30.017340    4496 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c08a1ba9-e651-4714-b84d-9b8161f6ae8f-model-cache\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:38:30.018944 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:30.018919    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c08a1ba9-e651-4714-b84d-9b8161f6ae8f-dshm" pod "c08a1ba9-e651-4714-b84d-9b8161f6ae8f" (UID: "c08a1ba9-e651-4714-b84d-9b8161f6ae8f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:38:30.019002 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:30.018924    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c08a1ba9-e651-4714-b84d-9b8161f6ae8f-tls-certs" pod "c08a1ba9-e651-4714-b84d-9b8161f6ae8f" (UID: "c08a1ba9-e651-4714-b84d-9b8161f6ae8f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 03:38:30.019002 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:30.018930    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c08a1ba9-e651-4714-b84d-9b8161f6ae8f-kube-api-access-kpcwd" pod "c08a1ba9-e651-4714-b84d-9b8161f6ae8f" (UID: "c08a1ba9-e651-4714-b84d-9b8161f6ae8f"). InnerVolumeSpecName "kube-api-access-kpcwd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:38:30.118914 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:30.118877    4496 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c08a1ba9-e651-4714-b84d-9b8161f6ae8f-tls-certs\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:38:30.118914 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:30.118908    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpcwd\" (UniqueName: \"kubernetes.io/projected/c08a1ba9-e651-4714-b84d-9b8161f6ae8f-kube-api-access-kpcwd\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:38:30.118914 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:30.118918    4496 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c08a1ba9-e651-4714-b84d-9b8161f6ae8f-dshm\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:38:30.226596 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:30.226568    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-cb56e9-545bcc7cc7-85plw"
Nov 02 03:38:30.226877 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:30.226860    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cb56e9-d444d6cbc-vvc97"
Nov 02 03:38:30.318605 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:30.318575    4496 generic.go:356] "Generic (PLEG): container finished" podID="c08a1ba9-e651-4714-b84d-9b8161f6ae8f" containerID="2cd43900264fe04ae9022fbbb3e46bc3c626dc980336a09be1adbcb51086a9cc" exitCode=0
Nov 02 03:38:30.318767 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:30.318602    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-4c3ea5/tenantcfg-model-4c3ea5-kserve-77d88775bf-gsngn" event={"ID":"c08a1ba9-e651-4714-b84d-9b8161f6ae8f","Type":"ContainerDied","Data":"2cd43900264fe04ae9022fbbb3e46bc3c626dc980336a09be1adbcb51086a9cc"}
Nov 02 03:38:30.318767 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:30.318616    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-4c3ea5/tenantcfg-model-4c3ea5-kserve-77d88775bf-gsngn"
Nov 02 03:38:30.318767 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:30.318645    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-4c3ea5/tenantcfg-model-4c3ea5-kserve-77d88775bf-gsngn" event={"ID":"c08a1ba9-e651-4714-b84d-9b8161f6ae8f","Type":"ContainerDied","Data":"58d616bea10a38e92f9dce668e14f88f51843ec206b49f8e15dd3985bcda3af4"}
Nov 02 03:38:30.318767 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:30.318665    4496 scope.go:122] "RemoveContainer" containerID="2cd43900264fe04ae9022fbbb3e46bc3c626dc980336a09be1adbcb51086a9cc"
Nov 02 03:38:30.332580 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:30.332560    4496 scope.go:122] "RemoveContainer" containerID="2cd43900264fe04ae9022fbbb3e46bc3c626dc980336a09be1adbcb51086a9cc"
Nov 02 03:38:30.332874 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:38:30.332853    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cd43900264fe04ae9022fbbb3e46bc3c626dc980336a09be1adbcb51086a9cc\": container with ID starting with 2cd43900264fe04ae9022fbbb3e46bc3c626dc980336a09be1adbcb51086a9cc not found: ID does not exist" containerID="2cd43900264fe04ae9022fbbb3e46bc3c626dc980336a09be1adbcb51086a9cc"
Nov 02 03:38:30.332926 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:30.332882    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cd43900264fe04ae9022fbbb3e46bc3c626dc980336a09be1adbcb51086a9cc"} err="failed to get container status \"2cd43900264fe04ae9022fbbb3e46bc3c626dc980336a09be1adbcb51086a9cc\": rpc error: code = NotFound desc = could not find container \"2cd43900264fe04ae9022fbbb3e46bc3c626dc980336a09be1adbcb51086a9cc\": container with ID starting with 2cd43900264fe04ae9022fbbb3e46bc3c626dc980336a09be1adbcb51086a9cc not found: ID does not exist"
Nov 02 03:38:30.570085 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:30.569413    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-4c3ea5/tenantcfg-model-4c3ea5-kserve-77d88775bf-gsngn"]
Nov 02 03:38:30.573492 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:30.573459    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-4c3ea5/tenantcfg-model-4c3ea5-kserve-77d88775bf-gsngn"]
Nov 02 03:38:31.584481 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:31.584444    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c08a1ba9-e651-4714-b84d-9b8161f6ae8f" path="/var/lib/kubelet/pods/c08a1ba9-e651-4714-b84d-9b8161f6ae8f/volumes"
Nov 02 03:38:33.305516 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:33.305484    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-167a01-7dbfb6897f-6xfws"
Nov 02 03:38:33.527556 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:38:33.527521    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 03:38:34.007658 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:34.007620    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d94cc3-55f675b9d8-46f48"]
Nov 02 03:38:34.008306 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:34.008290    4496 cpu_manager.go:414] "containerMap: removing container" podUID="c08a1ba9-e651-4714-b84d-9b8161f6ae8f" containerName="main"
Nov 02 03:38:34.008306 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:34.008306    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c08a1ba9-e651-4714-b84d-9b8161f6ae8f" containerName="main"
Nov 02 03:38:34.008439 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:34.008430    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="c08a1ba9-e651-4714-b84d-9b8161f6ae8f" containerName="main"
Nov 02 03:38:34.013511 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:34.013491    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d94cc3-55f675b9d8-46f48"
Nov 02 03:38:34.068107 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:34.068076    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d0cd262-c6a0-4b21-af3c-393ee8051f81-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-d94cc3-55f675b9d8-46f48\" (UID: \"2d0cd262-c6a0-4b21-af3c-393ee8051f81\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d94cc3-55f675b9d8-46f48"
Nov 02 03:38:34.068304 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:34.068284    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d0cd262-c6a0-4b21-af3c-393ee8051f81-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-d94cc3-55f675b9d8-46f48\" (UID: \"2d0cd262-c6a0-4b21-af3c-393ee8051f81\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d94cc3-55f675b9d8-46f48"
Nov 02 03:38:34.068394 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:34.068319    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d0cd262-c6a0-4b21-af3c-393ee8051f81-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-d94cc3-55f675b9d8-46f48\" (UID: \"2d0cd262-c6a0-4b21-af3c-393ee8051f81\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d94cc3-55f675b9d8-46f48"
Nov 02 03:38:34.068394 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:34.068357    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgkv8\" (UniqueName: \"kubernetes.io/projected/2d0cd262-c6a0-4b21-af3c-393ee8051f81-kube-api-access-cgkv8\") pod \"maas-api-e2e-shared-b-w2-d94cc3-55f675b9d8-46f48\" (UID: \"2d0cd262-c6a0-4b21-af3c-393ee8051f81\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d94cc3-55f675b9d8-46f48"
Nov 02 03:38:34.069544 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:34.069521    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d94cc3-55f675b9d8-46f48"]
Nov 02 03:38:34.169931 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:34.169889    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d0cd262-c6a0-4b21-af3c-393ee8051f81-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-d94cc3-55f675b9d8-46f48\" (UID: \"2d0cd262-c6a0-4b21-af3c-393ee8051f81\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d94cc3-55f675b9d8-46f48"
Nov 02 03:38:34.170157 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:34.170061    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d0cd262-c6a0-4b21-af3c-393ee8051f81-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-d94cc3-55f675b9d8-46f48\" (UID: \"2d0cd262-c6a0-4b21-af3c-393ee8051f81\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d94cc3-55f675b9d8-46f48"
Nov 02 03:38:34.170157 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:34.170101    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d0cd262-c6a0-4b21-af3c-393ee8051f81-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-d94cc3-55f675b9d8-46f48\" (UID: \"2d0cd262-c6a0-4b21-af3c-393ee8051f81\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d94cc3-55f675b9d8-46f48"
Nov 02 03:38:34.170157 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:34.170139    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgkv8\" (UniqueName: \"kubernetes.io/projected/2d0cd262-c6a0-4b21-af3c-393ee8051f81-kube-api-access-cgkv8\") pod \"maas-api-e2e-shared-b-w2-d94cc3-55f675b9d8-46f48\" (UID: \"2d0cd262-c6a0-4b21-af3c-393ee8051f81\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d94cc3-55f675b9d8-46f48"
Nov 02 03:38:34.170709 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:34.170684    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d0cd262-c6a0-4b21-af3c-393ee8051f81-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-d94cc3-55f675b9d8-46f48\" (UID: \"2d0cd262-c6a0-4b21-af3c-393ee8051f81\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d94cc3-55f675b9d8-46f48"
Nov 02 03:38:34.172677 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:34.172654    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d0cd262-c6a0-4b21-af3c-393ee8051f81-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-d94cc3-55f675b9d8-46f48\" (UID: \"2d0cd262-c6a0-4b21-af3c-393ee8051f81\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d94cc3-55f675b9d8-46f48"
Nov 02 03:38:34.172799 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:34.172780    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d0cd262-c6a0-4b21-af3c-393ee8051f81-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-d94cc3-55f675b9d8-46f48\" (UID: \"2d0cd262-c6a0-4b21-af3c-393ee8051f81\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d94cc3-55f675b9d8-46f48"
Nov 02 03:38:34.227234 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:34.227202    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgkv8\" (UniqueName: \"kubernetes.io/projected/2d0cd262-c6a0-4b21-af3c-393ee8051f81-kube-api-access-cgkv8\") pod \"maas-api-e2e-shared-b-w2-d94cc3-55f675b9d8-46f48\" (UID: \"2d0cd262-c6a0-4b21-af3c-393ee8051f81\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d94cc3-55f675b9d8-46f48"
Nov 02 03:38:34.327571 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:34.327494    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d94cc3-55f675b9d8-46f48"
Nov 02 03:38:34.742524 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:34.742495    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d94cc3-55f675b9d8-46f48"]
Nov 02 03:38:34.743885 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:38:34.743853    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d0cd262_c6a0_4b21_af3c_393ee8051f81.slice/crio-941248c46f4a58c6b51e480b6ec09381f0d483d759394ddb40e8b91e19700c15 WatchSource:0}: Error finding container 941248c46f4a58c6b51e480b6ec09381f0d483d759394ddb40e8b91e19700c15: Status 404 returned error can't find the container with id 941248c46f4a58c6b51e480b6ec09381f0d483d759394ddb40e8b91e19700c15
Nov 02 03:38:35.364351 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:35.364313    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d94cc3-55f675b9d8-46f48" event={"ID":"2d0cd262-c6a0-4b21-af3c-393ee8051f81","Type":"ContainerStarted","Data":"941248c46f4a58c6b51e480b6ec09381f0d483d759394ddb40e8b91e19700c15"}
Nov 02 03:38:36.373590 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:36.373532    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d94cc3-55f675b9d8-46f48" event={"ID":"2d0cd262-c6a0-4b21-af3c-393ee8051f81","Type":"ContainerStarted","Data":"7d8866a2ed44fbe8663451d206fa8e5e26f9559eacb89e93f861871d314e6af3"}
Nov 02 03:38:36.373974 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:36.373642    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d94cc3-55f675b9d8-46f48"
Nov 02 03:38:36.490763 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:36.490704    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d94cc3-55f675b9d8-46f48" podStartSLOduration=2.678877913 podStartE2EDuration="3.490686228s" totalImagesPullingTime="811.808315ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:38:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:38:36.490325328 +0000 UTC m=+3057.068932709" watchObservedRunningTime="2026-11-02 03:38:36.490686228 +0000 UTC m=+3057.069293612"
Nov 02 03:38:42.387632 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:42.387595    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d94cc3-55f675b9d8-46f48"
Nov 02 03:38:42.561916 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:42.561881    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d94cc3-874bc47d6-hs5kd"]
Nov 02 03:38:42.562456 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:42.562400    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d94cc3-874bc47d6-hs5kd" podUID="32681665-956b-4346-b466-f7257f9ba39f" containerName="maas-api" containerID="cri-o://84e512aca4cee294e5d1ef6e7141592687fb9ff6b4adcf1f8af4434e0e9772d8" gracePeriod=30
Nov 02 03:38:42.826290 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:42.826265    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d94cc3-874bc47d6-hs5kd"
Nov 02 03:38:42.975425 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:42.975398    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32681665-956b-4346-b466-f7257f9ba39f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32681665-956b-4346-b466-f7257f9ba39f-openshift-service-ca\") pod \"32681665-956b-4346-b466-f7257f9ba39f\" (UID: \"32681665-956b-4346-b466-f7257f9ba39f\") "
Nov 02 03:38:42.975607 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:42.975477    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32681665-956b-4346-b466-f7257f9ba39f-kube-api-access-s4zcb\" (UniqueName: \"kubernetes.io/projected/32681665-956b-4346-b466-f7257f9ba39f-kube-api-access-s4zcb\") pod \"32681665-956b-4346-b466-f7257f9ba39f\" (UID: \"32681665-956b-4346-b466-f7257f9ba39f\") "
Nov 02 03:38:42.975607 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:42.975555    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32681665-956b-4346-b466-f7257f9ba39f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32681665-956b-4346-b466-f7257f9ba39f-maas-api-tls\") pod \"32681665-956b-4346-b466-f7257f9ba39f\" (UID: \"32681665-956b-4346-b466-f7257f9ba39f\") "
Nov 02 03:38:42.975607 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:42.975586    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32681665-956b-4346-b466-f7257f9ba39f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/32681665-956b-4346-b466-f7257f9ba39f-metrics-certs\") pod \"32681665-956b-4346-b466-f7257f9ba39f\" (UID: \"32681665-956b-4346-b466-f7257f9ba39f\") "
Nov 02 03:38:42.975866 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:42.975840    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32681665-956b-4346-b466-f7257f9ba39f-openshift-service-ca" pod "32681665-956b-4346-b466-f7257f9ba39f" (UID: "32681665-956b-4346-b466-f7257f9ba39f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:38:42.976056 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:42.976039    4496 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32681665-956b-4346-b466-f7257f9ba39f-openshift-service-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:38:42.978453 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:42.978413    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32681665-956b-4346-b466-f7257f9ba39f-maas-api-tls" pod "32681665-956b-4346-b466-f7257f9ba39f" (UID: "32681665-956b-4346-b466-f7257f9ba39f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 03:38:42.978570 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:42.978475    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32681665-956b-4346-b466-f7257f9ba39f-metrics-certs" pod "32681665-956b-4346-b466-f7257f9ba39f" (UID: "32681665-956b-4346-b466-f7257f9ba39f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 03:38:42.978570 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:42.978491    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32681665-956b-4346-b466-f7257f9ba39f-kube-api-access-s4zcb" pod "32681665-956b-4346-b466-f7257f9ba39f" (UID: "32681665-956b-4346-b466-f7257f9ba39f"). InnerVolumeSpecName "kube-api-access-s4zcb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:38:43.077197 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:43.077160    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32681665-956b-4346-b466-f7257f9ba39f-maas-api-tls\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:38:43.077197 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:43.077192    4496 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32681665-956b-4346-b466-f7257f9ba39f-metrics-certs\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:38:43.077409 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:43.077208    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4zcb\" (UniqueName: \"kubernetes.io/projected/32681665-956b-4346-b466-f7257f9ba39f-kube-api-access-s4zcb\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:38:43.428472 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:43.428378    4496 generic.go:356] "Generic (PLEG): container finished" podID="32681665-956b-4346-b466-f7257f9ba39f" containerID="84e512aca4cee294e5d1ef6e7141592687fb9ff6b4adcf1f8af4434e0e9772d8" exitCode=0
Nov 02 03:38:43.428472 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:43.428425    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d94cc3-874bc47d6-hs5kd"
Nov 02 03:38:43.428472 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:43.428427    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d94cc3-874bc47d6-hs5kd" event={"ID":"32681665-956b-4346-b466-f7257f9ba39f","Type":"ContainerDied","Data":"84e512aca4cee294e5d1ef6e7141592687fb9ff6b4adcf1f8af4434e0e9772d8"}
Nov 02 03:38:43.429037 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:43.428479    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d94cc3-874bc47d6-hs5kd" event={"ID":"32681665-956b-4346-b466-f7257f9ba39f","Type":"ContainerDied","Data":"22e23b6eae0dff7374e01a356010a1f0ea74139068cd8ab3dc1a64adf04c9ab3"}
Nov 02 03:38:43.429037 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:43.428511    4496 scope.go:122] "RemoveContainer" containerID="84e512aca4cee294e5d1ef6e7141592687fb9ff6b4adcf1f8af4434e0e9772d8"
Nov 02 03:38:43.442773 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:43.442748    4496 scope.go:122] "RemoveContainer" containerID="84e512aca4cee294e5d1ef6e7141592687fb9ff6b4adcf1f8af4434e0e9772d8"
Nov 02 03:38:43.443141 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:38:43.443109    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84e512aca4cee294e5d1ef6e7141592687fb9ff6b4adcf1f8af4434e0e9772d8\": container with ID starting with 84e512aca4cee294e5d1ef6e7141592687fb9ff6b4adcf1f8af4434e0e9772d8 not found: ID does not exist" containerID="84e512aca4cee294e5d1ef6e7141592687fb9ff6b4adcf1f8af4434e0e9772d8"
Nov 02 03:38:43.443262 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:43.443149    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84e512aca4cee294e5d1ef6e7141592687fb9ff6b4adcf1f8af4434e0e9772d8"} err="failed to get container status \"84e512aca4cee294e5d1ef6e7141592687fb9ff6b4adcf1f8af4434e0e9772d8\": rpc error: code = NotFound desc = could not find container \"84e512aca4cee294e5d1ef6e7141592687fb9ff6b4adcf1f8af4434e0e9772d8\": container with ID starting with 84e512aca4cee294e5d1ef6e7141592687fb9ff6b4adcf1f8af4434e0e9772d8 not found: ID does not exist"
Nov 02 03:38:43.597730 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:43.597696    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d94cc3-874bc47d6-hs5kd"]
Nov 02 03:38:43.648804 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:43.648773    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d94cc3-874bc47d6-hs5kd"]
Nov 02 03:38:45.576576 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:45.576538    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32681665-956b-4346-b466-f7257f9ba39f" path="/var/lib/kubelet/pods/32681665-956b-4346-b466-f7257f9ba39f/volumes"
Nov 02 03:38:58.892109 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:58.892029    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-3d563c-55dbc6658b-tbmjl"]
Nov 02 03:38:58.892705 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:58.892690    4496 cpu_manager.go:414] "containerMap: removing container" podUID="32681665-956b-4346-b466-f7257f9ba39f" containerName="maas-api"
Nov 02 03:38:58.892750 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:58.892706    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32681665-956b-4346-b466-f7257f9ba39f" containerName="maas-api"
Nov 02 03:38:58.892834 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:58.892825    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="32681665-956b-4346-b466-f7257f9ba39f" containerName="maas-api"
Nov 02 03:38:58.897933 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:58.897917    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3d563c-55dbc6658b-tbmjl"
Nov 02 03:38:58.954715 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:58.954682    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-3d563c-55dbc6658b-tbmjl"]
Nov 02 03:38:58.961518 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:58.961492    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blhmh\" (UniqueName: \"kubernetes.io/projected/c6ab02a3-6d61-4e1b-a12e-d8c576c7377c-kube-api-access-blhmh\") pod \"maas-api-e2e-praxis-b-3d563c-55dbc6658b-tbmjl\" (UID: \"c6ab02a3-6d61-4e1b-a12e-d8c576c7377c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3d563c-55dbc6658b-tbmjl"
Nov 02 03:38:58.961720 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:58.961696    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6ab02a3-6d61-4e1b-a12e-d8c576c7377c-metrics-certs\") pod \"maas-api-e2e-praxis-b-3d563c-55dbc6658b-tbmjl\" (UID: \"c6ab02a3-6d61-4e1b-a12e-d8c576c7377c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3d563c-55dbc6658b-tbmjl"
Nov 02 03:38:58.961798 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:58.961784    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6ab02a3-6d61-4e1b-a12e-d8c576c7377c-maas-api-tls\") pod \"maas-api-e2e-praxis-b-3d563c-55dbc6658b-tbmjl\" (UID: \"c6ab02a3-6d61-4e1b-a12e-d8c576c7377c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3d563c-55dbc6658b-tbmjl"
Nov 02 03:38:58.961860 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:58.961812    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6ab02a3-6d61-4e1b-a12e-d8c576c7377c-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-3d563c-55dbc6658b-tbmjl\" (UID: \"c6ab02a3-6d61-4e1b-a12e-d8c576c7377c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3d563c-55dbc6658b-tbmjl"
Nov 02 03:38:58.999461 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:58.999433    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-3d563c-serving-cert\""
Nov 02 03:38:59.062739 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:59.062706    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6ab02a3-6d61-4e1b-a12e-d8c576c7377c-maas-api-tls\") pod \"maas-api-e2e-praxis-b-3d563c-55dbc6658b-tbmjl\" (UID: \"c6ab02a3-6d61-4e1b-a12e-d8c576c7377c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3d563c-55dbc6658b-tbmjl"
Nov 02 03:38:59.062902 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:59.062753    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6ab02a3-6d61-4e1b-a12e-d8c576c7377c-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-3d563c-55dbc6658b-tbmjl\" (UID: \"c6ab02a3-6d61-4e1b-a12e-d8c576c7377c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3d563c-55dbc6658b-tbmjl"
Nov 02 03:38:59.062902 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:59.062800    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blhmh\" (UniqueName: \"kubernetes.io/projected/c6ab02a3-6d61-4e1b-a12e-d8c576c7377c-kube-api-access-blhmh\") pod \"maas-api-e2e-praxis-b-3d563c-55dbc6658b-tbmjl\" (UID: \"c6ab02a3-6d61-4e1b-a12e-d8c576c7377c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3d563c-55dbc6658b-tbmjl"
Nov 02 03:38:59.063008 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:59.062991    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6ab02a3-6d61-4e1b-a12e-d8c576c7377c-metrics-certs\") pod \"maas-api-e2e-praxis-b-3d563c-55dbc6658b-tbmjl\" (UID: \"c6ab02a3-6d61-4e1b-a12e-d8c576c7377c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3d563c-55dbc6658b-tbmjl"
Nov 02 03:38:59.064224 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:38:59.064195    4496 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-3d563c-serving-cert: secret "maas-api-e2e-praxis-b-3d563c-serving-cert" not found
Nov 02 03:38:59.064224 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:59.064214    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6ab02a3-6d61-4e1b-a12e-d8c576c7377c-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-3d563c-55dbc6658b-tbmjl\" (UID: \"c6ab02a3-6d61-4e1b-a12e-d8c576c7377c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3d563c-55dbc6658b-tbmjl"
Nov 02 03:38:59.064436 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:38:59.064270    4496 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6ab02a3-6d61-4e1b-a12e-d8c576c7377c-maas-api-tls podName:c6ab02a3-6d61-4e1b-a12e-d8c576c7377c nodeName:}" failed. No retries permitted until 2026-11-02 03:38:59.564251545 +0000 UTC m=+3080.142858920 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c6ab02a3-6d61-4e1b-a12e-d8c576c7377c-maas-api-tls") pod "maas-api-e2e-praxis-b-3d563c-55dbc6658b-tbmjl" (UID: "c6ab02a3-6d61-4e1b-a12e-d8c576c7377c") : secret "maas-api-e2e-praxis-b-3d563c-serving-cert" not found
Nov 02 03:38:59.065733 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:59.065702    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6ab02a3-6d61-4e1b-a12e-d8c576c7377c-metrics-certs\") pod \"maas-api-e2e-praxis-b-3d563c-55dbc6658b-tbmjl\" (UID: \"c6ab02a3-6d61-4e1b-a12e-d8c576c7377c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3d563c-55dbc6658b-tbmjl"
Nov 02 03:38:59.122305 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:59.122280    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blhmh\" (UniqueName: \"kubernetes.io/projected/c6ab02a3-6d61-4e1b-a12e-d8c576c7377c-kube-api-access-blhmh\") pod \"maas-api-e2e-praxis-b-3d563c-55dbc6658b-tbmjl\" (UID: \"c6ab02a3-6d61-4e1b-a12e-d8c576c7377c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3d563c-55dbc6658b-tbmjl"
Nov 02 03:38:59.567644 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:59.567611    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6ab02a3-6d61-4e1b-a12e-d8c576c7377c-maas-api-tls\") pod \"maas-api-e2e-praxis-b-3d563c-55dbc6658b-tbmjl\" (UID: \"c6ab02a3-6d61-4e1b-a12e-d8c576c7377c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3d563c-55dbc6658b-tbmjl"
Nov 02 03:38:59.570158 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:59.570130    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6ab02a3-6d61-4e1b-a12e-d8c576c7377c-maas-api-tls\") pod \"maas-api-e2e-praxis-b-3d563c-55dbc6658b-tbmjl\" (UID: \"c6ab02a3-6d61-4e1b-a12e-d8c576c7377c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3d563c-55dbc6658b-tbmjl"
Nov 02 03:38:59.812796 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:38:59.812759    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3d563c-55dbc6658b-tbmjl"
Nov 02 03:39:00.198289 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:39:00.198252    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-3d563c-55dbc6658b-tbmjl"]
Nov 02 03:39:00.198715 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:39:00.198296    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6ab02a3_6d61_4e1b_a12e_d8c576c7377c.slice/crio-8fb373969071d721cdd1745545c0f963b39914a3aed50f87d057ec1bc3abfbc5 WatchSource:0}: Error finding container 8fb373969071d721cdd1745545c0f963b39914a3aed50f87d057ec1bc3abfbc5: Status 404 returned error can't find the container with id 8fb373969071d721cdd1745545c0f963b39914a3aed50f87d057ec1bc3abfbc5
Nov 02 03:39:00.572460 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:39:00.572425    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3d563c-55dbc6658b-tbmjl" event={"ID":"c6ab02a3-6d61-4e1b-a12e-d8c576c7377c","Type":"ContainerStarted","Data":"8fb373969071d721cdd1745545c0f963b39914a3aed50f87d057ec1bc3abfbc5"}
Nov 02 03:39:01.583416 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:39:01.583372    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3d563c-55dbc6658b-tbmjl" event={"ID":"c6ab02a3-6d61-4e1b-a12e-d8c576c7377c","Type":"ContainerStarted","Data":"b30586882ccbb5b59960347b148f0f2b127dc38efe1cbeb173335b6ce9eb3be6"}
Nov 02 03:39:01.585370 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:39:01.583463    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3d563c-55dbc6658b-tbmjl"
Nov 02 03:39:01.709217 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:39:01.709168    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3d563c-55dbc6658b-tbmjl" podStartSLOduration=2.869152594 podStartE2EDuration="3.709153339s" totalImagesPullingTime="840.000745ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:38:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:39:01.705511655 +0000 UTC m=+3082.284119048" watchObservedRunningTime="2026-11-02 03:39:01.709153339 +0000 UTC m=+3082.287760719"
Nov 02 03:39:03.530791 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:39:03.530752    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 03:39:07.597983 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:39:07.597950    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3d563c-55dbc6658b-tbmjl"
Nov 02 03:39:08.704798 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:39:08.704761    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-2912e6-55ccd89d6-w6hh7"]
Nov 02 03:39:08.709590 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:39:08.709568    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2912e6-55ccd89d6-w6hh7"
Nov 02 03:39:08.764865 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:39:08.764833    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-2912e6-55ccd89d6-w6hh7"]
Nov 02 03:39:08.775226 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:39:08.775193    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bn6s\" (UniqueName: \"kubernetes.io/projected/305183a6-4694-4195-98e1-a0a8a5deaf51-kube-api-access-7bn6s\") pod \"maas-api-e2e-praxis-a-2912e6-55ccd89d6-w6hh7\" (UID: \"305183a6-4694-4195-98e1-a0a8a5deaf51\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2912e6-55ccd89d6-w6hh7"
Nov 02 03:39:08.775372 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:39:08.775254    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/305183a6-4694-4195-98e1-a0a8a5deaf51-maas-api-tls\") pod \"maas-api-e2e-praxis-a-2912e6-55ccd89d6-w6hh7\" (UID: \"305183a6-4694-4195-98e1-a0a8a5deaf51\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2912e6-55ccd89d6-w6hh7"
Nov 02 03:39:08.775442 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:39:08.775407    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/305183a6-4694-4195-98e1-a0a8a5deaf51-metrics-certs\") pod \"maas-api-e2e-praxis-a-2912e6-55ccd89d6-w6hh7\" (UID: \"305183a6-4694-4195-98e1-a0a8a5deaf51\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2912e6-55ccd89d6-w6hh7"
Nov 02 03:39:08.775508 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:39:08.775467    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/305183a6-4694-4195-98e1-a0a8a5deaf51-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-2912e6-55ccd89d6-w6hh7\" (UID: \"305183a6-4694-4195-98e1-a0a8a5deaf51\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2912e6-55ccd89d6-w6hh7"
Nov 02 03:39:08.811232 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:39:08.811205    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-2912e6-serving-cert\""
Nov 02 03:39:08.876924 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:39:08.876888    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/305183a6-4694-4195-98e1-a0a8a5deaf51-maas-api-tls\") pod \"maas-api-e2e-praxis-a-2912e6-55ccd89d6-w6hh7\" (UID: \"305183a6-4694-4195-98e1-a0a8a5deaf51\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2912e6-55ccd89d6-w6hh7"
Nov 02 03:39:08.877096 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:39:08.876986    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/305183a6-4694-4195-98e1-a0a8a5deaf51-metrics-certs\") pod \"maas-api-e2e-praxis-a-2912e6-55ccd89d6-w6hh7\" (UID: \"305183a6-4694-4195-98e1-a0a8a5deaf51\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2912e6-55ccd89d6-w6hh7"
Nov 02 03:39:08.877096 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:39:08.877071    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/305183a6-4694-4195-98e1-a0a8a5deaf51-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-2912e6-55ccd89d6-w6hh7\" (UID: \"305183a6-4694-4195-98e1-a0a8a5deaf51\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2912e6-55ccd89d6-w6hh7"
Nov 02 03:39:08.877184 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:39:08.877088    4496 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-2912e6-serving-cert: secret "maas-api-e2e-praxis-a-2912e6-serving-cert" not found
Nov 02 03:39:08.877184 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:39:08.877164    4496 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/305183a6-4694-4195-98e1-a0a8a5deaf51-maas-api-tls podName:305183a6-4694-4195-98e1-a0a8a5deaf51 nodeName:}" failed. No retries permitted until 2026-11-02 03:39:09.377144534 +0000 UTC m=+3089.955751911 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/305183a6-4694-4195-98e1-a0a8a5deaf51-maas-api-tls") pod "maas-api-e2e-praxis-a-2912e6-55ccd89d6-w6hh7" (UID: "305183a6-4694-4195-98e1-a0a8a5deaf51") : secret "maas-api-e2e-praxis-a-2912e6-serving-cert" not found
Nov 02 03:39:08.877258 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:39:08.877209    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bn6s\" (UniqueName: \"kubernetes.io/projected/305183a6-4694-4195-98e1-a0a8a5deaf51-kube-api-access-7bn6s\") pod \"maas-api-e2e-praxis-a-2912e6-55ccd89d6-w6hh7\" (UID: \"305183a6-4694-4195-98e1-a0a8a5deaf51\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2912e6-55ccd89d6-w6hh7"
Nov 02 03:39:08.877837 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:39:08.877810    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/305183a6-4694-4195-98e1-a0a8a5deaf51-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-2912e6-55ccd89d6-w6hh7\" (UID: \"305183a6-4694-4195-98e1-a0a8a5deaf51\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2912e6-55ccd89d6-w6hh7"
Nov 02 03:39:08.879404 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:39:08.879385    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/305183a6-4694-4195-98e1-a0a8a5deaf51-metrics-certs\") pod \"maas-api-e2e-praxis-a-2912e6-55ccd89d6-w6hh7\" (UID: \"305183a6-4694-4195-98e1-a0a8a5deaf51\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2912e6-55ccd89d6-w6hh7"
Nov 02 03:39:08.935225 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:39:08.935195    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bn6s\" (UniqueName: \"kubernetes.io/projected/305183a6-4694-4195-98e1-a0a8a5deaf51-kube-api-access-7bn6s\") pod \"maas-api-e2e-praxis-a-2912e6-55ccd89d6-w6hh7\" (UID: \"305183a6-4694-4195-98e1-a0a8a5deaf51\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2912e6-55ccd89d6-w6hh7"
Nov 02 03:39:09.383917 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:39:09.383878    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/305183a6-4694-4195-98e1-a0a8a5deaf51-maas-api-tls\") pod \"maas-api-e2e-praxis-a-2912e6-55ccd89d6-w6hh7\" (UID: \"305183a6-4694-4195-98e1-a0a8a5deaf51\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2912e6-55ccd89d6-w6hh7"
Nov 02 03:39:09.386316 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:39:09.386289    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/305183a6-4694-4195-98e1-a0a8a5deaf51-maas-api-tls\") pod \"maas-api-e2e-praxis-a-2912e6-55ccd89d6-w6hh7\" (UID: \"305183a6-4694-4195-98e1-a0a8a5deaf51\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2912e6-55ccd89d6-w6hh7"
Nov 02 03:39:09.624535 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:39:09.624503    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2912e6-55ccd89d6-w6hh7"
Nov 02 03:39:10.004195 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:39:10.004166    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-2912e6-55ccd89d6-w6hh7"]
Nov 02 03:39:10.004653 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:39:10.004624    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod305183a6_4694_4195_98e1_a0a8a5deaf51.slice/crio-e8ed5a48fd584a2d53b98aed44c989636ca768ba967ecc8a5ce3ed7bad5008c5 WatchSource:0}: Error finding container e8ed5a48fd584a2d53b98aed44c989636ca768ba967ecc8a5ce3ed7bad5008c5: Status 404 returned error can't find the container with id e8ed5a48fd584a2d53b98aed44c989636ca768ba967ecc8a5ce3ed7bad5008c5
Nov 02 03:39:10.651868 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:39:10.651825    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2912e6-55ccd89d6-w6hh7" event={"ID":"305183a6-4694-4195-98e1-a0a8a5deaf51","Type":"ContainerStarted","Data":"e8ed5a48fd584a2d53b98aed44c989636ca768ba967ecc8a5ce3ed7bad5008c5"}
Nov 02 03:39:11.661027 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:39:11.660983    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2912e6-55ccd89d6-w6hh7" event={"ID":"305183a6-4694-4195-98e1-a0a8a5deaf51","Type":"ContainerStarted","Data":"b68ed305e4757636be099f6d212f6933e9475b184cb01cf57b9abe9a58754f15"}
Nov 02 03:39:11.661477 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:39:11.661063    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2912e6-55ccd89d6-w6hh7"
Nov 02 03:39:11.825776 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:39:11.825712    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2912e6-55ccd89d6-w6hh7" podStartSLOduration=2.994134504 podStartE2EDuration="3.825693533s" totalImagesPullingTime="831.559029ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:39:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:39:11.819809659 +0000 UTC m=+3092.398417038" watchObservedRunningTime="2026-11-02 03:39:11.825693533 +0000 UTC m=+3092.404300915"
Nov 02 03:39:17.675566 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:39:17.675531    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2912e6-55ccd89d6-w6hh7"
Nov 02 03:39:18.552119 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:39:18.552081    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-cb56e9-64bd7c87c6-wcgp6"]
Nov 02 03:39:18.556743 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:39:18.556718    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cb56e9-64bd7c87c6-wcgp6"
Nov 02 03:39:18.614176 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:39:18.614143    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-cb56e9-64bd7c87c6-wcgp6"]
Nov 02 03:39:18.660430 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:39:18.660393    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-cb56e9-serving-cert\""
Nov 02 03:39:18.694364 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:39:18.694334    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68f8442f-e148-4090-a92c-af97a033cb47-metrics-certs\") pod \"maas-api-e2e-praxis-b-cb56e9-64bd7c87c6-wcgp6\" (UID: \"68f8442f-e148-4090-a92c-af97a033cb47\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cb56e9-64bd7c87c6-wcgp6"
Nov 02 03:39:18.694836 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:39:18.694406    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmwlf\" (UniqueName: \"kubernetes.io/projected/68f8442f-e148-4090-a92c-af97a033cb47-kube-api-access-xmwlf\") pod \"maas-api-e2e-praxis-b-cb56e9-64bd7c87c6-wcgp6\" (UID: \"68f8442f-e148-4090-a92c-af97a033cb47\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cb56e9-64bd7c87c6-wcgp6"
Nov 02 03:39:18.694836 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:39:18.694525    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68f8442f-e148-4090-a92c-af97a033cb47-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-cb56e9-64bd7c87c6-wcgp6\" (UID: \"68f8442f-e148-4090-a92c-af97a033cb47\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cb56e9-64bd7c87c6-wcgp6"
Nov 02 03:39:18.694836 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:39:18.694628    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68f8442f-e148-4090-a92c-af97a033cb47-maas-api-tls\") pod \"maas-api-e2e-praxis-b-cb56e9-64bd7c87c6-wcgp6\" (UID: \"68f8442f-e148-4090-a92c-af97a033cb47\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cb56e9-64bd7c87c6-wcgp6"
Nov 02 03:39:18.795920 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:39:18.795877    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68f8442f-e148-4090-a92c-af97a033cb47-metrics-certs\") pod \"maas-api-e2e-praxis-b-cb56e9-64bd7c87c6-wcgp6\" (UID: \"68f8442f-e148-4090-a92c-af97a033cb47\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cb56e9-64bd7c87c6-wcgp6"
Nov 02 03:39:18.796143 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:39:18.795949    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmwlf\" (UniqueName: \"kubernetes.io/projected/68f8442f-e148-4090-a92c-af97a033cb47-kube-api-access-xmwlf\") pod \"maas-api-e2e-praxis-b-cb56e9-64bd7c87c6-wcgp6\" (UID: \"68f8442f-e148-4090-a92c-af97a033cb47\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cb56e9-64bd7c87c6-wcgp6"
Nov 02 03:39:18.796143 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:39:18.795988    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68f8442f-e148-4090-a92c-af97a033cb47-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-cb56e9-64bd7c87c6-wcgp6\" (UID: \"68f8442f-e148-4090-a92c-af97a033cb47\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cb56e9-64bd7c87c6-wcgp6"
Nov 02 03:39:18.796143 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:39:18.796049    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68f8442f-e148-4090-a92c-af97a033cb47-maas-api-tls\") pod \"maas-api-e2e-praxis-b-cb56e9-64bd7c87c6-wcgp6\" (UID: \"68f8442f-e148-4090-a92c-af97a033cb47\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cb56e9-64bd7c87c6-wcgp6"
Nov 02 03:39:18.796336 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:39:18.796215    4496 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-cb56e9-serving-cert: secret "maas-api-e2e-praxis-b-cb56e9-serving-cert" not found
Nov 02 03:39:18.796336 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:39:18.796283    4496 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68f8442f-e148-4090-a92c-af97a033cb47-maas-api-tls podName:68f8442f-e148-4090-a92c-af97a033cb47 nodeName:}" failed. No retries permitted until 2026-11-02 03:39:19.296265617 +0000 UTC m=+3099.874872988 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/68f8442f-e148-4090-a92c-af97a033cb47-maas-api-tls") pod "maas-api-e2e-praxis-b-cb56e9-64bd7c87c6-wcgp6" (UID: "68f8442f-e148-4090-a92c-af97a033cb47") : secret "maas-api-e2e-praxis-b-cb56e9-serving-cert" not found
Nov 02 03:39:18.796997 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:39:18.796960    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68f8442f-e148-4090-a92c-af97a033cb47-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-cb56e9-64bd7c87c6-wcgp6\" (UID: \"68f8442f-e148-4090-a92c-af97a033cb47\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cb56e9-64bd7c87c6-wcgp6"
Nov 02 03:39:18.799896 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:39:18.799870    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68f8442f-e148-4090-a92c-af97a033cb47-metrics-certs\") pod \"maas-api-e2e-praxis-b-cb56e9-64bd7c87c6-wcgp6\" (UID: \"68f8442f-e148-4090-a92c-af97a033cb47\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cb56e9-64bd7c87c6-wcgp6"
Nov 02 03:39:18.858686 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:39:18.858613    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmwlf\" (UniqueName: \"kubernetes.io/projected/68f8442f-e148-4090-a92c-af97a033cb47-kube-api-access-xmwlf\") pod \"maas-api-e2e-praxis-b-cb56e9-64bd7c87c6-wcgp6\" (UID: \"68f8442f-e148-4090-a92c-af97a033cb47\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cb56e9-64bd7c87c6-wcgp6"
Nov 02 03:39:19.301409 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:39:19.301369    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68f8442f-e148-4090-a92c-af97a033cb47-maas-api-tls\") pod \"maas-api-e2e-praxis-b-cb56e9-64bd7c87c6-wcgp6\" (UID: \"68f8442f-e148-4090-a92c-af97a033cb47\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cb56e9-64bd7c87c6-wcgp6"
Nov 02 03:39:19.304094 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:39:19.304062    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68f8442f-e148-4090-a92c-af97a033cb47-maas-api-tls\") pod \"maas-api-e2e-praxis-b-cb56e9-64bd7c87c6-wcgp6\" (UID: \"68f8442f-e148-4090-a92c-af97a033cb47\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cb56e9-64bd7c87c6-wcgp6"
Nov 02 03:39:19.472737 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:39:19.472706    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cb56e9-64bd7c87c6-wcgp6"
Nov 02 03:39:19.863268 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:39:19.863234    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68f8442f_e148_4090_a92c_af97a033cb47.slice/crio-c8a08555274e328511473d3d2fd5063a50b6590bd8b129f9bc6bdacfc17fdb39 WatchSource:0}: Error finding container c8a08555274e328511473d3d2fd5063a50b6590bd8b129f9bc6bdacfc17fdb39: Status 404 returned error can't find the container with id c8a08555274e328511473d3d2fd5063a50b6590bd8b129f9bc6bdacfc17fdb39
Nov 02 03:39:19.910145 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:39:19.910116    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-cb56e9-64bd7c87c6-wcgp6"]
Nov 02 03:39:20.733194 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:39:20.733168    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cb56e9-64bd7c87c6-wcgp6" event={"ID":"68f8442f-e148-4090-a92c-af97a033cb47","Type":"ContainerStarted","Data":"c8a08555274e328511473d3d2fd5063a50b6590bd8b129f9bc6bdacfc17fdb39"}
Nov 02 03:39:21.746356 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:39:21.746313    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cb56e9-64bd7c87c6-wcgp6" event={"ID":"68f8442f-e148-4090-a92c-af97a033cb47","Type":"ContainerStarted","Data":"f50c364669055fe56a0bfac3c719de642f0bde7bbe511fb3c5377a6a7bb96019"}
Nov 02 03:39:21.867209 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:39:21.867151    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cb56e9-64bd7c87c6-wcgp6" podStartSLOduration=3.057987365 podStartE2EDuration="3.867133622s" totalImagesPullingTime="809.146257ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:39:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:39:21.865941933 +0000 UTC m=+3102.444549314" watchObservedRunningTime="2026-11-02 03:39:21.867133622 +0000 UTC m=+3102.445741001"
Nov 02 03:39:22.754386 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:39:22.754300    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cb56e9-64bd7c87c6-wcgp6"
Nov 02 03:39:28.768538 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:39:28.768502    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cb56e9-64bd7c87c6-wcgp6"
Nov 02 03:39:33.526289 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:39:33.526250    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 03:39:41.786940 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:39:41.786905    4496 scope.go:122] "RemoveContainer" containerID="6d9c8f6917cb46568343696957cf6816a16612aec548514bba6054065d1f9a35"
Nov 02 03:39:41.798141 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:39:41.798121    4496 scope.go:122] "RemoveContainer" containerID="78a0b6d27239fcf8d5a8b0a897d462ceb5e501b5b877af016da19766336a01d5"
Nov 02 03:39:42.829539 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:39:42.829499    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-1b1008-6b77584db9-pdxnh"]
Nov 02 03:39:42.833825 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:39:42.833803    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1b1008-6b77584db9-pdxnh"
Nov 02 03:39:42.891875 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:39:42.891844    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-1b1008-6b77584db9-pdxnh"]
Nov 02 03:39:42.925789 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:39:42.925737    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-1b1008-serving-cert\""
Nov 02 03:39:42.983977 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:39:42.983939    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57c07a0d-0051-43d1-b102-f6f47f6c9151-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-1b1008-6b77584db9-pdxnh\" (UID: \"57c07a0d-0051-43d1-b102-f6f47f6c9151\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1b1008-6b77584db9-pdxnh"
Nov 02 03:39:42.984201 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:39:42.984167    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fshf5\" (UniqueName: \"kubernetes.io/projected/57c07a0d-0051-43d1-b102-f6f47f6c9151-kube-api-access-fshf5\") pod \"maas-api-e2e-praxis-a-1b1008-6b77584db9-pdxnh\" (UID: \"57c07a0d-0051-43d1-b102-f6f47f6c9151\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1b1008-6b77584db9-pdxnh"
Nov 02 03:39:42.984353 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:39:42.984255    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57c07a0d-0051-43d1-b102-f6f47f6c9151-metrics-certs\") pod \"maas-api-e2e-praxis-a-1b1008-6b77584db9-pdxnh\" (UID: \"57c07a0d-0051-43d1-b102-f6f47f6c9151\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1b1008-6b77584db9-pdxnh"
Nov 02 03:39:42.984353 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:39:42.984334    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57c07a0d-0051-43d1-b102-f6f47f6c9151-maas-api-tls\") pod \"maas-api-e2e-praxis-a-1b1008-6b77584db9-pdxnh\" (UID: \"57c07a0d-0051-43d1-b102-f6f47f6c9151\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1b1008-6b77584db9-pdxnh"
Nov 02 03:39:43.086100 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:39:43.085992    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57c07a0d-0051-43d1-b102-f6f47f6c9151-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-1b1008-6b77584db9-pdxnh\" (UID: \"57c07a0d-0051-43d1-b102-f6f47f6c9151\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1b1008-6b77584db9-pdxnh"
Nov 02 03:39:43.086290 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:39:43.086106    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fshf5\" (UniqueName: \"kubernetes.io/projected/57c07a0d-0051-43d1-b102-f6f47f6c9151-kube-api-access-fshf5\") pod \"maas-api-e2e-praxis-a-1b1008-6b77584db9-pdxnh\" (UID: \"57c07a0d-0051-43d1-b102-f6f47f6c9151\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1b1008-6b77584db9-pdxnh"
Nov 02 03:39:43.086290 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:39:43.086140    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57c07a0d-0051-43d1-b102-f6f47f6c9151-metrics-certs\") pod \"maas-api-e2e-praxis-a-1b1008-6b77584db9-pdxnh\" (UID: \"57c07a0d-0051-43d1-b102-f6f47f6c9151\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1b1008-6b77584db9-pdxnh"
Nov 02 03:39:43.086290 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:39:43.086174    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57c07a0d-0051-43d1-b102-f6f47f6c9151-maas-api-tls\") pod \"maas-api-e2e-praxis-a-1b1008-6b77584db9-pdxnh\" (UID: \"57c07a0d-0051-43d1-b102-f6f47f6c9151\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1b1008-6b77584db9-pdxnh"
Nov 02 03:39:43.086472 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:39:43.086346    4496 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-1b1008-serving-cert: secret "maas-api-e2e-praxis-a-1b1008-serving-cert" not found
Nov 02 03:39:43.086472 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:39:43.086425    4496 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57c07a0d-0051-43d1-b102-f6f47f6c9151-maas-api-tls podName:57c07a0d-0051-43d1-b102-f6f47f6c9151 nodeName:}" failed. No retries permitted until 2026-11-02 03:39:43.586402714 +0000 UTC m=+3124.165010091 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/57c07a0d-0051-43d1-b102-f6f47f6c9151-maas-api-tls") pod "maas-api-e2e-praxis-a-1b1008-6b77584db9-pdxnh" (UID: "57c07a0d-0051-43d1-b102-f6f47f6c9151") : secret "maas-api-e2e-praxis-a-1b1008-serving-cert" not found
Nov 02 03:39:43.086781 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:39:43.086759    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57c07a0d-0051-43d1-b102-f6f47f6c9151-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-1b1008-6b77584db9-pdxnh\" (UID: \"57c07a0d-0051-43d1-b102-f6f47f6c9151\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1b1008-6b77584db9-pdxnh"
Nov 02 03:39:43.088554 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:39:43.088524    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57c07a0d-0051-43d1-b102-f6f47f6c9151-metrics-certs\") pod \"maas-api-e2e-praxis-a-1b1008-6b77584db9-pdxnh\" (UID: \"57c07a0d-0051-43d1-b102-f6f47f6c9151\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1b1008-6b77584db9-pdxnh"
Nov 02 03:39:43.144103 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:39:43.144052    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fshf5\" (UniqueName: \"kubernetes.io/projected/57c07a0d-0051-43d1-b102-f6f47f6c9151-kube-api-access-fshf5\") pod \"maas-api-e2e-praxis-a-1b1008-6b77584db9-pdxnh\" (UID: \"57c07a0d-0051-43d1-b102-f6f47f6c9151\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1b1008-6b77584db9-pdxnh"
Nov 02 03:39:43.591613 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:39:43.591569    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57c07a0d-0051-43d1-b102-f6f47f6c9151-maas-api-tls\") pod \"maas-api-e2e-praxis-a-1b1008-6b77584db9-pdxnh\" (UID: \"57c07a0d-0051-43d1-b102-f6f47f6c9151\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1b1008-6b77584db9-pdxnh"
Nov 02 03:39:43.594192 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:39:43.594167    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57c07a0d-0051-43d1-b102-f6f47f6c9151-maas-api-tls\") pod \"maas-api-e2e-praxis-a-1b1008-6b77584db9-pdxnh\" (UID: \"57c07a0d-0051-43d1-b102-f6f47f6c9151\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1b1008-6b77584db9-pdxnh"
Nov 02 03:39:43.749506 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:39:43.749467    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1b1008-6b77584db9-pdxnh"
Nov 02 03:39:44.130379 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:39:44.130353    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-1b1008-6b77584db9-pdxnh"]
Nov 02 03:39:44.130967 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:39:44.130942    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57c07a0d_0051_43d1_b102_f6f47f6c9151.slice/crio-60ff7c6f72eacb0a8f453da6d2945fd9ddc1ff93371a7e271e07d2298bf1348d WatchSource:0}: Error finding container 60ff7c6f72eacb0a8f453da6d2945fd9ddc1ff93371a7e271e07d2298bf1348d: Status 404 returned error can't find the container with id 60ff7c6f72eacb0a8f453da6d2945fd9ddc1ff93371a7e271e07d2298bf1348d
Nov 02 03:39:44.928388 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:39:44.928357    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1b1008-6b77584db9-pdxnh" event={"ID":"57c07a0d-0051-43d1-b102-f6f47f6c9151","Type":"ContainerStarted","Data":"60ff7c6f72eacb0a8f453da6d2945fd9ddc1ff93371a7e271e07d2298bf1348d"}
Nov 02 03:39:45.937833 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:39:45.937787    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1b1008-6b77584db9-pdxnh" event={"ID":"57c07a0d-0051-43d1-b102-f6f47f6c9151","Type":"ContainerStarted","Data":"81156118dafe93425d8c8967feaad8aa9642cb43085abbb53aa9a2d61e1062ca"}
Nov 02 03:39:45.938353 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:39:45.937888    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1b1008-6b77584db9-pdxnh"
Nov 02 03:39:46.059550 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:39:46.059482    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1b1008-6b77584db9-pdxnh" podStartSLOduration=3.164592731 podStartE2EDuration="4.059462739s" totalImagesPullingTime="894.870008ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:39:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:39:46.054746517 +0000 UTC m=+3126.633353909" watchObservedRunningTime="2026-11-02 03:39:46.059462739 +0000 UTC m=+3126.638070121"
Nov 02 03:39:51.951529 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:39:51.951498    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1b1008-6b77584db9-pdxnh"
Nov 02 03:40:03.527448 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:40:03.527410    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 03:40:33.526441 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:40:33.526400    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 03:40:41.838233 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:40:41.838197    4496 scope.go:122] "RemoveContainer" containerID="75ccc98797c6967cbce0093b4f2e511ce680a7634d982c4a62f4cbdda69d6963"
Nov 02 03:40:42.998644 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:40:42.998604    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-e44998-764fd4f68c-5ggnl"]
Nov 02 03:40:47.999546 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:40:47.999498    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e44998-764fd4f68c-5ggnl" podUID="ea831751-f990-46e0-bd3d-49b395a46fdd" containerName="payload-pre-processing" containerID="cri-o://604c1ff9ede27eb83012e738cfc4905a715cd0870d9773a5d25dc1acfd79e695" gracePeriod=25
Nov 02 03:40:48.048045 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:40:48.047991    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13a4b385-pnc2c"]
Nov 02 03:40:48.096448 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:40:48.096414    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13a4b385-pnc2c"]
Nov 02 03:40:48.289267 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:40:48.289238    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e44998-764fd4f68c-5ggnl"
Nov 02 03:40:48.367432 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:40:48.367395    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea831751-f990-46e0-bd3d-49b395a46fdd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea831751-f990-46e0-bd3d-49b395a46fdd-plugins-config-volume\") pod \"ea831751-f990-46e0-bd3d-49b395a46fdd\" (UID: \"ea831751-f990-46e0-bd3d-49b395a46fdd\") "
Nov 02 03:40:48.367629 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:40:48.367494    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea831751-f990-46e0-bd3d-49b395a46fdd-kube-api-access-f88kn\" (UniqueName: \"kubernetes.io/projected/ea831751-f990-46e0-bd3d-49b395a46fdd-kube-api-access-f88kn\") pod \"ea831751-f990-46e0-bd3d-49b395a46fdd\" (UID: \"ea831751-f990-46e0-bd3d-49b395a46fdd\") "
Nov 02 03:40:48.367819 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:40:48.367795    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea831751-f990-46e0-bd3d-49b395a46fdd-plugins-config-volume" pod "ea831751-f990-46e0-bd3d-49b395a46fdd" (UID: "ea831751-f990-46e0-bd3d-49b395a46fdd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:40:48.369705 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:40:48.369677    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea831751-f990-46e0-bd3d-49b395a46fdd-kube-api-access-f88kn" pod "ea831751-f990-46e0-bd3d-49b395a46fdd" (UID: "ea831751-f990-46e0-bd3d-49b395a46fdd"). InnerVolumeSpecName "kube-api-access-f88kn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:40:48.451495 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:40:48.451458    4496 generic.go:356] "Generic (PLEG): container finished" podID="ea831751-f990-46e0-bd3d-49b395a46fdd" containerID="604c1ff9ede27eb83012e738cfc4905a715cd0870d9773a5d25dc1acfd79e695" exitCode=0
Nov 02 03:40:48.451876 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:40:48.451763    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e44998-764fd4f68c-5ggnl"
Nov 02 03:40:48.452038 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:40:48.451784    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e44998-764fd4f68c-5ggnl" event={"ID":"ea831751-f990-46e0-bd3d-49b395a46fdd","Type":"ContainerDied","Data":"604c1ff9ede27eb83012e738cfc4905a715cd0870d9773a5d25dc1acfd79e695"}
Nov 02 03:40:48.458199 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:40:48.452172    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e44998-764fd4f68c-5ggnl" event={"ID":"ea831751-f990-46e0-bd3d-49b395a46fdd","Type":"ContainerDied","Data":"d35ed65af162987ec1a152c8c1c14e5b3cae00dc87fb4c217bd54c579d324208"}
Nov 02 03:40:48.458199 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:40:48.452213    4496 scope.go:122] "RemoveContainer" containerID="604c1ff9ede27eb83012e738cfc4905a715cd0870d9773a5d25dc1acfd79e695"
Nov 02 03:40:48.467100 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:40:48.467080    4496 scope.go:122] "RemoveContainer" containerID="604c1ff9ede27eb83012e738cfc4905a715cd0870d9773a5d25dc1acfd79e695"
Nov 02 03:40:48.467669 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:40:48.467641    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"604c1ff9ede27eb83012e738cfc4905a715cd0870d9773a5d25dc1acfd79e695\": container with ID starting with 604c1ff9ede27eb83012e738cfc4905a715cd0870d9773a5d25dc1acfd79e695 not found: ID does not exist" containerID="604c1ff9ede27eb83012e738cfc4905a715cd0870d9773a5d25dc1acfd79e695"
Nov 02 03:40:48.467783 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:40:48.467679    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"604c1ff9ede27eb83012e738cfc4905a715cd0870d9773a5d25dc1acfd79e695"} err="failed to get container status \"604c1ff9ede27eb83012e738cfc4905a715cd0870d9773a5d25dc1acfd79e695\": rpc error: code = NotFound desc = could not find container \"604c1ff9ede27eb83012e738cfc4905a715cd0870d9773a5d25dc1acfd79e695\": container with ID starting with 604c1ff9ede27eb83012e738cfc4905a715cd0870d9773a5d25dc1acfd79e695 not found: ID does not exist"
Nov 02 03:40:48.469521 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:40:48.469500    4496 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea831751-f990-46e0-bd3d-49b395a46fdd-plugins-config-volume\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:40:48.469631 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:40:48.469525    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f88kn\" (UniqueName: \"kubernetes.io/projected/ea831751-f990-46e0-bd3d-49b395a46fdd-kube-api-access-f88kn\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:40:48.621276 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:40:48.621192    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-e44998-764fd4f68c-5ggnl"]
Nov 02 03:40:48.628470 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:40:48.628438    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-e44998-764fd4f68c-5ggnl"]
Nov 02 03:40:49.577152 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:40:49.577114    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a2685ed-24a0-4ac6-838f-d47c7441f9a2" path="/var/lib/kubelet/pods/9a2685ed-24a0-4ac6-838f-d47c7441f9a2/volumes"
Nov 02 03:40:49.577666 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:40:49.577530    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea831751-f990-46e0-bd3d-49b395a46fdd" path="/var/lib/kubelet/pods/ea831751-f990-46e0-bd3d-49b395a46fdd/volumes"
Nov 02 03:40:54.621199 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:40:54.621164    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-0a04e0-5db76b8859-d6szv"]
Nov 02 03:40:54.729910 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:40:54.729875    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-0a04e0-bc9cc6559-cvsjg"]
Nov 02 03:40:59.621924 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:40:59.621878    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-0a04e0-5db76b8859-d6szv" podUID="2a3941e2-4480-4bc8-8b7a-b8f7a093a609" containerName="payload-processing" containerID="cri-o://e03e3af415607aa62cfea4a2dfb9cc842e74a558d24f550fe4e81e54093ae559" gracePeriod=25
Nov 02 03:40:59.730783 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:40:59.730732    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0a04e0-bc9cc6559-cvsjg" podUID="d9dbfe75-2071-49ea-a422-88c87d623118" containerName="payload-pre-processing" containerID="cri-o://5d3de9e6cec98eec1a77be771cbdea52fb037f009cc109f12360dd633afa6261" gracePeriod=25
Nov 02 03:40:59.894957 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:40:59.894891    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-0a04e0-5db76b8859-d6szv"
Nov 02 03:41:00.001945 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:00.001911    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a3941e2-4480-4bc8-8b7a-b8f7a093a609-kube-api-access-lpb2x\" (UniqueName: \"kubernetes.io/projected/2a3941e2-4480-4bc8-8b7a-b8f7a093a609-kube-api-access-lpb2x\") pod \"2a3941e2-4480-4bc8-8b7a-b8f7a093a609\" (UID: \"2a3941e2-4480-4bc8-8b7a-b8f7a093a609\") "
Nov 02 03:41:00.002100 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:00.002052    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a3941e2-4480-4bc8-8b7a-b8f7a093a609-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a3941e2-4480-4bc8-8b7a-b8f7a093a609-plugins-config-volume\") pod \"2a3941e2-4480-4bc8-8b7a-b8f7a093a609\" (UID: \"2a3941e2-4480-4bc8-8b7a-b8f7a093a609\") "
Nov 02 03:41:00.002480 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:00.002452    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a3941e2-4480-4bc8-8b7a-b8f7a093a609-plugins-config-volume" pod "2a3941e2-4480-4bc8-8b7a-b8f7a093a609" (UID: "2a3941e2-4480-4bc8-8b7a-b8f7a093a609"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:41:00.002790 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:00.002766    4496 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a3941e2-4480-4bc8-8b7a-b8f7a093a609-plugins-config-volume\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:41:00.012069 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:00.011061    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a3941e2-4480-4bc8-8b7a-b8f7a093a609-kube-api-access-lpb2x" pod "2a3941e2-4480-4bc8-8b7a-b8f7a093a609" (UID: "2a3941e2-4480-4bc8-8b7a-b8f7a093a609"). InnerVolumeSpecName "kube-api-access-lpb2x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:41:00.018784 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:00.018756    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0a04e0-bc9cc6559-cvsjg"
Nov 02 03:41:00.103875 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:00.103841    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9dbfe75-2071-49ea-a422-88c87d623118-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9dbfe75-2071-49ea-a422-88c87d623118-plugins-config-volume\") pod \"d9dbfe75-2071-49ea-a422-88c87d623118\" (UID: \"d9dbfe75-2071-49ea-a422-88c87d623118\") "
Nov 02 03:41:00.104110 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:00.104080    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9dbfe75-2071-49ea-a422-88c87d623118-kube-api-access-ckmbh\" (UniqueName: \"kubernetes.io/projected/d9dbfe75-2071-49ea-a422-88c87d623118-kube-api-access-ckmbh\") pod \"d9dbfe75-2071-49ea-a422-88c87d623118\" (UID: \"d9dbfe75-2071-49ea-a422-88c87d623118\") "
Nov 02 03:41:00.104290 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:00.104264    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9dbfe75-2071-49ea-a422-88c87d623118-plugins-config-volume" pod "d9dbfe75-2071-49ea-a422-88c87d623118" (UID: "d9dbfe75-2071-49ea-a422-88c87d623118"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:41:00.104435 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:00.104420    4496 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9dbfe75-2071-49ea-a422-88c87d623118-plugins-config-volume\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:41:00.104486 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:00.104437    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpb2x\" (UniqueName: \"kubernetes.io/projected/2a3941e2-4480-4bc8-8b7a-b8f7a093a609-kube-api-access-lpb2x\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:41:00.106163 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:00.106140    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9dbfe75-2071-49ea-a422-88c87d623118-kube-api-access-ckmbh" pod "d9dbfe75-2071-49ea-a422-88c87d623118" (UID: "d9dbfe75-2071-49ea-a422-88c87d623118"). InnerVolumeSpecName "kube-api-access-ckmbh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:41:00.205791 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:00.205708    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckmbh\" (UniqueName: \"kubernetes.io/projected/d9dbfe75-2071-49ea-a422-88c87d623118-kube-api-access-ckmbh\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:41:00.550387 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:00.550351    4496 generic.go:356] "Generic (PLEG): container finished" podID="2a3941e2-4480-4bc8-8b7a-b8f7a093a609" containerID="e03e3af415607aa62cfea4a2dfb9cc842e74a558d24f550fe4e81e54093ae559" exitCode=0
Nov 02 03:41:00.550578 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:00.550410    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-0a04e0-5db76b8859-d6szv"
Nov 02 03:41:00.550578 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:00.550427    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-0a04e0-5db76b8859-d6szv" event={"ID":"2a3941e2-4480-4bc8-8b7a-b8f7a093a609","Type":"ContainerDied","Data":"e03e3af415607aa62cfea4a2dfb9cc842e74a558d24f550fe4e81e54093ae559"}
Nov 02 03:41:00.550578 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:00.550503    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-0a04e0-5db76b8859-d6szv" event={"ID":"2a3941e2-4480-4bc8-8b7a-b8f7a093a609","Type":"ContainerDied","Data":"359c4effbe5a402285dc2bddb704f797048cd1af0c8cc7a295212f46fddc67f4"}
Nov 02 03:41:00.550578 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:00.550531    4496 scope.go:122] "RemoveContainer" containerID="e03e3af415607aa62cfea4a2dfb9cc842e74a558d24f550fe4e81e54093ae559"
Nov 02 03:41:00.551970 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:00.551945    4496 generic.go:356] "Generic (PLEG): container finished" podID="d9dbfe75-2071-49ea-a422-88c87d623118" containerID="5d3de9e6cec98eec1a77be771cbdea52fb037f009cc109f12360dd633afa6261" exitCode=0
Nov 02 03:41:00.552194 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:00.551974    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0a04e0-bc9cc6559-cvsjg" event={"ID":"d9dbfe75-2071-49ea-a422-88c87d623118","Type":"ContainerDied","Data":"5d3de9e6cec98eec1a77be771cbdea52fb037f009cc109f12360dd633afa6261"}
Nov 02 03:41:00.552194 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:00.551994    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0a04e0-bc9cc6559-cvsjg"
Nov 02 03:41:00.552194 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:00.552049    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0a04e0-bc9cc6559-cvsjg" event={"ID":"d9dbfe75-2071-49ea-a422-88c87d623118","Type":"ContainerDied","Data":"558726d96f09da68bcf828d830c80c89f754ae2fa63b92f1881f8bcda9f175c1"}
Nov 02 03:41:00.562777 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:00.562755    4496 scope.go:122] "RemoveContainer" containerID="e03e3af415607aa62cfea4a2dfb9cc842e74a558d24f550fe4e81e54093ae559"
Nov 02 03:41:00.563096 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:41:00.563079    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e03e3af415607aa62cfea4a2dfb9cc842e74a558d24f550fe4e81e54093ae559\": container with ID starting with e03e3af415607aa62cfea4a2dfb9cc842e74a558d24f550fe4e81e54093ae559 not found: ID does not exist" containerID="e03e3af415607aa62cfea4a2dfb9cc842e74a558d24f550fe4e81e54093ae559"
Nov 02 03:41:00.563163 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:00.563102    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e03e3af415607aa62cfea4a2dfb9cc842e74a558d24f550fe4e81e54093ae559"} err="failed to get container status \"e03e3af415607aa62cfea4a2dfb9cc842e74a558d24f550fe4e81e54093ae559\": rpc error: code = NotFound desc = could not find container \"e03e3af415607aa62cfea4a2dfb9cc842e74a558d24f550fe4e81e54093ae559\": container with ID starting with e03e3af415607aa62cfea4a2dfb9cc842e74a558d24f550fe4e81e54093ae559 not found: ID does not exist"
Nov 02 03:41:00.563163 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:00.563119    4496 scope.go:122] "RemoveContainer" containerID="5d3de9e6cec98eec1a77be771cbdea52fb037f009cc109f12360dd633afa6261"
Nov 02 03:41:00.572652 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:00.572627    4496 scope.go:122] "RemoveContainer" containerID="5d3de9e6cec98eec1a77be771cbdea52fb037f009cc109f12360dd633afa6261"
Nov 02 03:41:00.572892 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:41:00.572872    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d3de9e6cec98eec1a77be771cbdea52fb037f009cc109f12360dd633afa6261\": container with ID starting with 5d3de9e6cec98eec1a77be771cbdea52fb037f009cc109f12360dd633afa6261 not found: ID does not exist" containerID="5d3de9e6cec98eec1a77be771cbdea52fb037f009cc109f12360dd633afa6261"
Nov 02 03:41:00.572944 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:00.572898    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d3de9e6cec98eec1a77be771cbdea52fb037f009cc109f12360dd633afa6261"} err="failed to get container status \"5d3de9e6cec98eec1a77be771cbdea52fb037f009cc109f12360dd633afa6261\": rpc error: code = NotFound desc = could not find container \"5d3de9e6cec98eec1a77be771cbdea52fb037f009cc109f12360dd633afa6261\": container with ID starting with 5d3de9e6cec98eec1a77be771cbdea52fb037f009cc109f12360dd633afa6261 not found: ID does not exist"
Nov 02 03:41:00.717499 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:00.717468    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-0a04e0-bc9cc6559-cvsjg"]
Nov 02 03:41:00.722740 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:00.722713    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-0a04e0-bc9cc6559-cvsjg"]
Nov 02 03:41:00.834879 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:00.834803    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-0a04e0-5db76b8859-d6szv"]
Nov 02 03:41:00.838806 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:00.838770    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-0a04e0-5db76b8859-d6szv"]
Nov 02 03:41:01.574694 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:01.574660    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a3941e2-4480-4bc8-8b7a-b8f7a093a609" path="/var/lib/kubelet/pods/2a3941e2-4480-4bc8-8b7a-b8f7a093a609/volumes"
Nov 02 03:41:01.575088 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:01.575075    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9dbfe75-2071-49ea-a422-88c87d623118" path="/var/lib/kubelet/pods/d9dbfe75-2071-49ea-a422-88c87d623118/volumes"
Nov 02 03:41:02.390700 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:02.390660    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-9b06f8-56944cd46c-rgw5t"]
Nov 02 03:41:02.391341 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:02.391326    4496 cpu_manager.go:414] "containerMap: removing container" podUID="2a3941e2-4480-4bc8-8b7a-b8f7a093a609" containerName="payload-processing"
Nov 02 03:41:02.391396 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:02.391342    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a3941e2-4480-4bc8-8b7a-b8f7a093a609" containerName="payload-processing"
Nov 02 03:41:02.391396 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:02.391364    4496 cpu_manager.go:414] "containerMap: removing container" podUID="ea831751-f990-46e0-bd3d-49b395a46fdd" containerName="payload-pre-processing"
Nov 02 03:41:02.391396 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:02.391370    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea831751-f990-46e0-bd3d-49b395a46fdd" containerName="payload-pre-processing"
Nov 02 03:41:02.391396 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:02.391394    4496 cpu_manager.go:414] "containerMap: removing container" podUID="d9dbfe75-2071-49ea-a422-88c87d623118" containerName="payload-pre-processing"
Nov 02 03:41:02.391523 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:02.391400    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9dbfe75-2071-49ea-a422-88c87d623118" containerName="payload-pre-processing"
Nov 02 03:41:02.391523 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:02.391493    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a3941e2-4480-4bc8-8b7a-b8f7a093a609" containerName="payload-processing"
Nov 02 03:41:02.391523 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:02.391506    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea831751-f990-46e0-bd3d-49b395a46fdd" containerName="payload-pre-processing"
Nov 02 03:41:02.391523 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:02.391515    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9dbfe75-2071-49ea-a422-88c87d623118" containerName="payload-pre-processing"
Nov 02 03:41:02.396393 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:02.396377    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9b06f8-56944cd46c-rgw5t"
Nov 02 03:41:02.452769 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:02.452735    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-9b06f8-56944cd46c-rgw5t"]
Nov 02 03:41:02.497964 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:02.497930    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-9b06f8-serving-cert\""
Nov 02 03:41:02.531627 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:02.531592    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7801d774-3dc0-4f41-ae02-ad732b765da7-maas-api-tls\") pod \"maas-api-e2e-praxis-b-9b06f8-56944cd46c-rgw5t\" (UID: \"7801d774-3dc0-4f41-ae02-ad732b765da7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9b06f8-56944cd46c-rgw5t"
Nov 02 03:41:02.531823 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:02.531661    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpxrn\" (UniqueName: \"kubernetes.io/projected/7801d774-3dc0-4f41-ae02-ad732b765da7-kube-api-access-rpxrn\") pod \"maas-api-e2e-praxis-b-9b06f8-56944cd46c-rgw5t\" (UID: \"7801d774-3dc0-4f41-ae02-ad732b765da7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9b06f8-56944cd46c-rgw5t"
Nov 02 03:41:02.531823 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:02.531799    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7801d774-3dc0-4f41-ae02-ad732b765da7-metrics-certs\") pod \"maas-api-e2e-praxis-b-9b06f8-56944cd46c-rgw5t\" (UID: \"7801d774-3dc0-4f41-ae02-ad732b765da7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9b06f8-56944cd46c-rgw5t"
Nov 02 03:41:02.531943 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:02.531848    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7801d774-3dc0-4f41-ae02-ad732b765da7-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-9b06f8-56944cd46c-rgw5t\" (UID: \"7801d774-3dc0-4f41-ae02-ad732b765da7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9b06f8-56944cd46c-rgw5t"
Nov 02 03:41:02.633693 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:02.633648    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpxrn\" (UniqueName: \"kubernetes.io/projected/7801d774-3dc0-4f41-ae02-ad732b765da7-kube-api-access-rpxrn\") pod \"maas-api-e2e-praxis-b-9b06f8-56944cd46c-rgw5t\" (UID: \"7801d774-3dc0-4f41-ae02-ad732b765da7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9b06f8-56944cd46c-rgw5t"
Nov 02 03:41:02.633882 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:02.633728    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7801d774-3dc0-4f41-ae02-ad732b765da7-metrics-certs\") pod \"maas-api-e2e-praxis-b-9b06f8-56944cd46c-rgw5t\" (UID: \"7801d774-3dc0-4f41-ae02-ad732b765da7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9b06f8-56944cd46c-rgw5t"
Nov 02 03:41:02.633977 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:02.633898    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7801d774-3dc0-4f41-ae02-ad732b765da7-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-9b06f8-56944cd46c-rgw5t\" (UID: \"7801d774-3dc0-4f41-ae02-ad732b765da7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9b06f8-56944cd46c-rgw5t"
Nov 02 03:41:02.634093 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:02.634071    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7801d774-3dc0-4f41-ae02-ad732b765da7-maas-api-tls\") pod \"maas-api-e2e-praxis-b-9b06f8-56944cd46c-rgw5t\" (UID: \"7801d774-3dc0-4f41-ae02-ad732b765da7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9b06f8-56944cd46c-rgw5t"
Nov 02 03:41:02.634657 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:02.634634    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7801d774-3dc0-4f41-ae02-ad732b765da7-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-9b06f8-56944cd46c-rgw5t\" (UID: \"7801d774-3dc0-4f41-ae02-ad732b765da7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9b06f8-56944cd46c-rgw5t"
Nov 02 03:41:02.636522 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:02.636501    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7801d774-3dc0-4f41-ae02-ad732b765da7-maas-api-tls\") pod \"maas-api-e2e-praxis-b-9b06f8-56944cd46c-rgw5t\" (UID: \"7801d774-3dc0-4f41-ae02-ad732b765da7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9b06f8-56944cd46c-rgw5t"
Nov 02 03:41:02.636522 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:02.636509    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7801d774-3dc0-4f41-ae02-ad732b765da7-metrics-certs\") pod \"maas-api-e2e-praxis-b-9b06f8-56944cd46c-rgw5t\" (UID: \"7801d774-3dc0-4f41-ae02-ad732b765da7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9b06f8-56944cd46c-rgw5t"
Nov 02 03:41:02.697258 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:02.697188    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpxrn\" (UniqueName: \"kubernetes.io/projected/7801d774-3dc0-4f41-ae02-ad732b765da7-kube-api-access-rpxrn\") pod \"maas-api-e2e-praxis-b-9b06f8-56944cd46c-rgw5t\" (UID: \"7801d774-3dc0-4f41-ae02-ad732b765da7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9b06f8-56944cd46c-rgw5t"
Nov 02 03:41:02.710076 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:02.710047    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9b06f8-56944cd46c-rgw5t"
Nov 02 03:41:03.093762 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:03.093734    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-9b06f8-56944cd46c-rgw5t"]
Nov 02 03:41:03.094152 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:41:03.094124    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7801d774_3dc0_4f41_ae02_ad732b765da7.slice/crio-ecb7655dfa158969b27665bb13efd2de33ad11168d59dee429045aace9fdde90 WatchSource:0}: Error finding container ecb7655dfa158969b27665bb13efd2de33ad11168d59dee429045aace9fdde90: Status 404 returned error can't find the container with id ecb7655dfa158969b27665bb13efd2de33ad11168d59dee429045aace9fdde90
Nov 02 03:41:03.526563 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:41:03.526529    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 03:41:03.581687 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:03.581655    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9b06f8-56944cd46c-rgw5t" event={"ID":"7801d774-3dc0-4f41-ae02-ad732b765da7","Type":"ContainerStarted","Data":"ecb7655dfa158969b27665bb13efd2de33ad11168d59dee429045aace9fdde90"}
Nov 02 03:41:04.592186 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:04.592149    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9b06f8-56944cd46c-rgw5t" event={"ID":"7801d774-3dc0-4f41-ae02-ad732b765da7","Type":"ContainerStarted","Data":"fb6212fd410ac9e24c1bb509b001d867b16c7fb7672f770da176db0f7f9fadb1"}
Nov 02 03:41:04.592616 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:04.592410    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9b06f8-56944cd46c-rgw5t"
Nov 02 03:41:04.735706 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:04.735653    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9b06f8-56944cd46c-rgw5t" podStartSLOduration=1.359650721 podStartE2EDuration="2.735635721s" totalImagesPullingTime="1.375985s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:41:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:41:04.732995834 +0000 UTC m=+3205.311603215" watchObservedRunningTime="2026-11-02 03:41:04.735635721 +0000 UTC m=+3205.314243101"
Nov 02 03:41:10.605277 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:10.605248    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9b06f8-56944cd46c-rgw5t"
Nov 02 03:41:23.776770 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:23.776734    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-0a04e0-785685977-slvzx"]
Nov 02 03:41:23.778044 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:23.776962    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0a04e0-785685977-slvzx" podUID="95349ebe-7d57-45bf-8143-c82c14bd43fa" containerName="maas-api" containerID="cri-o://db0133280d738cfb5f11e6c30657fa7be80b8758c3c76fb9fa1a43a3896c0c3c" gracePeriod=30
Nov 02 03:41:24.063063 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:24.063032    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0a04e0-785685977-slvzx"
Nov 02 03:41:24.190290 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:24.190255    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95349ebe-7d57-45bf-8143-c82c14bd43fa-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95349ebe-7d57-45bf-8143-c82c14bd43fa-maas-api-tls\") pod \"95349ebe-7d57-45bf-8143-c82c14bd43fa\" (UID: \"95349ebe-7d57-45bf-8143-c82c14bd43fa\") "
Nov 02 03:41:24.190498 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:24.190346    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95349ebe-7d57-45bf-8143-c82c14bd43fa-metrics-certs\" (UniqueName: \"kubernetes.io/secret/95349ebe-7d57-45bf-8143-c82c14bd43fa-metrics-certs\") pod \"95349ebe-7d57-45bf-8143-c82c14bd43fa\" (UID: \"95349ebe-7d57-45bf-8143-c82c14bd43fa\") "
Nov 02 03:41:24.190745 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:24.190712    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95349ebe-7d57-45bf-8143-c82c14bd43fa-kube-api-access-c62hm\" (UniqueName: \"kubernetes.io/projected/95349ebe-7d57-45bf-8143-c82c14bd43fa-kube-api-access-c62hm\") pod \"95349ebe-7d57-45bf-8143-c82c14bd43fa\" (UID: \"95349ebe-7d57-45bf-8143-c82c14bd43fa\") "
Nov 02 03:41:24.190887 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:24.190819    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95349ebe-7d57-45bf-8143-c82c14bd43fa-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95349ebe-7d57-45bf-8143-c82c14bd43fa-openshift-service-ca\") pod \"95349ebe-7d57-45bf-8143-c82c14bd43fa\" (UID: \"95349ebe-7d57-45bf-8143-c82c14bd43fa\") "
Nov 02 03:41:24.191732 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:24.191701    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95349ebe-7d57-45bf-8143-c82c14bd43fa-openshift-service-ca" pod "95349ebe-7d57-45bf-8143-c82c14bd43fa" (UID: "95349ebe-7d57-45bf-8143-c82c14bd43fa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:41:24.192950 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:24.192917    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95349ebe-7d57-45bf-8143-c82c14bd43fa-maas-api-tls" pod "95349ebe-7d57-45bf-8143-c82c14bd43fa" (UID: "95349ebe-7d57-45bf-8143-c82c14bd43fa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 03:41:24.193074 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:24.193056    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95349ebe-7d57-45bf-8143-c82c14bd43fa-metrics-certs" pod "95349ebe-7d57-45bf-8143-c82c14bd43fa" (UID: "95349ebe-7d57-45bf-8143-c82c14bd43fa"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 03:41:24.193148 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:24.193091    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95349ebe-7d57-45bf-8143-c82c14bd43fa-kube-api-access-c62hm" pod "95349ebe-7d57-45bf-8143-c82c14bd43fa" (UID: "95349ebe-7d57-45bf-8143-c82c14bd43fa"). InnerVolumeSpecName "kube-api-access-c62hm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:41:24.293070 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:24.292956    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95349ebe-7d57-45bf-8143-c82c14bd43fa-maas-api-tls\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:41:24.293070 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:24.292986    4496 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95349ebe-7d57-45bf-8143-c82c14bd43fa-metrics-certs\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:41:24.293070 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:24.292997    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c62hm\" (UniqueName: \"kubernetes.io/projected/95349ebe-7d57-45bf-8143-c82c14bd43fa-kube-api-access-c62hm\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:41:24.293070 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:24.293007    4496 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95349ebe-7d57-45bf-8143-c82c14bd43fa-openshift-service-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:41:24.788762 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:24.788724    4496 generic.go:356] "Generic (PLEG): container finished" podID="95349ebe-7d57-45bf-8143-c82c14bd43fa" containerID="db0133280d738cfb5f11e6c30657fa7be80b8758c3c76fb9fa1a43a3896c0c3c" exitCode=0
Nov 02 03:41:24.789321 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:24.788788    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0a04e0-785685977-slvzx"
Nov 02 03:41:24.789321 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:24.788825    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0a04e0-785685977-slvzx" event={"ID":"95349ebe-7d57-45bf-8143-c82c14bd43fa","Type":"ContainerDied","Data":"db0133280d738cfb5f11e6c30657fa7be80b8758c3c76fb9fa1a43a3896c0c3c"}
Nov 02 03:41:24.789321 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:24.788881    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0a04e0-785685977-slvzx" event={"ID":"95349ebe-7d57-45bf-8143-c82c14bd43fa","Type":"ContainerDied","Data":"75fd863f25e17679ab40ac6659a63603e6e3ebb06eb7fc3c3878feaa8f6b0221"}
Nov 02 03:41:24.789321 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:24.788906    4496 scope.go:122] "RemoveContainer" containerID="db0133280d738cfb5f11e6c30657fa7be80b8758c3c76fb9fa1a43a3896c0c3c"
Nov 02 03:41:24.802856 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:24.802832    4496 scope.go:122] "RemoveContainer" containerID="db0133280d738cfb5f11e6c30657fa7be80b8758c3c76fb9fa1a43a3896c0c3c"
Nov 02 03:41:24.803203 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:41:24.803182    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db0133280d738cfb5f11e6c30657fa7be80b8758c3c76fb9fa1a43a3896c0c3c\": container with ID starting with db0133280d738cfb5f11e6c30657fa7be80b8758c3c76fb9fa1a43a3896c0c3c not found: ID does not exist" containerID="db0133280d738cfb5f11e6c30657fa7be80b8758c3c76fb9fa1a43a3896c0c3c"
Nov 02 03:41:24.803291 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:24.803208    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db0133280d738cfb5f11e6c30657fa7be80b8758c3c76fb9fa1a43a3896c0c3c"} err="failed to get container status \"db0133280d738cfb5f11e6c30657fa7be80b8758c3c76fb9fa1a43a3896c0c3c\": rpc error: code = NotFound desc = could not find container \"db0133280d738cfb5f11e6c30657fa7be80b8758c3c76fb9fa1a43a3896c0c3c\": container with ID starting with db0133280d738cfb5f11e6c30657fa7be80b8758c3c76fb9fa1a43a3896c0c3c not found: ID does not exist"
Nov 02 03:41:24.956181 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:24.956149    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-0a04e0-785685977-slvzx"]
Nov 02 03:41:25.006381 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:25.006353    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-0a04e0-785685977-slvzx"]
Nov 02 03:41:25.583441 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:25.583403    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95349ebe-7d57-45bf-8143-c82c14bd43fa" path="/var/lib/kubelet/pods/95349ebe-7d57-45bf-8143-c82c14bd43fa/volumes"
Nov 02 03:41:26.366121 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:26.366080    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-e44998-openshift-default-6bfcf667b5-mwgmh"]
Nov 02 03:41:26.366646 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:26.366367    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-e44998-openshift-default-6bfcf667b5-mwgmh" podUID="c01165ba-6eca-4e2a-81b1-c743acb0fbd6" containerName="istio-proxy" containerID="cri-o://f9bbcbb98fc8a0dd9df9bc6b8731d1e9fb4c9452a7fd70684ab0e3f4f4cb0f36" gracePeriod=30
Nov 02 03:41:31.650233 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:31.650157    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-e44998-openshift-default-6bfcf667b5-mwgmh"
Nov 02 03:41:31.779104 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:31.779063    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c01165ba-6eca-4e2a-81b1-c743acb0fbd6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c01165ba-6eca-4e2a-81b1-c743acb0fbd6-istio-envoy\") pod \"c01165ba-6eca-4e2a-81b1-c743acb0fbd6\" (UID: \"c01165ba-6eca-4e2a-81b1-c743acb0fbd6\") "
Nov 02 03:41:31.779304 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:31.779127    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c01165ba-6eca-4e2a-81b1-c743acb0fbd6-kube-api-access-l7pk4\" (UniqueName: \"kubernetes.io/projected/c01165ba-6eca-4e2a-81b1-c743acb0fbd6-kube-api-access-l7pk4\") pod \"c01165ba-6eca-4e2a-81b1-c743acb0fbd6\" (UID: \"c01165ba-6eca-4e2a-81b1-c743acb0fbd6\") "
Nov 02 03:41:31.779304 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:31.779155    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c01165ba-6eca-4e2a-81b1-c743acb0fbd6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c01165ba-6eca-4e2a-81b1-c743acb0fbd6-workload-certs\") pod \"c01165ba-6eca-4e2a-81b1-c743acb0fbd6\" (UID: \"c01165ba-6eca-4e2a-81b1-c743acb0fbd6\") "
Nov 02 03:41:31.779304 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:31.779187    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c01165ba-6eca-4e2a-81b1-c743acb0fbd6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c01165ba-6eca-4e2a-81b1-c743acb0fbd6-credential-socket\") pod \"c01165ba-6eca-4e2a-81b1-c743acb0fbd6\" (UID: \"c01165ba-6eca-4e2a-81b1-c743acb0fbd6\") "
Nov 02 03:41:31.779304 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:31.779236    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c01165ba-6eca-4e2a-81b1-c743acb0fbd6-istio-token\" (UniqueName: \"kubernetes.io/projected/c01165ba-6eca-4e2a-81b1-c743acb0fbd6-istio-token\") pod \"c01165ba-6eca-4e2a-81b1-c743acb0fbd6\" (UID: \"c01165ba-6eca-4e2a-81b1-c743acb0fbd6\") "
Nov 02 03:41:31.779304 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:31.779285    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c01165ba-6eca-4e2a-81b1-c743acb0fbd6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c01165ba-6eca-4e2a-81b1-c743acb0fbd6-istio-podinfo\") pod \"c01165ba-6eca-4e2a-81b1-c743acb0fbd6\" (UID: \"c01165ba-6eca-4e2a-81b1-c743acb0fbd6\") "
Nov 02 03:41:31.779618 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:31.779350    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c01165ba-6eca-4e2a-81b1-c743acb0fbd6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c01165ba-6eca-4e2a-81b1-c743acb0fbd6-workload-socket\") pod \"c01165ba-6eca-4e2a-81b1-c743acb0fbd6\" (UID: \"c01165ba-6eca-4e2a-81b1-c743acb0fbd6\") "
Nov 02 03:41:31.779618 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:31.779488    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c01165ba-6eca-4e2a-81b1-c743acb0fbd6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c01165ba-6eca-4e2a-81b1-c743acb0fbd6-istiod-ca-cert\") pod \"c01165ba-6eca-4e2a-81b1-c743acb0fbd6\" (UID: \"c01165ba-6eca-4e2a-81b1-c743acb0fbd6\") "
Nov 02 03:41:31.779618 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:31.779490    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c01165ba-6eca-4e2a-81b1-c743acb0fbd6-credential-socket" pod "c01165ba-6eca-4e2a-81b1-c743acb0fbd6" (UID: "c01165ba-6eca-4e2a-81b1-c743acb0fbd6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:41:31.779618 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:31.779539    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c01165ba-6eca-4e2a-81b1-c743acb0fbd6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c01165ba-6eca-4e2a-81b1-c743acb0fbd6-istio-data\") pod \"c01165ba-6eca-4e2a-81b1-c743acb0fbd6\" (UID: \"c01165ba-6eca-4e2a-81b1-c743acb0fbd6\") "
Nov 02 03:41:31.779842 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:31.779661    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c01165ba-6eca-4e2a-81b1-c743acb0fbd6-workload-socket" pod "c01165ba-6eca-4e2a-81b1-c743acb0fbd6" (UID: "c01165ba-6eca-4e2a-81b1-c743acb0fbd6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:41:31.779910 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:31.779890    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c01165ba-6eca-4e2a-81b1-c743acb0fbd6-istio-data" pod "c01165ba-6eca-4e2a-81b1-c743acb0fbd6" (UID: "c01165ba-6eca-4e2a-81b1-c743acb0fbd6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:41:31.779976 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:31.779954    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c01165ba-6eca-4e2a-81b1-c743acb0fbd6-istiod-ca-cert" pod "c01165ba-6eca-4e2a-81b1-c743acb0fbd6" (UID: "c01165ba-6eca-4e2a-81b1-c743acb0fbd6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:41:31.780137 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:31.780103    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c01165ba-6eca-4e2a-81b1-c743acb0fbd6-workload-certs" pod "c01165ba-6eca-4e2a-81b1-c743acb0fbd6" (UID: "c01165ba-6eca-4e2a-81b1-c743acb0fbd6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:41:31.780269 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:31.780246    4496 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c01165ba-6eca-4e2a-81b1-c743acb0fbd6-workload-certs\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:41:31.780360 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:31.780275    4496 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c01165ba-6eca-4e2a-81b1-c743acb0fbd6-credential-socket\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:41:31.780360 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:31.780290    4496 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c01165ba-6eca-4e2a-81b1-c743acb0fbd6-workload-socket\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:41:31.780360 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:31.780305    4496 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c01165ba-6eca-4e2a-81b1-c743acb0fbd6-istiod-ca-cert\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:41:31.780360 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:31.780318    4496 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c01165ba-6eca-4e2a-81b1-c743acb0fbd6-istio-data\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:41:31.781642 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:31.781610    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c01165ba-6eca-4e2a-81b1-c743acb0fbd6-kube-api-access-l7pk4" pod "c01165ba-6eca-4e2a-81b1-c743acb0fbd6" (UID: "c01165ba-6eca-4e2a-81b1-c743acb0fbd6"). InnerVolumeSpecName "kube-api-access-l7pk4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:41:31.781817 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:31.781794    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c01165ba-6eca-4e2a-81b1-c743acb0fbd6-istio-envoy" pod "c01165ba-6eca-4e2a-81b1-c743acb0fbd6" (UID: "c01165ba-6eca-4e2a-81b1-c743acb0fbd6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:41:31.781889 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:31.781813    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c01165ba-6eca-4e2a-81b1-c743acb0fbd6-istio-token" pod "c01165ba-6eca-4e2a-81b1-c743acb0fbd6" (UID: "c01165ba-6eca-4e2a-81b1-c743acb0fbd6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:41:31.782082 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:31.782057    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c01165ba-6eca-4e2a-81b1-c743acb0fbd6-istio-podinfo" pod "c01165ba-6eca-4e2a-81b1-c743acb0fbd6" (UID: "c01165ba-6eca-4e2a-81b1-c743acb0fbd6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 03:41:31.842190 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:31.842141    4496 generic.go:356] "Generic (PLEG): container finished" podID="c01165ba-6eca-4e2a-81b1-c743acb0fbd6" containerID="f9bbcbb98fc8a0dd9df9bc6b8731d1e9fb4c9452a7fd70684ab0e3f4f4cb0f36" exitCode=0
Nov 02 03:41:31.842361 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:31.842196    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-e44998-openshift-default-6bfcf667b5-mwgmh"
Nov 02 03:41:31.842361 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:31.842195    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-e44998-openshift-default-6bfcf667b5-mwgmh" event={"ID":"c01165ba-6eca-4e2a-81b1-c743acb0fbd6","Type":"ContainerDied","Data":"f9bbcbb98fc8a0dd9df9bc6b8731d1e9fb4c9452a7fd70684ab0e3f4f4cb0f36"}
Nov 02 03:41:31.842361 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:31.842259    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-e44998-openshift-default-6bfcf667b5-mwgmh" event={"ID":"c01165ba-6eca-4e2a-81b1-c743acb0fbd6","Type":"ContainerDied","Data":"413e111269e470e1c0006a4461fbe9cefc7bf7312d9c3ef63f66f440a79c8ed7"}
Nov 02 03:41:31.842361 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:31.842296    4496 scope.go:122] "RemoveContainer" containerID="f9bbcbb98fc8a0dd9df9bc6b8731d1e9fb4c9452a7fd70684ab0e3f4f4cb0f36"
Nov 02 03:41:31.855067 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:31.855047    4496 scope.go:122] "RemoveContainer" containerID="534e1c32f59035b693638a67a1c0bab60c38914d4ffe15952783556e06b7fdb4"
Nov 02 03:41:31.864885 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:31.864861    4496 scope.go:122] "RemoveContainer" containerID="f9bbcbb98fc8a0dd9df9bc6b8731d1e9fb4c9452a7fd70684ab0e3f4f4cb0f36"
Nov 02 03:41:31.865205 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:41:31.865187    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9bbcbb98fc8a0dd9df9bc6b8731d1e9fb4c9452a7fd70684ab0e3f4f4cb0f36\": container with ID starting with f9bbcbb98fc8a0dd9df9bc6b8731d1e9fb4c9452a7fd70684ab0e3f4f4cb0f36 not found: ID does not exist" containerID="f9bbcbb98fc8a0dd9df9bc6b8731d1e9fb4c9452a7fd70684ab0e3f4f4cb0f36"
Nov 02 03:41:31.865250 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:31.865213    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9bbcbb98fc8a0dd9df9bc6b8731d1e9fb4c9452a7fd70684ab0e3f4f4cb0f36"} err="failed to get container status \"f9bbcbb98fc8a0dd9df9bc6b8731d1e9fb4c9452a7fd70684ab0e3f4f4cb0f36\": rpc error: code = NotFound desc = could not find container \"f9bbcbb98fc8a0dd9df9bc6b8731d1e9fb4c9452a7fd70684ab0e3f4f4cb0f36\": container with ID starting with f9bbcbb98fc8a0dd9df9bc6b8731d1e9fb4c9452a7fd70684ab0e3f4f4cb0f36 not found: ID does not exist"
Nov 02 03:41:31.865250 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:31.865230    4496 scope.go:122] "RemoveContainer" containerID="534e1c32f59035b693638a67a1c0bab60c38914d4ffe15952783556e06b7fdb4"
Nov 02 03:41:31.865497 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:41:31.865473    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"534e1c32f59035b693638a67a1c0bab60c38914d4ffe15952783556e06b7fdb4\": container with ID starting with 534e1c32f59035b693638a67a1c0bab60c38914d4ffe15952783556e06b7fdb4 not found: ID does not exist" containerID="534e1c32f59035b693638a67a1c0bab60c38914d4ffe15952783556e06b7fdb4"
Nov 02 03:41:31.865539 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:31.865509    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"534e1c32f59035b693638a67a1c0bab60c38914d4ffe15952783556e06b7fdb4"} err="failed to get container status \"534e1c32f59035b693638a67a1c0bab60c38914d4ffe15952783556e06b7fdb4\": rpc error: code = NotFound desc = could not find container \"534e1c32f59035b693638a67a1c0bab60c38914d4ffe15952783556e06b7fdb4\": container with ID starting with 534e1c32f59035b693638a67a1c0bab60c38914d4ffe15952783556e06b7fdb4 not found: ID does not exist"
Nov 02 03:41:31.881254 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:31.881232    4496 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c01165ba-6eca-4e2a-81b1-c743acb0fbd6-istio-envoy\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:41:31.881254 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:31.881253    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7pk4\" (UniqueName: \"kubernetes.io/projected/c01165ba-6eca-4e2a-81b1-c743acb0fbd6-kube-api-access-l7pk4\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:41:31.881363 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:31.881264    4496 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c01165ba-6eca-4e2a-81b1-c743acb0fbd6-istio-token\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:41:31.881363 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:31.881273    4496 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c01165ba-6eca-4e2a-81b1-c743acb0fbd6-istio-podinfo\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:41:32.010026 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:32.009993    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-e44998-openshift-default-6bfcf667b5-mwgmh"]
Nov 02 03:41:32.013271 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:32.013248    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-e44998-openshift-default-6bfcf667b5-mwgmh"]
Nov 02 03:41:33.525970 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:41:33.525937    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 03:41:33.575913 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:33.575878    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c01165ba-6eca-4e2a-81b1-c743acb0fbd6" path="/var/lib/kubelet/pods/c01165ba-6eca-4e2a-81b1-c743acb0fbd6/volumes"
Nov 02 03:41:38.724154 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:38.724110    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-3d563c-79fbbc9545-sv8h8"]
Nov 02 03:41:38.832288 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:38.832252    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-3d563c-64d4df97cf-v5sg9"]
Nov 02 03:41:41.868474 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:41.868439    4496 scope.go:122] "RemoveContainer" containerID="ddd693c34dcbe085cff011ef95c7b4c7dcda920477fb97210bd7342d9e182588"
Nov 02 03:41:41.878752 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:41.878723    4496 scope.go:122] "RemoveContainer" containerID="dabf6b33ba89435c774afa3c9306bc2b0e7a40870c3f3a8f417b702b44160091"
Nov 02 03:41:43.725122 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:43.725081    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-3d563c-79fbbc9545-sv8h8" podUID="319aacd9-1b90-48cf-8b87-04561e177fe5" containerName="payload-processing" containerID="cri-o://82d601d0f4ee56e0be2b9341923b80946f68549c6214bd4380dfc105fb741a97" gracePeriod=25
Nov 02 03:41:43.832641 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:43.832601    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3d563c-64d4df97cf-v5sg9" podUID="9afd306f-bfae-4fe0-b61f-317e997876a0" containerName="payload-pre-processing" containerID="cri-o://38df86d12c20044180ff5932ff08edfa6b81d2288e1cdacd9a29fb4e09a07b5d" gracePeriod=25
Nov 02 03:41:43.941618 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:43.941589    4496 generic.go:356] "Generic (PLEG): container finished" podID="319aacd9-1b90-48cf-8b87-04561e177fe5" containerID="82d601d0f4ee56e0be2b9341923b80946f68549c6214bd4380dfc105fb741a97" exitCode=0
Nov 02 03:41:43.941778 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:43.941656    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-3d563c-79fbbc9545-sv8h8" event={"ID":"319aacd9-1b90-48cf-8b87-04561e177fe5","Type":"ContainerDied","Data":"82d601d0f4ee56e0be2b9341923b80946f68549c6214bd4380dfc105fb741a97"}
Nov 02 03:41:43.993116 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:43.993093    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-3d563c-79fbbc9545-sv8h8"
Nov 02 03:41:44.020793 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:44.020756    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/319aacd9-1b90-48cf-8b87-04561e177fe5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/319aacd9-1b90-48cf-8b87-04561e177fe5-plugins-config-volume\") pod \"319aacd9-1b90-48cf-8b87-04561e177fe5\" (UID: \"319aacd9-1b90-48cf-8b87-04561e177fe5\") "
Nov 02 03:41:44.021033 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:44.020851    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/319aacd9-1b90-48cf-8b87-04561e177fe5-kube-api-access-lg2qc\" (UniqueName: \"kubernetes.io/projected/319aacd9-1b90-48cf-8b87-04561e177fe5-kube-api-access-lg2qc\") pod \"319aacd9-1b90-48cf-8b87-04561e177fe5\" (UID: \"319aacd9-1b90-48cf-8b87-04561e177fe5\") "
Nov 02 03:41:44.021257 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:44.021231    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/319aacd9-1b90-48cf-8b87-04561e177fe5-plugins-config-volume" pod "319aacd9-1b90-48cf-8b87-04561e177fe5" (UID: "319aacd9-1b90-48cf-8b87-04561e177fe5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:41:44.021382 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:44.021365    4496 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/319aacd9-1b90-48cf-8b87-04561e177fe5-plugins-config-volume\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:41:44.024939 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:44.024908    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/319aacd9-1b90-48cf-8b87-04561e177fe5-kube-api-access-lg2qc" pod "319aacd9-1b90-48cf-8b87-04561e177fe5" (UID: "319aacd9-1b90-48cf-8b87-04561e177fe5"). InnerVolumeSpecName "kube-api-access-lg2qc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:41:44.077437 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:44.077413    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3d563c-64d4df97cf-v5sg9"
Nov 02 03:41:44.121852 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:44.121821    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9afd306f-bfae-4fe0-b61f-317e997876a0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9afd306f-bfae-4fe0-b61f-317e997876a0-plugins-config-volume\") pod \"9afd306f-bfae-4fe0-b61f-317e997876a0\" (UID: \"9afd306f-bfae-4fe0-b61f-317e997876a0\") "
Nov 02 03:41:44.122075 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:44.122059    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9afd306f-bfae-4fe0-b61f-317e997876a0-kube-api-access-4jdzn\" (UniqueName: \"kubernetes.io/projected/9afd306f-bfae-4fe0-b61f-317e997876a0-kube-api-access-4jdzn\") pod \"9afd306f-bfae-4fe0-b61f-317e997876a0\" (UID: \"9afd306f-bfae-4fe0-b61f-317e997876a0\") "
Nov 02 03:41:44.122305 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:44.122279    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9afd306f-bfae-4fe0-b61f-317e997876a0-plugins-config-volume" pod "9afd306f-bfae-4fe0-b61f-317e997876a0" (UID: "9afd306f-bfae-4fe0-b61f-317e997876a0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:41:44.122475 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:44.122450    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lg2qc\" (UniqueName: \"kubernetes.io/projected/319aacd9-1b90-48cf-8b87-04561e177fe5-kube-api-access-lg2qc\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:41:44.122585 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:44.122478    4496 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9afd306f-bfae-4fe0-b61f-317e997876a0-plugins-config-volume\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:41:44.124210 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:44.124186    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9afd306f-bfae-4fe0-b61f-317e997876a0-kube-api-access-4jdzn" pod "9afd306f-bfae-4fe0-b61f-317e997876a0" (UID: "9afd306f-bfae-4fe0-b61f-317e997876a0"). InnerVolumeSpecName "kube-api-access-4jdzn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:41:44.223617 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:44.223578    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jdzn\" (UniqueName: \"kubernetes.io/projected/9afd306f-bfae-4fe0-b61f-317e997876a0-kube-api-access-4jdzn\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:41:44.808347 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:44.808308    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-cb56e9-545bcc7cc7-85plw"]
Nov 02 03:41:44.918060 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:44.918007    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-cb56e9-d444d6cbc-vvc97"]
Nov 02 03:41:44.949542 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:44.949509    4496 generic.go:356] "Generic (PLEG): container finished" podID="9afd306f-bfae-4fe0-b61f-317e997876a0" containerID="38df86d12c20044180ff5932ff08edfa6b81d2288e1cdacd9a29fb4e09a07b5d" exitCode=0
Nov 02 03:41:44.949746 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:44.949565    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3d563c-64d4df97cf-v5sg9"
Nov 02 03:41:44.949746 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:44.949589    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3d563c-64d4df97cf-v5sg9" event={"ID":"9afd306f-bfae-4fe0-b61f-317e997876a0","Type":"ContainerDied","Data":"38df86d12c20044180ff5932ff08edfa6b81d2288e1cdacd9a29fb4e09a07b5d"}
Nov 02 03:41:44.949746 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:44.949632    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3d563c-64d4df97cf-v5sg9" event={"ID":"9afd306f-bfae-4fe0-b61f-317e997876a0","Type":"ContainerDied","Data":"30fb2d0de74dcb53d29f5f5c91039379a51e18a3000199fcadfbd0dac2f872bf"}
Nov 02 03:41:44.949746 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:44.949653    4496 scope.go:122] "RemoveContainer" containerID="38df86d12c20044180ff5932ff08edfa6b81d2288e1cdacd9a29fb4e09a07b5d"
Nov 02 03:41:44.951088 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:44.951068    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-3d563c-79fbbc9545-sv8h8" event={"ID":"319aacd9-1b90-48cf-8b87-04561e177fe5","Type":"ContainerDied","Data":"08b3c406d71bab99d2e7ac2c6beb97d9b0b0ea81a4e1f64d96be9336e7346004"}
Nov 02 03:41:44.951189 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:44.951074    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-3d563c-79fbbc9545-sv8h8"
Nov 02 03:41:44.961773 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:44.961752    4496 scope.go:122] "RemoveContainer" containerID="38df86d12c20044180ff5932ff08edfa6b81d2288e1cdacd9a29fb4e09a07b5d"
Nov 02 03:41:44.962111 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:41:44.962089    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38df86d12c20044180ff5932ff08edfa6b81d2288e1cdacd9a29fb4e09a07b5d\": container with ID starting with 38df86d12c20044180ff5932ff08edfa6b81d2288e1cdacd9a29fb4e09a07b5d not found: ID does not exist" containerID="38df86d12c20044180ff5932ff08edfa6b81d2288e1cdacd9a29fb4e09a07b5d"
Nov 02 03:41:44.962171 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:44.962119    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38df86d12c20044180ff5932ff08edfa6b81d2288e1cdacd9a29fb4e09a07b5d"} err="failed to get container status \"38df86d12c20044180ff5932ff08edfa6b81d2288e1cdacd9a29fb4e09a07b5d\": rpc error: code = NotFound desc = could not find container \"38df86d12c20044180ff5932ff08edfa6b81d2288e1cdacd9a29fb4e09a07b5d\": container with ID starting with 38df86d12c20044180ff5932ff08edfa6b81d2288e1cdacd9a29fb4e09a07b5d not found: ID does not exist"
Nov 02 03:41:44.962171 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:44.962134    4496 scope.go:122] "RemoveContainer" containerID="82d601d0f4ee56e0be2b9341923b80946f68549c6214bd4380dfc105fb741a97"
Nov 02 03:41:45.158093 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:45.157985    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-3d563c-79fbbc9545-sv8h8"]
Nov 02 03:41:45.161309 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:45.161283    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-3d563c-79fbbc9545-sv8h8"]
Nov 02 03:41:45.184875 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:45.184853    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-cb56e9-64bd7c87c6-wcgp6"]
Nov 02 03:41:45.185103 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:45.185080    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cb56e9-64bd7c87c6-wcgp6" podUID="68f8442f-e148-4090-a92c-af97a033cb47" containerName="maas-api" containerID="cri-o://f50c364669055fe56a0bfac3c719de642f0bde7bbe511fb3c5377a6a7bb96019" gracePeriod=30
Nov 02 03:41:45.274611 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:45.274569    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-3d563c-64d4df97cf-v5sg9"]
Nov 02 03:41:45.280298 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:45.280272    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-3d563c-64d4df97cf-v5sg9"]
Nov 02 03:41:45.463451 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:45.463421    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cb56e9-64bd7c87c6-wcgp6"
Nov 02 03:41:45.536571 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:45.536540    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68f8442f-e148-4090-a92c-af97a033cb47-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68f8442f-e148-4090-a92c-af97a033cb47-openshift-service-ca\") pod \"68f8442f-e148-4090-a92c-af97a033cb47\" (UID: \"68f8442f-e148-4090-a92c-af97a033cb47\") "
Nov 02 03:41:45.536738 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:45.536593    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68f8442f-e148-4090-a92c-af97a033cb47-metrics-certs\" (UniqueName: \"kubernetes.io/secret/68f8442f-e148-4090-a92c-af97a033cb47-metrics-certs\") pod \"68f8442f-e148-4090-a92c-af97a033cb47\" (UID: \"68f8442f-e148-4090-a92c-af97a033cb47\") "
Nov 02 03:41:45.536738 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:45.536729    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68f8442f-e148-4090-a92c-af97a033cb47-kube-api-access-xmwlf\" (UniqueName: \"kubernetes.io/projected/68f8442f-e148-4090-a92c-af97a033cb47-kube-api-access-xmwlf\") pod \"68f8442f-e148-4090-a92c-af97a033cb47\" (UID: \"68f8442f-e148-4090-a92c-af97a033cb47\") "
Nov 02 03:41:45.536865 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:45.536757    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68f8442f-e148-4090-a92c-af97a033cb47-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68f8442f-e148-4090-a92c-af97a033cb47-maas-api-tls\") pod \"68f8442f-e148-4090-a92c-af97a033cb47\" (UID: \"68f8442f-e148-4090-a92c-af97a033cb47\") "
Nov 02 03:41:45.537042 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:45.536957    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68f8442f-e148-4090-a92c-af97a033cb47-openshift-service-ca" pod "68f8442f-e148-4090-a92c-af97a033cb47" (UID: "68f8442f-e148-4090-a92c-af97a033cb47"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:41:45.537260 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:45.537241    4496 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68f8442f-e148-4090-a92c-af97a033cb47-openshift-service-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:41:45.539124 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:45.539093    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68f8442f-e148-4090-a92c-af97a033cb47-kube-api-access-xmwlf" pod "68f8442f-e148-4090-a92c-af97a033cb47" (UID: "68f8442f-e148-4090-a92c-af97a033cb47"). InnerVolumeSpecName "kube-api-access-xmwlf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:41:45.539236 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:45.539152    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68f8442f-e148-4090-a92c-af97a033cb47-metrics-certs" pod "68f8442f-e148-4090-a92c-af97a033cb47" (UID: "68f8442f-e148-4090-a92c-af97a033cb47"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 03:41:45.539311 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:45.539291    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68f8442f-e148-4090-a92c-af97a033cb47-maas-api-tls" pod "68f8442f-e148-4090-a92c-af97a033cb47" (UID: "68f8442f-e148-4090-a92c-af97a033cb47"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 03:41:45.577147 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:45.577115    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="319aacd9-1b90-48cf-8b87-04561e177fe5" path="/var/lib/kubelet/pods/319aacd9-1b90-48cf-8b87-04561e177fe5/volumes"
Nov 02 03:41:45.577711 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:45.577684    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9afd306f-bfae-4fe0-b61f-317e997876a0" path="/var/lib/kubelet/pods/9afd306f-bfae-4fe0-b61f-317e997876a0/volumes"
Nov 02 03:41:45.638252 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:45.638215    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmwlf\" (UniqueName: \"kubernetes.io/projected/68f8442f-e148-4090-a92c-af97a033cb47-kube-api-access-xmwlf\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:41:45.638252 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:45.638256    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68f8442f-e148-4090-a92c-af97a033cb47-maas-api-tls\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:41:45.638456 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:45.638362    4496 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68f8442f-e148-4090-a92c-af97a033cb47-metrics-certs\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:41:45.962539 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:45.962501    4496 generic.go:356] "Generic (PLEG): container finished" podID="68f8442f-e148-4090-a92c-af97a033cb47" containerID="f50c364669055fe56a0bfac3c719de642f0bde7bbe511fb3c5377a6a7bb96019" exitCode=0
Nov 02 03:41:45.962999 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:45.962541    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cb56e9-64bd7c87c6-wcgp6"
Nov 02 03:41:45.962999 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:45.962587    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cb56e9-64bd7c87c6-wcgp6" event={"ID":"68f8442f-e148-4090-a92c-af97a033cb47","Type":"ContainerDied","Data":"f50c364669055fe56a0bfac3c719de642f0bde7bbe511fb3c5377a6a7bb96019"}
Nov 02 03:41:45.962999 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:45.962640    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cb56e9-64bd7c87c6-wcgp6" event={"ID":"68f8442f-e148-4090-a92c-af97a033cb47","Type":"ContainerDied","Data":"c8a08555274e328511473d3d2fd5063a50b6590bd8b129f9bc6bdacfc17fdb39"}
Nov 02 03:41:45.962999 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:45.962662    4496 scope.go:122] "RemoveContainer" containerID="f50c364669055fe56a0bfac3c719de642f0bde7bbe511fb3c5377a6a7bb96019"
Nov 02 03:41:45.974853 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:45.974824    4496 scope.go:122] "RemoveContainer" containerID="f50c364669055fe56a0bfac3c719de642f0bde7bbe511fb3c5377a6a7bb96019"
Nov 02 03:41:45.975169 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:41:45.975150    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f50c364669055fe56a0bfac3c719de642f0bde7bbe511fb3c5377a6a7bb96019\": container with ID starting with f50c364669055fe56a0bfac3c719de642f0bde7bbe511fb3c5377a6a7bb96019 not found: ID does not exist" containerID="f50c364669055fe56a0bfac3c719de642f0bde7bbe511fb3c5377a6a7bb96019"
Nov 02 03:41:45.975217 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:45.975177    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f50c364669055fe56a0bfac3c719de642f0bde7bbe511fb3c5377a6a7bb96019"} err="failed to get container status \"f50c364669055fe56a0bfac3c719de642f0bde7bbe511fb3c5377a6a7bb96019\": rpc error: code = NotFound desc = could not find container \"f50c364669055fe56a0bfac3c719de642f0bde7bbe511fb3c5377a6a7bb96019\": container with ID starting with f50c364669055fe56a0bfac3c719de642f0bde7bbe511fb3c5377a6a7bb96019 not found: ID does not exist"
Nov 02 03:41:46.130240 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:46.130205    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-cb56e9-64bd7c87c6-wcgp6"]
Nov 02 03:41:46.137764 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:46.137728    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-cb56e9-64bd7c87c6-wcgp6"]
Nov 02 03:41:46.790350 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:46.790311    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-ad96f4-5487969c88-xt82w"]
Nov 02 03:41:46.791056 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:46.791035    4496 cpu_manager.go:414] "containerMap: removing container" podUID="319aacd9-1b90-48cf-8b87-04561e177fe5" containerName="payload-processing"
Nov 02 03:41:46.791056 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:46.791056    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="319aacd9-1b90-48cf-8b87-04561e177fe5" containerName="payload-processing"
Nov 02 03:41:46.791229 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:46.791086    4496 cpu_manager.go:414] "containerMap: removing container" podUID="c01165ba-6eca-4e2a-81b1-c743acb0fbd6" containerName="istio-proxy"
Nov 02 03:41:46.791229 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:46.791094    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c01165ba-6eca-4e2a-81b1-c743acb0fbd6" containerName="istio-proxy"
Nov 02 03:41:46.791229 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:46.791117    4496 cpu_manager.go:414] "containerMap: removing container" podUID="95349ebe-7d57-45bf-8143-c82c14bd43fa" containerName="maas-api"
Nov 02 03:41:46.791229 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:46.791125    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95349ebe-7d57-45bf-8143-c82c14bd43fa" containerName="maas-api"
Nov 02 03:41:46.791229 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:46.791138    4496 cpu_manager.go:414] "containerMap: removing container" podUID="68f8442f-e148-4090-a92c-af97a033cb47" containerName="maas-api"
Nov 02 03:41:46.791229 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:46.791145    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68f8442f-e148-4090-a92c-af97a033cb47" containerName="maas-api"
Nov 02 03:41:46.791229 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:46.791158    4496 cpu_manager.go:414] "containerMap: removing container" podUID="9afd306f-bfae-4fe0-b61f-317e997876a0" containerName="payload-pre-processing"
Nov 02 03:41:46.791229 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:46.791166    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9afd306f-bfae-4fe0-b61f-317e997876a0" containerName="payload-pre-processing"
Nov 02 03:41:46.791229 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:46.791191    4496 cpu_manager.go:414] "containerMap: removing container" podUID="c01165ba-6eca-4e2a-81b1-c743acb0fbd6" containerName="istio-proxy"
Nov 02 03:41:46.791229 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:46.791198    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c01165ba-6eca-4e2a-81b1-c743acb0fbd6" containerName="istio-proxy"
Nov 02 03:41:46.791717 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:46.791359    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="68f8442f-e148-4090-a92c-af97a033cb47" containerName="maas-api"
Nov 02 03:41:46.791717 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:46.791372    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="9afd306f-bfae-4fe0-b61f-317e997876a0" containerName="payload-pre-processing"
Nov 02 03:41:46.791717 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:46.791387    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="c01165ba-6eca-4e2a-81b1-c743acb0fbd6" containerName="istio-proxy"
Nov 02 03:41:46.791717 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:46.791403    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="319aacd9-1b90-48cf-8b87-04561e177fe5" containerName="payload-processing"
Nov 02 03:41:46.791717 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:46.791418    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="95349ebe-7d57-45bf-8143-c82c14bd43fa" containerName="maas-api"
Nov 02 03:41:46.796425 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:46.796394    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ad96f4-5487969c88-xt82w"
Nov 02 03:41:46.849220 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:46.849189    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-ad96f4-serving-cert\""
Nov 02 03:41:46.853708 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:46.853684    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b335eb4b-7295-45f3-9dd4-637e3c7c9c58-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-ad96f4-5487969c88-xt82w\" (UID: \"b335eb4b-7295-45f3-9dd4-637e3c7c9c58\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ad96f4-5487969c88-xt82w"
Nov 02 03:41:46.853821 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:46.853787    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2zx8\" (UniqueName: \"kubernetes.io/projected/b335eb4b-7295-45f3-9dd4-637e3c7c9c58-kube-api-access-q2zx8\") pod \"maas-api-e2e-praxis-b-ad96f4-5487969c88-xt82w\" (UID: \"b335eb4b-7295-45f3-9dd4-637e3c7c9c58\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ad96f4-5487969c88-xt82w"
Nov 02 03:41:46.853877 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:46.853823    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b335eb4b-7295-45f3-9dd4-637e3c7c9c58-metrics-certs\") pod \"maas-api-e2e-praxis-b-ad96f4-5487969c88-xt82w\" (UID: \"b335eb4b-7295-45f3-9dd4-637e3c7c9c58\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ad96f4-5487969c88-xt82w"
Nov 02 03:41:46.853877 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:46.853842    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b335eb4b-7295-45f3-9dd4-637e3c7c9c58-maas-api-tls\") pod \"maas-api-e2e-praxis-b-ad96f4-5487969c88-xt82w\" (UID: \"b335eb4b-7295-45f3-9dd4-637e3c7c9c58\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ad96f4-5487969c88-xt82w"
Nov 02 03:41:46.865843 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:46.865813    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-ad96f4-5487969c88-xt82w"]
Nov 02 03:41:46.955299 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:46.955269    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b335eb4b-7295-45f3-9dd4-637e3c7c9c58-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-ad96f4-5487969c88-xt82w\" (UID: \"b335eb4b-7295-45f3-9dd4-637e3c7c9c58\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ad96f4-5487969c88-xt82w"
Nov 02 03:41:46.955513 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:46.955378    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2zx8\" (UniqueName: \"kubernetes.io/projected/b335eb4b-7295-45f3-9dd4-637e3c7c9c58-kube-api-access-q2zx8\") pod \"maas-api-e2e-praxis-b-ad96f4-5487969c88-xt82w\" (UID: \"b335eb4b-7295-45f3-9dd4-637e3c7c9c58\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ad96f4-5487969c88-xt82w"
Nov 02 03:41:46.955513 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:46.955419    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b335eb4b-7295-45f3-9dd4-637e3c7c9c58-metrics-certs\") pod \"maas-api-e2e-praxis-b-ad96f4-5487969c88-xt82w\" (UID: \"b335eb4b-7295-45f3-9dd4-637e3c7c9c58\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ad96f4-5487969c88-xt82w"
Nov 02 03:41:46.955513 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:46.955443    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b335eb4b-7295-45f3-9dd4-637e3c7c9c58-maas-api-tls\") pod \"maas-api-e2e-praxis-b-ad96f4-5487969c88-xt82w\" (UID: \"b335eb4b-7295-45f3-9dd4-637e3c7c9c58\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ad96f4-5487969c88-xt82w"
Nov 02 03:41:46.955696 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:41:46.955604    4496 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-ad96f4-serving-cert: secret "maas-api-e2e-praxis-b-ad96f4-serving-cert" not found
Nov 02 03:41:46.955696 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:41:46.955676    4496 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b335eb4b-7295-45f3-9dd4-637e3c7c9c58-maas-api-tls podName:b335eb4b-7295-45f3-9dd4-637e3c7c9c58 nodeName:}" failed. No retries permitted until 2026-11-02 03:41:47.45565577 +0000 UTC m=+3248.034263143 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b335eb4b-7295-45f3-9dd4-637e3c7c9c58-maas-api-tls") pod "maas-api-e2e-praxis-b-ad96f4-5487969c88-xt82w" (UID: "b335eb4b-7295-45f3-9dd4-637e3c7c9c58") : secret "maas-api-e2e-praxis-b-ad96f4-serving-cert" not found
Nov 02 03:41:46.956141 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:46.956118    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b335eb4b-7295-45f3-9dd4-637e3c7c9c58-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-ad96f4-5487969c88-xt82w\" (UID: \"b335eb4b-7295-45f3-9dd4-637e3c7c9c58\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ad96f4-5487969c88-xt82w"
Nov 02 03:41:46.958125 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:46.958094    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b335eb4b-7295-45f3-9dd4-637e3c7c9c58-metrics-certs\") pod \"maas-api-e2e-praxis-b-ad96f4-5487969c88-xt82w\" (UID: \"b335eb4b-7295-45f3-9dd4-637e3c7c9c58\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ad96f4-5487969c88-xt82w"
Nov 02 03:41:47.013867 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:47.013840    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2zx8\" (UniqueName: \"kubernetes.io/projected/b335eb4b-7295-45f3-9dd4-637e3c7c9c58-kube-api-access-q2zx8\") pod \"maas-api-e2e-praxis-b-ad96f4-5487969c88-xt82w\" (UID: \"b335eb4b-7295-45f3-9dd4-637e3c7c9c58\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ad96f4-5487969c88-xt82w"
Nov 02 03:41:47.462532 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:47.462497    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b335eb4b-7295-45f3-9dd4-637e3c7c9c58-maas-api-tls\") pod \"maas-api-e2e-praxis-b-ad96f4-5487969c88-xt82w\" (UID: \"b335eb4b-7295-45f3-9dd4-637e3c7c9c58\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ad96f4-5487969c88-xt82w"
Nov 02 03:41:47.464884 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:47.464866    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b335eb4b-7295-45f3-9dd4-637e3c7c9c58-maas-api-tls\") pod \"maas-api-e2e-praxis-b-ad96f4-5487969c88-xt82w\" (UID: \"b335eb4b-7295-45f3-9dd4-637e3c7c9c58\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ad96f4-5487969c88-xt82w"
Nov 02 03:41:47.574458 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:47.574428    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68f8442f-e148-4090-a92c-af97a033cb47" path="/var/lib/kubelet/pods/68f8442f-e148-4090-a92c-af97a033cb47/volumes"
Nov 02 03:41:47.712828 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:47.712753    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ad96f4-5487969c88-xt82w"
Nov 02 03:41:48.095236 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:48.095203    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-ad96f4-5487969c88-xt82w"]
Nov 02 03:41:48.095721 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:41:48.095446    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb335eb4b_7295_45f3_9dd4_637e3c7c9c58.slice/crio-49cf61ee1bbf096e3d857df03d94ac613332232769c15b38aaa1c1fef7fa2c68 WatchSource:0}: Error finding container 49cf61ee1bbf096e3d857df03d94ac613332232769c15b38aaa1c1fef7fa2c68: Status 404 returned error can't find the container with id 49cf61ee1bbf096e3d857df03d94ac613332232769c15b38aaa1c1fef7fa2c68
Nov 02 03:41:48.990761 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:48.990725    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ad96f4-5487969c88-xt82w" event={"ID":"b335eb4b-7295-45f3-9dd4-637e3c7c9c58","Type":"ContainerStarted","Data":"49cf61ee1bbf096e3d857df03d94ac613332232769c15b38aaa1c1fef7fa2c68"}
Nov 02 03:41:49.808982 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:49.808937    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-cb56e9-545bcc7cc7-85plw" podUID="1975c9eb-9a68-4d15-9515-8a03d8e63fdd" containerName="payload-processing" containerID="cri-o://0489f9db4d35bdc8e040da7a42fae9e618cbd1ecb09ce29396c868f78188bfa3" gracePeriod=25
Nov 02 03:41:49.919454 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:49.919415    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cb56e9-d444d6cbc-vvc97" podUID="cf4aaa9a-1039-4f41-ad04-2ef9c946fe4f" containerName="payload-pre-processing" containerID="cri-o://54a286e31d5f8304e4a572bbaac5bd993bf22affe04d6f42ff1b55794a1d0ef3" gracePeriod=25
Nov 02 03:41:50.008479 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:50.008442    4496 generic.go:356] "Generic (PLEG): container finished" podID="1975c9eb-9a68-4d15-9515-8a03d8e63fdd" containerID="0489f9db4d35bdc8e040da7a42fae9e618cbd1ecb09ce29396c868f78188bfa3" exitCode=0
Nov 02 03:41:50.008645 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:50.008511    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-cb56e9-545bcc7cc7-85plw" event={"ID":"1975c9eb-9a68-4d15-9515-8a03d8e63fdd","Type":"ContainerDied","Data":"0489f9db4d35bdc8e040da7a42fae9e618cbd1ecb09ce29396c868f78188bfa3"}
Nov 02 03:41:50.011093 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:50.010731    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ad96f4-5487969c88-xt82w" event={"ID":"b335eb4b-7295-45f3-9dd4-637e3c7c9c58","Type":"ContainerStarted","Data":"75ebd9cdace4ab091697af66db848dc3813d027ce4655d706658839f2406d4b2"}
Nov 02 03:41:50.011093 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:50.010901    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ad96f4-5487969c88-xt82w"
Nov 02 03:41:50.080977 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:50.080944    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-cb56e9-545bcc7cc7-85plw"
Nov 02 03:41:50.127728 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:50.126572    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ad96f4-5487969c88-xt82w" podStartSLOduration=3.197853156 podStartE2EDuration="4.126552897s" totalImagesPullingTime="928.699741ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:41:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:41:50.125398244 +0000 UTC m=+3250.704005625" watchObservedRunningTime="2026-11-02 03:41:50.126552897 +0000 UTC m=+3250.705160278"
Nov 02 03:41:50.179176 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:50.179148    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cb56e9-d444d6cbc-vvc97"
Nov 02 03:41:50.192035 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:50.191985    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1975c9eb-9a68-4d15-9515-8a03d8e63fdd-kube-api-access-8mwgs\" (UniqueName: \"kubernetes.io/projected/1975c9eb-9a68-4d15-9515-8a03d8e63fdd-kube-api-access-8mwgs\") pod \"1975c9eb-9a68-4d15-9515-8a03d8e63fdd\" (UID: \"1975c9eb-9a68-4d15-9515-8a03d8e63fdd\") "
Nov 02 03:41:50.192179 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:50.192139    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1975c9eb-9a68-4d15-9515-8a03d8e63fdd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1975c9eb-9a68-4d15-9515-8a03d8e63fdd-plugins-config-volume\") pod \"1975c9eb-9a68-4d15-9515-8a03d8e63fdd\" (UID: \"1975c9eb-9a68-4d15-9515-8a03d8e63fdd\") "
Nov 02 03:41:50.192484 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:50.192457    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1975c9eb-9a68-4d15-9515-8a03d8e63fdd-plugins-config-volume" pod "1975c9eb-9a68-4d15-9515-8a03d8e63fdd" (UID: "1975c9eb-9a68-4d15-9515-8a03d8e63fdd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:41:50.194053 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:50.194030    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1975c9eb-9a68-4d15-9515-8a03d8e63fdd-kube-api-access-8mwgs" pod "1975c9eb-9a68-4d15-9515-8a03d8e63fdd" (UID: "1975c9eb-9a68-4d15-9515-8a03d8e63fdd"). InnerVolumeSpecName "kube-api-access-8mwgs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:41:50.293115 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:50.293080    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf4aaa9a-1039-4f41-ad04-2ef9c946fe4f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf4aaa9a-1039-4f41-ad04-2ef9c946fe4f-plugins-config-volume\") pod \"cf4aaa9a-1039-4f41-ad04-2ef9c946fe4f\" (UID: \"cf4aaa9a-1039-4f41-ad04-2ef9c946fe4f\") "
Nov 02 03:41:50.293300 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:50.293156    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf4aaa9a-1039-4f41-ad04-2ef9c946fe4f-kube-api-access-lkg44\" (UniqueName: \"kubernetes.io/projected/cf4aaa9a-1039-4f41-ad04-2ef9c946fe4f-kube-api-access-lkg44\") pod \"cf4aaa9a-1039-4f41-ad04-2ef9c946fe4f\" (UID: \"cf4aaa9a-1039-4f41-ad04-2ef9c946fe4f\") "
Nov 02 03:41:50.293502 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:50.293484    4496 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1975c9eb-9a68-4d15-9515-8a03d8e63fdd-plugins-config-volume\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:41:50.293594 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:50.293503    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mwgs\" (UniqueName: \"kubernetes.io/projected/1975c9eb-9a68-4d15-9515-8a03d8e63fdd-kube-api-access-8mwgs\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:41:50.293594 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:50.293524    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf4aaa9a-1039-4f41-ad04-2ef9c946fe4f-plugins-config-volume" pod "cf4aaa9a-1039-4f41-ad04-2ef9c946fe4f" (UID: "cf4aaa9a-1039-4f41-ad04-2ef9c946fe4f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:41:50.295287 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:50.295268    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf4aaa9a-1039-4f41-ad04-2ef9c946fe4f-kube-api-access-lkg44" pod "cf4aaa9a-1039-4f41-ad04-2ef9c946fe4f" (UID: "cf4aaa9a-1039-4f41-ad04-2ef9c946fe4f"). InnerVolumeSpecName "kube-api-access-lkg44". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:41:50.394700 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:50.394615    4496 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf4aaa9a-1039-4f41-ad04-2ef9c946fe4f-plugins-config-volume\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:41:50.394700 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:50.394658    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkg44\" (UniqueName: \"kubernetes.io/projected/cf4aaa9a-1039-4f41-ad04-2ef9c946fe4f-kube-api-access-lkg44\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:41:51.019391 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:51.019354    4496 generic.go:356] "Generic (PLEG): container finished" podID="cf4aaa9a-1039-4f41-ad04-2ef9c946fe4f" containerID="54a286e31d5f8304e4a572bbaac5bd993bf22affe04d6f42ff1b55794a1d0ef3" exitCode=0
Nov 02 03:41:51.019874 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:51.019403    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cb56e9-d444d6cbc-vvc97"
Nov 02 03:41:51.019874 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:51.019432    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cb56e9-d444d6cbc-vvc97" event={"ID":"cf4aaa9a-1039-4f41-ad04-2ef9c946fe4f","Type":"ContainerDied","Data":"54a286e31d5f8304e4a572bbaac5bd993bf22affe04d6f42ff1b55794a1d0ef3"}
Nov 02 03:41:51.019874 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:51.019485    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cb56e9-d444d6cbc-vvc97" event={"ID":"cf4aaa9a-1039-4f41-ad04-2ef9c946fe4f","Type":"ContainerDied","Data":"1cc9a3a257d2523ebc4321c87607e859353fba89915dab69aed708be7d95a0ac"}
Nov 02 03:41:51.019874 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:51.019509    4496 scope.go:122] "RemoveContainer" containerID="54a286e31d5f8304e4a572bbaac5bd993bf22affe04d6f42ff1b55794a1d0ef3"
Nov 02 03:41:51.021081 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:51.021057    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-cb56e9-545bcc7cc7-85plw" event={"ID":"1975c9eb-9a68-4d15-9515-8a03d8e63fdd","Type":"ContainerDied","Data":"1d347c379b21a201a66e57568c3001d2e98cde8533e746dd46a756380c24259f"}
Nov 02 03:41:51.021199 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:51.021065    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-cb56e9-545bcc7cc7-85plw"
Nov 02 03:41:51.030566 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:51.030462    4496 scope.go:122] "RemoveContainer" containerID="54a286e31d5f8304e4a572bbaac5bd993bf22affe04d6f42ff1b55794a1d0ef3"
Nov 02 03:41:51.030781 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:41:51.030752    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54a286e31d5f8304e4a572bbaac5bd993bf22affe04d6f42ff1b55794a1d0ef3\": container with ID starting with 54a286e31d5f8304e4a572bbaac5bd993bf22affe04d6f42ff1b55794a1d0ef3 not found: ID does not exist" containerID="54a286e31d5f8304e4a572bbaac5bd993bf22affe04d6f42ff1b55794a1d0ef3"
Nov 02 03:41:51.030884 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:51.030789    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54a286e31d5f8304e4a572bbaac5bd993bf22affe04d6f42ff1b55794a1d0ef3"} err="failed to get container status \"54a286e31d5f8304e4a572bbaac5bd993bf22affe04d6f42ff1b55794a1d0ef3\": rpc error: code = NotFound desc = could not find container \"54a286e31d5f8304e4a572bbaac5bd993bf22affe04d6f42ff1b55794a1d0ef3\": container with ID starting with 54a286e31d5f8304e4a572bbaac5bd993bf22affe04d6f42ff1b55794a1d0ef3 not found: ID does not exist"
Nov 02 03:41:51.030884 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:51.030810    4496 scope.go:122] "RemoveContainer" containerID="0489f9db4d35bdc8e040da7a42fae9e618cbd1ecb09ce29396c868f78188bfa3"
Nov 02 03:41:51.189894 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:51.189801    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-cb56e9-545bcc7cc7-85plw"]
Nov 02 03:41:51.237232 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:51.237198    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-cb56e9-545bcc7cc7-85plw"]
Nov 02 03:41:51.309359 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:51.309323    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-cb56e9-d444d6cbc-vvc97"]
Nov 02 03:41:51.315505 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:51.315480    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-cb56e9-d444d6cbc-vvc97"]
Nov 02 03:41:51.574382 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:51.574348    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1975c9eb-9a68-4d15-9515-8a03d8e63fdd" path="/var/lib/kubelet/pods/1975c9eb-9a68-4d15-9515-8a03d8e63fdd/volumes"
Nov 02 03:41:51.574796 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:51.574780    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf4aaa9a-1039-4f41-ad04-2ef9c946fe4f" path="/var/lib/kubelet/pods/cf4aaa9a-1039-4f41-ad04-2ef9c946fe4f/volumes"
Nov 02 03:41:56.026953 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:41:56.026926    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ad96f4-5487969c88-xt82w"
Nov 02 03:42:03.527296 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:42:03.527255    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 03:42:25.592797 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:42:25.592712    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-f68c7e-5c59f9756f-mqvxw"]
Nov 02 03:42:25.593338 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:42:25.593323    4496 cpu_manager.go:414] "containerMap: removing container" podUID="1975c9eb-9a68-4d15-9515-8a03d8e63fdd" containerName="payload-processing"
Nov 02 03:42:25.593385 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:42:25.593339    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1975c9eb-9a68-4d15-9515-8a03d8e63fdd" containerName="payload-processing"
Nov 02 03:42:25.593385 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:42:25.593358    4496 cpu_manager.go:414] "containerMap: removing container" podUID="cf4aaa9a-1039-4f41-ad04-2ef9c946fe4f" containerName="payload-pre-processing"
Nov 02 03:42:25.593385 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:42:25.593363    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf4aaa9a-1039-4f41-ad04-2ef9c946fe4f" containerName="payload-pre-processing"
Nov 02 03:42:25.593486 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:42:25.593473    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="1975c9eb-9a68-4d15-9515-8a03d8e63fdd" containerName="payload-processing"
Nov 02 03:42:25.593486 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:42:25.593483    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf4aaa9a-1039-4f41-ad04-2ef9c946fe4f" containerName="payload-pre-processing"
Nov 02 03:42:25.593555 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:42:25.593495    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="c01165ba-6eca-4e2a-81b1-c743acb0fbd6" containerName="istio-proxy"
Nov 02 03:42:25.597131 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:42:25.597115    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f68c7e-5c59f9756f-mqvxw"
Nov 02 03:42:25.650106 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:42:25.650076    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-f68c7e-serving-cert\""
Nov 02 03:42:25.694499 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:42:25.694469    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c9230a0-ebae-42a0-b100-5c48210e8cd3-maas-api-tls\") pod \"maas-api-e2e-praxis-b-f68c7e-5c59f9756f-mqvxw\" (UID: \"0c9230a0-ebae-42a0-b100-5c48210e8cd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f68c7e-5c59f9756f-mqvxw"
Nov 02 03:42:25.694659 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:42:25.694516    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c9230a0-ebae-42a0-b100-5c48210e8cd3-metrics-certs\") pod \"maas-api-e2e-praxis-b-f68c7e-5c59f9756f-mqvxw\" (UID: \"0c9230a0-ebae-42a0-b100-5c48210e8cd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f68c7e-5c59f9756f-mqvxw"
Nov 02 03:42:25.694771 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:42:25.694748    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5v6qx\" (UniqueName: \"kubernetes.io/projected/0c9230a0-ebae-42a0-b100-5c48210e8cd3-kube-api-access-5v6qx\") pod \"maas-api-e2e-praxis-b-f68c7e-5c59f9756f-mqvxw\" (UID: \"0c9230a0-ebae-42a0-b100-5c48210e8cd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f68c7e-5c59f9756f-mqvxw"
Nov 02 03:42:25.694875 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:42:25.694834    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c9230a0-ebae-42a0-b100-5c48210e8cd3-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-f68c7e-5c59f9756f-mqvxw\" (UID: \"0c9230a0-ebae-42a0-b100-5c48210e8cd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f68c7e-5c59f9756f-mqvxw"
Nov 02 03:42:25.707188 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:42:25.707153    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-f68c7e-5c59f9756f-mqvxw"]
Nov 02 03:42:25.795856 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:42:25.795822    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5v6qx\" (UniqueName: \"kubernetes.io/projected/0c9230a0-ebae-42a0-b100-5c48210e8cd3-kube-api-access-5v6qx\") pod \"maas-api-e2e-praxis-b-f68c7e-5c59f9756f-mqvxw\" (UID: \"0c9230a0-ebae-42a0-b100-5c48210e8cd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f68c7e-5c59f9756f-mqvxw"
Nov 02 03:42:25.796052 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:42:25.795884    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c9230a0-ebae-42a0-b100-5c48210e8cd3-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-f68c7e-5c59f9756f-mqvxw\" (UID: \"0c9230a0-ebae-42a0-b100-5c48210e8cd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f68c7e-5c59f9756f-mqvxw"
Nov 02 03:42:25.796052 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:42:25.795935    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c9230a0-ebae-42a0-b100-5c48210e8cd3-maas-api-tls\") pod \"maas-api-e2e-praxis-b-f68c7e-5c59f9756f-mqvxw\" (UID: \"0c9230a0-ebae-42a0-b100-5c48210e8cd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f68c7e-5c59f9756f-mqvxw"
Nov 02 03:42:25.796052 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:42:25.795961    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c9230a0-ebae-42a0-b100-5c48210e8cd3-metrics-certs\") pod \"maas-api-e2e-praxis-b-f68c7e-5c59f9756f-mqvxw\" (UID: \"0c9230a0-ebae-42a0-b100-5c48210e8cd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f68c7e-5c59f9756f-mqvxw"
Nov 02 03:42:25.796239 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:42:25.796110    4496 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-f68c7e-serving-cert: secret "maas-api-e2e-praxis-b-f68c7e-serving-cert" not found
Nov 02 03:42:25.796239 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:42:25.796197    4496 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c9230a0-ebae-42a0-b100-5c48210e8cd3-maas-api-tls podName:0c9230a0-ebae-42a0-b100-5c48210e8cd3 nodeName:}" failed. No retries permitted until 2026-11-02 03:42:26.296175372 +0000 UTC m=+3286.874782747 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0c9230a0-ebae-42a0-b100-5c48210e8cd3-maas-api-tls") pod "maas-api-e2e-praxis-b-f68c7e-5c59f9756f-mqvxw" (UID: "0c9230a0-ebae-42a0-b100-5c48210e8cd3") : secret "maas-api-e2e-praxis-b-f68c7e-serving-cert" not found
Nov 02 03:42:25.796640 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:42:25.796619    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c9230a0-ebae-42a0-b100-5c48210e8cd3-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-f68c7e-5c59f9756f-mqvxw\" (UID: \"0c9230a0-ebae-42a0-b100-5c48210e8cd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f68c7e-5c59f9756f-mqvxw"
Nov 02 03:42:25.798454 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:42:25.798429    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c9230a0-ebae-42a0-b100-5c48210e8cd3-metrics-certs\") pod \"maas-api-e2e-praxis-b-f68c7e-5c59f9756f-mqvxw\" (UID: \"0c9230a0-ebae-42a0-b100-5c48210e8cd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f68c7e-5c59f9756f-mqvxw"
Nov 02 03:42:25.853987 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:42:25.853921    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5v6qx\" (UniqueName: \"kubernetes.io/projected/0c9230a0-ebae-42a0-b100-5c48210e8cd3-kube-api-access-5v6qx\") pod \"maas-api-e2e-praxis-b-f68c7e-5c59f9756f-mqvxw\" (UID: \"0c9230a0-ebae-42a0-b100-5c48210e8cd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f68c7e-5c59f9756f-mqvxw"
Nov 02 03:42:26.303314 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:42:26.303280    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c9230a0-ebae-42a0-b100-5c48210e8cd3-maas-api-tls\") pod \"maas-api-e2e-praxis-b-f68c7e-5c59f9756f-mqvxw\" (UID: \"0c9230a0-ebae-42a0-b100-5c48210e8cd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f68c7e-5c59f9756f-mqvxw"
Nov 02 03:42:26.305891 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:42:26.305867    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c9230a0-ebae-42a0-b100-5c48210e8cd3-maas-api-tls\") pod \"maas-api-e2e-praxis-b-f68c7e-5c59f9756f-mqvxw\" (UID: \"0c9230a0-ebae-42a0-b100-5c48210e8cd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f68c7e-5c59f9756f-mqvxw"
Nov 02 03:42:26.511688 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:42:26.511652    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f68c7e-5c59f9756f-mqvxw"
Nov 02 03:42:26.891725 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:42:26.891694    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-f68c7e-5c59f9756f-mqvxw"]
Nov 02 03:42:26.892281 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:42:26.892258    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c9230a0_ebae_42a0_b100_5c48210e8cd3.slice/crio-587f36934ec481a486468ea131d5c9dcf9bba6c05310d22a0ec67c6c588b13c6 WatchSource:0}: Error finding container 587f36934ec481a486468ea131d5c9dcf9bba6c05310d22a0ec67c6c588b13c6: Status 404 returned error can't find the container with id 587f36934ec481a486468ea131d5c9dcf9bba6c05310d22a0ec67c6c588b13c6
Nov 02 03:42:26.893725 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:42:26.893707    4496 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 03:42:27.291098 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:42:27.291059    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f68c7e-5c59f9756f-mqvxw" event={"ID":"0c9230a0-ebae-42a0-b100-5c48210e8cd3","Type":"ContainerStarted","Data":"587f36934ec481a486468ea131d5c9dcf9bba6c05310d22a0ec67c6c588b13c6"}
Nov 02 03:42:28.299446 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:42:28.299409    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f68c7e-5c59f9756f-mqvxw" event={"ID":"0c9230a0-ebae-42a0-b100-5c48210e8cd3","Type":"ContainerStarted","Data":"5d82d880615764d6700babd69cc368557cef6915a6f13450abf9dabd9577485a"}
Nov 02 03:42:28.299851 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:42:28.299632    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f68c7e-5c59f9756f-mqvxw"
Nov 02 03:42:28.418928 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:42:28.418869    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f68c7e-5c59f9756f-mqvxw" podStartSLOduration=2.519463942 podStartE2EDuration="3.418851587s" totalImagesPullingTime="899.387645ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:42:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:42:28.414734052 +0000 UTC m=+3288.993341433" watchObservedRunningTime="2026-11-02 03:42:28.418851587 +0000 UTC m=+3288.997458969"
Nov 02 03:42:33.536723 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:42:33.536679    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 03:42:34.313936 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:42:34.313902    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f68c7e-5c59f9756f-mqvxw"
Nov 02 03:42:38.264034 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:42:38.263989    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-167a01-7b99c886bc-k8rsp"]
Nov 02 03:42:38.373878 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:42:38.373836    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-167a01-88474547-k8xhk"]
Nov 02 03:42:40.268307 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:42:40.268272    4496 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hww8q_5c797b08-a314-44ed-8026-8a0ce1f6c36f/ovn-acl-logging/0.log"
Nov 02 03:42:40.277734 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:42:40.277714    4496 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hww8q_5c797b08-a314-44ed-8026-8a0ce1f6c36f/ovn-acl-logging/0.log"
Nov 02 03:42:40.608460 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:42:40.608384    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-3d563c-55dbc6658b-tbmjl"]
Nov 02 03:42:40.608609 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:42:40.608583    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3d563c-55dbc6658b-tbmjl" podUID="c6ab02a3-6d61-4e1b-a12e-d8c576c7377c" containerName="maas-api" containerID="cri-o://b30586882ccbb5b59960347b148f0f2b127dc38efe1cbeb173335b6ce9eb3be6" gracePeriod=30
Nov 02 03:42:40.875393 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:42:40.875322    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3d563c-55dbc6658b-tbmjl"
Nov 02 03:42:40.966864 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:42:40.966830    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6ab02a3-6d61-4e1b-a12e-d8c576c7377c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6ab02a3-6d61-4e1b-a12e-d8c576c7377c-metrics-certs\") pod \"c6ab02a3-6d61-4e1b-a12e-d8c576c7377c\" (UID: \"c6ab02a3-6d61-4e1b-a12e-d8c576c7377c\") "
Nov 02 03:42:40.967084 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:42:40.966991    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6ab02a3-6d61-4e1b-a12e-d8c576c7377c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6ab02a3-6d61-4e1b-a12e-d8c576c7377c-openshift-service-ca\") pod \"c6ab02a3-6d61-4e1b-a12e-d8c576c7377c\" (UID: \"c6ab02a3-6d61-4e1b-a12e-d8c576c7377c\") "
Nov 02 03:42:40.967167 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:42:40.967087    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6ab02a3-6d61-4e1b-a12e-d8c576c7377c-kube-api-access-blhmh\" (UniqueName: \"kubernetes.io/projected/c6ab02a3-6d61-4e1b-a12e-d8c576c7377c-kube-api-access-blhmh\") pod \"c6ab02a3-6d61-4e1b-a12e-d8c576c7377c\" (UID: \"c6ab02a3-6d61-4e1b-a12e-d8c576c7377c\") "
Nov 02 03:42:40.967167 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:42:40.967119    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6ab02a3-6d61-4e1b-a12e-d8c576c7377c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6ab02a3-6d61-4e1b-a12e-d8c576c7377c-maas-api-tls\") pod \"c6ab02a3-6d61-4e1b-a12e-d8c576c7377c\" (UID: \"c6ab02a3-6d61-4e1b-a12e-d8c576c7377c\") "
Nov 02 03:42:40.967461 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:42:40.967427    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6ab02a3-6d61-4e1b-a12e-d8c576c7377c-openshift-service-ca" pod "c6ab02a3-6d61-4e1b-a12e-d8c576c7377c" (UID: "c6ab02a3-6d61-4e1b-a12e-d8c576c7377c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:42:40.967720 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:42:40.967691    4496 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6ab02a3-6d61-4e1b-a12e-d8c576c7377c-openshift-service-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:42:40.969306 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:42:40.969283    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6ab02a3-6d61-4e1b-a12e-d8c576c7377c-maas-api-tls" pod "c6ab02a3-6d61-4e1b-a12e-d8c576c7377c" (UID: "c6ab02a3-6d61-4e1b-a12e-d8c576c7377c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 03:42:40.969407 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:42:40.969366    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6ab02a3-6d61-4e1b-a12e-d8c576c7377c-metrics-certs" pod "c6ab02a3-6d61-4e1b-a12e-d8c576c7377c" (UID: "c6ab02a3-6d61-4e1b-a12e-d8c576c7377c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 03:42:40.969576 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:42:40.969550    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6ab02a3-6d61-4e1b-a12e-d8c576c7377c-kube-api-access-blhmh" pod "c6ab02a3-6d61-4e1b-a12e-d8c576c7377c" (UID: "c6ab02a3-6d61-4e1b-a12e-d8c576c7377c"). InnerVolumeSpecName "kube-api-access-blhmh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:42:41.068785 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:42:41.068754    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blhmh\" (UniqueName: \"kubernetes.io/projected/c6ab02a3-6d61-4e1b-a12e-d8c576c7377c-kube-api-access-blhmh\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:42:41.068785 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:42:41.068782    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6ab02a3-6d61-4e1b-a12e-d8c576c7377c-maas-api-tls\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:42:41.068785 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:42:41.068791    4496 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6ab02a3-6d61-4e1b-a12e-d8c576c7377c-metrics-certs\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:42:41.401317 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:42:41.401286    4496 generic.go:356] "Generic (PLEG): container finished" podID="c6ab02a3-6d61-4e1b-a12e-d8c576c7377c" containerID="b30586882ccbb5b59960347b148f0f2b127dc38efe1cbeb173335b6ce9eb3be6" exitCode=0
Nov 02 03:42:41.401723 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:42:41.401331    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3d563c-55dbc6658b-tbmjl"
Nov 02 03:42:41.401723 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:42:41.401408    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3d563c-55dbc6658b-tbmjl" event={"ID":"c6ab02a3-6d61-4e1b-a12e-d8c576c7377c","Type":"ContainerDied","Data":"b30586882ccbb5b59960347b148f0f2b127dc38efe1cbeb173335b6ce9eb3be6"}
Nov 02 03:42:41.401723 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:42:41.401458    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3d563c-55dbc6658b-tbmjl" event={"ID":"c6ab02a3-6d61-4e1b-a12e-d8c576c7377c","Type":"ContainerDied","Data":"8fb373969071d721cdd1745545c0f963b39914a3aed50f87d057ec1bc3abfbc5"}
Nov 02 03:42:41.401723 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:42:41.401480    4496 scope.go:122] "RemoveContainer" containerID="b30586882ccbb5b59960347b148f0f2b127dc38efe1cbeb173335b6ce9eb3be6"
Nov 02 03:42:41.412985 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:42:41.412967    4496 scope.go:122] "RemoveContainer" containerID="b30586882ccbb5b59960347b148f0f2b127dc38efe1cbeb173335b6ce9eb3be6"
Nov 02 03:42:41.413297 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:42:41.413275    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b30586882ccbb5b59960347b148f0f2b127dc38efe1cbeb173335b6ce9eb3be6\": container with ID starting with b30586882ccbb5b59960347b148f0f2b127dc38efe1cbeb173335b6ce9eb3be6 not found: ID does not exist" containerID="b30586882ccbb5b59960347b148f0f2b127dc38efe1cbeb173335b6ce9eb3be6"
Nov 02 03:42:41.413394 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:42:41.413302    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b30586882ccbb5b59960347b148f0f2b127dc38efe1cbeb173335b6ce9eb3be6"} err="failed to get container status \"b30586882ccbb5b59960347b148f0f2b127dc38efe1cbeb173335b6ce9eb3be6\": rpc error: code = NotFound desc = could not find container \"b30586882ccbb5b59960347b148f0f2b127dc38efe1cbeb173335b6ce9eb3be6\": container with ID starting with b30586882ccbb5b59960347b148f0f2b127dc38efe1cbeb173335b6ce9eb3be6 not found: ID does not exist"
Nov 02 03:42:41.583518 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:42:41.583476    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-3d563c-55dbc6658b-tbmjl"]
Nov 02 03:42:41.583721 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:42:41.583645    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-3d563c-55dbc6658b-tbmjl"]
Nov 02 03:42:41.936168 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:42:41.936136    4496 scope.go:122] "RemoveContainer" containerID="aa62060c88809929e9ffc9de4c20f01ba8c60d77ec1d379d19e578da792d1e94"
Nov 02 03:42:41.945871 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:42:41.945849    4496 scope.go:122] "RemoveContainer" containerID="f00177b934ebd86616420812c5424536829b95ea4a321fae58813671a79f51d2"
Nov 02 03:42:43.264513 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:42:43.264459    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-167a01-7b99c886bc-k8rsp" podUID="3524ec44-5fdd-4669-b169-99b1d1f4f353" containerName="payload-processing" containerID="cri-o://eb25bae3e2c93486a2bd913199cda2e4639d25363952aa4935ba874e6b5cd38b" gracePeriod=25
Nov 02 03:42:43.374827 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:42:43.374769    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-167a01-88474547-k8xhk" podUID="c0e67481-3663-4f05-aff8-3a38bc7cc8c5" containerName="payload-pre-processing" containerID="cri-o://829189e483e18ce71d90b877a03c32142ba96a4d6f89aa8e2db51aac759f9869" gracePeriod=25
Nov 02 03:42:43.419884 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:42:43.419850    4496 generic.go:356] "Generic (PLEG): container finished" podID="3524ec44-5fdd-4669-b169-99b1d1f4f353" containerID="eb25bae3e2c93486a2bd913199cda2e4639d25363952aa4935ba874e6b5cd38b" exitCode=0
Nov 02 03:42:43.420076 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:42:43.419925    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-167a01-7b99c886bc-k8rsp" event={"ID":"3524ec44-5fdd-4669-b169-99b1d1f4f353","Type":"ContainerDied","Data":"eb25bae3e2c93486a2bd913199cda2e4639d25363952aa4935ba874e6b5cd38b"}
Nov 02 03:42:43.544044 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:42:43.543985    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-167a01-7b99c886bc-k8rsp"
Nov 02 03:42:43.575166 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:42:43.575137    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6ab02a3-6d61-4e1b-a12e-d8c576c7377c" path="/var/lib/kubelet/pods/c6ab02a3-6d61-4e1b-a12e-d8c576c7377c/volumes"
Nov 02 03:42:43.606966 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:42:43.606936    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3524ec44-5fdd-4669-b169-99b1d1f4f353-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3524ec44-5fdd-4669-b169-99b1d1f4f353-plugins-config-volume\") pod \"3524ec44-5fdd-4669-b169-99b1d1f4f353\" (UID: \"3524ec44-5fdd-4669-b169-99b1d1f4f353\") "
Nov 02 03:42:43.607212 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:42:43.607190    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3524ec44-5fdd-4669-b169-99b1d1f4f353-kube-api-access-9vcll\" (UniqueName: \"kubernetes.io/projected/3524ec44-5fdd-4669-b169-99b1d1f4f353-kube-api-access-9vcll\") pod \"3524ec44-5fdd-4669-b169-99b1d1f4f353\" (UID: \"3524ec44-5fdd-4669-b169-99b1d1f4f353\") "
Nov 02 03:42:43.607325 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:42:43.607302    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3524ec44-5fdd-4669-b169-99b1d1f4f353-plugins-config-volume" pod "3524ec44-5fdd-4669-b169-99b1d1f4f353" (UID: "3524ec44-5fdd-4669-b169-99b1d1f4f353"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:42:43.607882 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:42:43.607755    4496 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3524ec44-5fdd-4669-b169-99b1d1f4f353-plugins-config-volume\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:42:43.609893 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:42:43.609864    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3524ec44-5fdd-4669-b169-99b1d1f4f353-kube-api-access-9vcll" pod "3524ec44-5fdd-4669-b169-99b1d1f4f353" (UID: "3524ec44-5fdd-4669-b169-99b1d1f4f353"). InnerVolumeSpecName "kube-api-access-9vcll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:42:43.640224 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:42:43.640201    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-167a01-88474547-k8xhk"
Nov 02 03:42:43.708263 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:42:43.708233    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0e67481-3663-4f05-aff8-3a38bc7cc8c5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0e67481-3663-4f05-aff8-3a38bc7cc8c5-plugins-config-volume\") pod \"c0e67481-3663-4f05-aff8-3a38bc7cc8c5\" (UID: \"c0e67481-3663-4f05-aff8-3a38bc7cc8c5\") "
Nov 02 03:42:43.708432 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:42:43.708287    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0e67481-3663-4f05-aff8-3a38bc7cc8c5-kube-api-access-mdwpg\" (UniqueName: \"kubernetes.io/projected/c0e67481-3663-4f05-aff8-3a38bc7cc8c5-kube-api-access-mdwpg\") pod \"c0e67481-3663-4f05-aff8-3a38bc7cc8c5\" (UID: \"c0e67481-3663-4f05-aff8-3a38bc7cc8c5\") "
Nov 02 03:42:43.708605 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:42:43.708586    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0e67481-3663-4f05-aff8-3a38bc7cc8c5-plugins-config-volume" pod "c0e67481-3663-4f05-aff8-3a38bc7cc8c5" (UID: "c0e67481-3663-4f05-aff8-3a38bc7cc8c5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:42:43.708762 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:42:43.708741    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vcll\" (UniqueName: \"kubernetes.io/projected/3524ec44-5fdd-4669-b169-99b1d1f4f353-kube-api-access-9vcll\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:42:43.708762 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:42:43.708757    4496 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0e67481-3663-4f05-aff8-3a38bc7cc8c5-plugins-config-volume\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:42:43.710381 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:42:43.710359    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0e67481-3663-4f05-aff8-3a38bc7cc8c5-kube-api-access-mdwpg" pod "c0e67481-3663-4f05-aff8-3a38bc7cc8c5" (UID: "c0e67481-3663-4f05-aff8-3a38bc7cc8c5"). InnerVolumeSpecName "kube-api-access-mdwpg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:42:43.809922 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:42:43.809843    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdwpg\" (UniqueName: \"kubernetes.io/projected/c0e67481-3663-4f05-aff8-3a38bc7cc8c5-kube-api-access-mdwpg\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:42:44.427819 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:42:44.427769    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-167a01-7b99c886bc-k8rsp"
Nov 02 03:42:44.427819 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:42:44.427815    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-167a01-7b99c886bc-k8rsp" event={"ID":"3524ec44-5fdd-4669-b169-99b1d1f4f353","Type":"ContainerDied","Data":"f6aa835a9ec4029af1133199c27ba2bd1bee831e2d8d90b4c4b7e603faf861f9"}
Nov 02 03:42:44.428349 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:42:44.427875    4496 scope.go:122] "RemoveContainer" containerID="eb25bae3e2c93486a2bd913199cda2e4639d25363952aa4935ba874e6b5cd38b"
Nov 02 03:42:44.429289 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:42:44.429242    4496 generic.go:356] "Generic (PLEG): container finished" podID="c0e67481-3663-4f05-aff8-3a38bc7cc8c5" containerID="829189e483e18ce71d90b877a03c32142ba96a4d6f89aa8e2db51aac759f9869" exitCode=0
Nov 02 03:42:44.429289 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:42:44.429278    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-167a01-88474547-k8xhk"
Nov 02 03:42:44.429431 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:42:44.429309    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-167a01-88474547-k8xhk" event={"ID":"c0e67481-3663-4f05-aff8-3a38bc7cc8c5","Type":"ContainerDied","Data":"829189e483e18ce71d90b877a03c32142ba96a4d6f89aa8e2db51aac759f9869"}
Nov 02 03:42:44.429431 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:42:44.429378    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-167a01-88474547-k8xhk" event={"ID":"c0e67481-3663-4f05-aff8-3a38bc7cc8c5","Type":"ContainerDied","Data":"e970409f741661308f3bdb7189181ab55257d4d42ec4609c6d1acef93a1d2180"}
Nov 02 03:42:44.440520 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:42:44.440502    4496 scope.go:122] "RemoveContainer" containerID="829189e483e18ce71d90b877a03c32142ba96a4d6f89aa8e2db51aac759f9869"
Nov 02 03:42:44.450008 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:42:44.449987    4496 scope.go:122] "RemoveContainer" containerID="829189e483e18ce71d90b877a03c32142ba96a4d6f89aa8e2db51aac759f9869"
Nov 02 03:42:44.450303 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:42:44.450287    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"829189e483e18ce71d90b877a03c32142ba96a4d6f89aa8e2db51aac759f9869\": container with ID starting with 829189e483e18ce71d90b877a03c32142ba96a4d6f89aa8e2db51aac759f9869 not found: ID does not exist" containerID="829189e483e18ce71d90b877a03c32142ba96a4d6f89aa8e2db51aac759f9869"
Nov 02 03:42:44.450364 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:42:44.450311    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"829189e483e18ce71d90b877a03c32142ba96a4d6f89aa8e2db51aac759f9869"} err="failed to get container status \"829189e483e18ce71d90b877a03c32142ba96a4d6f89aa8e2db51aac759f9869\": rpc error: code = NotFound desc = could not find container \"829189e483e18ce71d90b877a03c32142ba96a4d6f89aa8e2db51aac759f9869\": container with ID starting with 829189e483e18ce71d90b877a03c32142ba96a4d6f89aa8e2db51aac759f9869 not found: ID does not exist"
Nov 02 03:42:44.595337 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:42:44.595266    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-167a01-7b99c886bc-k8rsp"]
Nov 02 03:42:44.645953 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:42:44.645918    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-167a01-7b99c886bc-k8rsp"]
Nov 02 03:42:44.756784 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:42:44.756754    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-167a01-88474547-k8xhk"]
Nov 02 03:42:44.759908 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:42:44.759882    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-167a01-88474547-k8xhk"]
Nov 02 03:42:45.573792 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:42:45.573746    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3524ec44-5fdd-4669-b169-99b1d1f4f353" path="/var/lib/kubelet/pods/3524ec44-5fdd-4669-b169-99b1d1f4f353/volumes"
Nov 02 03:42:45.574389 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:42:45.574365    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0e67481-3663-4f05-aff8-3a38bc7cc8c5" path="/var/lib/kubelet/pods/c0e67481-3663-4f05-aff8-3a38bc7cc8c5/volumes"
Nov 02 03:42:53.046394 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:42:53.046360    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29ea31d1-clkzv"]
Nov 02 03:42:53.096704 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:42:53.096661    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29ea31d1-clkzv"]
Nov 02 03:42:53.574150 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:42:53.574115    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ab84d1d-22c4-40a8-b0d5-9e78a4f8923a" path="/var/lib/kubelet/pods/7ab84d1d-22c4-40a8-b0d5-9e78a4f8923a/volumes"
Nov 02 03:43:03.050026 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:03.049985    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d98689e2-ztq7z"]
Nov 02 03:43:03.100620 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:03.100581    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d98689e2-ztq7z"]
Nov 02 03:43:03.526925 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:43:03.526889    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 03:43:03.573546 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:03.573509    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e768b97b-caa5-43c5-9db4-156a02a156a4" path="/var/lib/kubelet/pods/e768b97b-caa5-43c5-9db4-156a02a156a4/volumes"
Nov 02 03:43:08.379937 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:08.379895    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-f68c7e-openshift-default-db7d68949-c7q6l"]
Nov 02 03:43:08.380856 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:08.380833    4496 cpu_manager.go:414] "containerMap: removing container" podUID="c0e67481-3663-4f05-aff8-3a38bc7cc8c5" containerName="payload-pre-processing"
Nov 02 03:43:08.380948 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:08.380861    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0e67481-3663-4f05-aff8-3a38bc7cc8c5" containerName="payload-pre-processing"
Nov 02 03:43:08.380948 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:08.380890    4496 cpu_manager.go:414] "containerMap: removing container" podUID="3524ec44-5fdd-4669-b169-99b1d1f4f353" containerName="payload-processing"
Nov 02 03:43:08.380948 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:08.380898    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3524ec44-5fdd-4669-b169-99b1d1f4f353" containerName="payload-processing"
Nov 02 03:43:08.380948 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:08.380929    4496 cpu_manager.go:414] "containerMap: removing container" podUID="c6ab02a3-6d61-4e1b-a12e-d8c576c7377c" containerName="maas-api"
Nov 02 03:43:08.380948 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:08.380937    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6ab02a3-6d61-4e1b-a12e-d8c576c7377c" containerName="maas-api"
Nov 02 03:43:08.381134 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:08.381109    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="3524ec44-5fdd-4669-b169-99b1d1f4f353" containerName="payload-processing"
Nov 02 03:43:08.381134 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:08.381125    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6ab02a3-6d61-4e1b-a12e-d8c576c7377c" containerName="maas-api"
Nov 02 03:43:08.381207 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:08.381146    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0e67481-3663-4f05-aff8-3a38bc7cc8c5" containerName="payload-pre-processing"
Nov 02 03:43:08.386864 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:08.386843    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-f68c7e-openshift-default-db7d68949-c7q6l"
Nov 02 03:43:08.497698 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:08.497660    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-f68c7e-openshift-default-db7d68949-c7q6l"]
Nov 02 03:43:08.499770 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:08.499744    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/568d206f-e9d9-4f0a-b641-d7b2d90c8ee7-istio-envoy\") pod \"e2e-praxis-b-f68c7e-openshift-default-db7d68949-c7q6l\" (UID: \"568d206f-e9d9-4f0a-b641-d7b2d90c8ee7\") " pod="openshift-ingress/e2e-praxis-b-f68c7e-openshift-default-db7d68949-c7q6l"
Nov 02 03:43:08.499916 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:08.499791    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/568d206f-e9d9-4f0a-b641-d7b2d90c8ee7-istio-token\") pod \"e2e-praxis-b-f68c7e-openshift-default-db7d68949-c7q6l\" (UID: \"568d206f-e9d9-4f0a-b641-d7b2d90c8ee7\") " pod="openshift-ingress/e2e-praxis-b-f68c7e-openshift-default-db7d68949-c7q6l"
Nov 02 03:43:08.499916 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:08.499823    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/568d206f-e9d9-4f0a-b641-d7b2d90c8ee7-credential-socket\") pod \"e2e-praxis-b-f68c7e-openshift-default-db7d68949-c7q6l\" (UID: \"568d206f-e9d9-4f0a-b641-d7b2d90c8ee7\") " pod="openshift-ingress/e2e-praxis-b-f68c7e-openshift-default-db7d68949-c7q6l"
Nov 02 03:43:08.500029 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:08.499930    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/568d206f-e9d9-4f0a-b641-d7b2d90c8ee7-istio-data\") pod \"e2e-praxis-b-f68c7e-openshift-default-db7d68949-c7q6l\" (UID: \"568d206f-e9d9-4f0a-b641-d7b2d90c8ee7\") " pod="openshift-ingress/e2e-praxis-b-f68c7e-openshift-default-db7d68949-c7q6l"
Nov 02 03:43:08.500029 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:08.499948    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/568d206f-e9d9-4f0a-b641-d7b2d90c8ee7-istiod-ca-cert\") pod \"e2e-praxis-b-f68c7e-openshift-default-db7d68949-c7q6l\" (UID: \"568d206f-e9d9-4f0a-b641-d7b2d90c8ee7\") " pod="openshift-ingress/e2e-praxis-b-f68c7e-openshift-default-db7d68949-c7q6l"
Nov 02 03:43:08.500029 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:08.499970    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrgq8\" (UniqueName: \"kubernetes.io/projected/568d206f-e9d9-4f0a-b641-d7b2d90c8ee7-kube-api-access-nrgq8\") pod \"e2e-praxis-b-f68c7e-openshift-default-db7d68949-c7q6l\" (UID: \"568d206f-e9d9-4f0a-b641-d7b2d90c8ee7\") " pod="openshift-ingress/e2e-praxis-b-f68c7e-openshift-default-db7d68949-c7q6l"
Nov 02 03:43:08.500150 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:08.500060    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/568d206f-e9d9-4f0a-b641-d7b2d90c8ee7-workload-socket\") pod \"e2e-praxis-b-f68c7e-openshift-default-db7d68949-c7q6l\" (UID: \"568d206f-e9d9-4f0a-b641-d7b2d90c8ee7\") " pod="openshift-ingress/e2e-praxis-b-f68c7e-openshift-default-db7d68949-c7q6l"
Nov 02 03:43:08.500150 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:08.500091    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/568d206f-e9d9-4f0a-b641-d7b2d90c8ee7-workload-certs\") pod \"e2e-praxis-b-f68c7e-openshift-default-db7d68949-c7q6l\" (UID: \"568d206f-e9d9-4f0a-b641-d7b2d90c8ee7\") " pod="openshift-ingress/e2e-praxis-b-f68c7e-openshift-default-db7d68949-c7q6l"
Nov 02 03:43:08.500150 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:08.500115    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/568d206f-e9d9-4f0a-b641-d7b2d90c8ee7-istio-podinfo\") pod \"e2e-praxis-b-f68c7e-openshift-default-db7d68949-c7q6l\" (UID: \"568d206f-e9d9-4f0a-b641-d7b2d90c8ee7\") " pod="openshift-ingress/e2e-praxis-b-f68c7e-openshift-default-db7d68949-c7q6l"
Nov 02 03:43:08.523639 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:08.523617    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-f68c7e-openshift-default-dockercfg-9xx98\""
Nov 02 03:43:08.601028 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:08.600980    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/568d206f-e9d9-4f0a-b641-d7b2d90c8ee7-istio-token\") pod \"e2e-praxis-b-f68c7e-openshift-default-db7d68949-c7q6l\" (UID: \"568d206f-e9d9-4f0a-b641-d7b2d90c8ee7\") " pod="openshift-ingress/e2e-praxis-b-f68c7e-openshift-default-db7d68949-c7q6l"
Nov 02 03:43:08.601208 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:08.601061    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/568d206f-e9d9-4f0a-b641-d7b2d90c8ee7-credential-socket\") pod \"e2e-praxis-b-f68c7e-openshift-default-db7d68949-c7q6l\" (UID: \"568d206f-e9d9-4f0a-b641-d7b2d90c8ee7\") " pod="openshift-ingress/e2e-praxis-b-f68c7e-openshift-default-db7d68949-c7q6l"
Nov 02 03:43:08.601208 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:08.601095    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/568d206f-e9d9-4f0a-b641-d7b2d90c8ee7-istio-data\") pod \"e2e-praxis-b-f68c7e-openshift-default-db7d68949-c7q6l\" (UID: \"568d206f-e9d9-4f0a-b641-d7b2d90c8ee7\") " pod="openshift-ingress/e2e-praxis-b-f68c7e-openshift-default-db7d68949-c7q6l"
Nov 02 03:43:08.601208 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:08.601116    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/568d206f-e9d9-4f0a-b641-d7b2d90c8ee7-istiod-ca-cert\") pod \"e2e-praxis-b-f68c7e-openshift-default-db7d68949-c7q6l\" (UID: \"568d206f-e9d9-4f0a-b641-d7b2d90c8ee7\") " pod="openshift-ingress/e2e-praxis-b-f68c7e-openshift-default-db7d68949-c7q6l"
Nov 02 03:43:08.601208 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:08.601140    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrgq8\" (UniqueName: \"kubernetes.io/projected/568d206f-e9d9-4f0a-b641-d7b2d90c8ee7-kube-api-access-nrgq8\") pod \"e2e-praxis-b-f68c7e-openshift-default-db7d68949-c7q6l\" (UID: \"568d206f-e9d9-4f0a-b641-d7b2d90c8ee7\") " pod="openshift-ingress/e2e-praxis-b-f68c7e-openshift-default-db7d68949-c7q6l"
Nov 02 03:43:08.601208 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:08.601171    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/568d206f-e9d9-4f0a-b641-d7b2d90c8ee7-workload-socket\") pod \"e2e-praxis-b-f68c7e-openshift-default-db7d68949-c7q6l\" (UID: \"568d206f-e9d9-4f0a-b641-d7b2d90c8ee7\") " pod="openshift-ingress/e2e-praxis-b-f68c7e-openshift-default-db7d68949-c7q6l"
Nov 02 03:43:08.601208 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:08.601200    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/568d206f-e9d9-4f0a-b641-d7b2d90c8ee7-workload-certs\") pod \"e2e-praxis-b-f68c7e-openshift-default-db7d68949-c7q6l\" (UID: \"568d206f-e9d9-4f0a-b641-d7b2d90c8ee7\") " pod="openshift-ingress/e2e-praxis-b-f68c7e-openshift-default-db7d68949-c7q6l"
Nov 02 03:43:08.601542 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:08.601297    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/568d206f-e9d9-4f0a-b641-d7b2d90c8ee7-istio-podinfo\") pod \"e2e-praxis-b-f68c7e-openshift-default-db7d68949-c7q6l\" (UID: \"568d206f-e9d9-4f0a-b641-d7b2d90c8ee7\") " pod="openshift-ingress/e2e-praxis-b-f68c7e-openshift-default-db7d68949-c7q6l"
Nov 02 03:43:08.601591 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:08.601547    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/568d206f-e9d9-4f0a-b641-d7b2d90c8ee7-credential-socket\") pod \"e2e-praxis-b-f68c7e-openshift-default-db7d68949-c7q6l\" (UID: \"568d206f-e9d9-4f0a-b641-d7b2d90c8ee7\") " pod="openshift-ingress/e2e-praxis-b-f68c7e-openshift-default-db7d68949-c7q6l"
Nov 02 03:43:08.601716 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:08.601691    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/568d206f-e9d9-4f0a-b641-d7b2d90c8ee7-istio-data\") pod \"e2e-praxis-b-f68c7e-openshift-default-db7d68949-c7q6l\" (UID: \"568d206f-e9d9-4f0a-b641-d7b2d90c8ee7\") " pod="openshift-ingress/e2e-praxis-b-f68c7e-openshift-default-db7d68949-c7q6l"
Nov 02 03:43:08.601821 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:08.601727    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/568d206f-e9d9-4f0a-b641-d7b2d90c8ee7-istio-envoy\") pod \"e2e-praxis-b-f68c7e-openshift-default-db7d68949-c7q6l\" (UID: \"568d206f-e9d9-4f0a-b641-d7b2d90c8ee7\") " pod="openshift-ingress/e2e-praxis-b-f68c7e-openshift-default-db7d68949-c7q6l"
Nov 02 03:43:08.601821 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:08.601786    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/568d206f-e9d9-4f0a-b641-d7b2d90c8ee7-workload-socket\") pod \"e2e-praxis-b-f68c7e-openshift-default-db7d68949-c7q6l\" (UID: \"568d206f-e9d9-4f0a-b641-d7b2d90c8ee7\") " pod="openshift-ingress/e2e-praxis-b-f68c7e-openshift-default-db7d68949-c7q6l"
Nov 02 03:43:08.602067 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:08.602044    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/568d206f-e9d9-4f0a-b641-d7b2d90c8ee7-istiod-ca-cert\") pod \"e2e-praxis-b-f68c7e-openshift-default-db7d68949-c7q6l\" (UID: \"568d206f-e9d9-4f0a-b641-d7b2d90c8ee7\") " pod="openshift-ingress/e2e-praxis-b-f68c7e-openshift-default-db7d68949-c7q6l"
Nov 02 03:43:08.602181 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:08.602163    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/568d206f-e9d9-4f0a-b641-d7b2d90c8ee7-workload-certs\") pod \"e2e-praxis-b-f68c7e-openshift-default-db7d68949-c7q6l\" (UID: \"568d206f-e9d9-4f0a-b641-d7b2d90c8ee7\") " pod="openshift-ingress/e2e-praxis-b-f68c7e-openshift-default-db7d68949-c7q6l"
Nov 02 03:43:08.603905 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:08.603884    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/568d206f-e9d9-4f0a-b641-d7b2d90c8ee7-istio-envoy\") pod \"e2e-praxis-b-f68c7e-openshift-default-db7d68949-c7q6l\" (UID: \"568d206f-e9d9-4f0a-b641-d7b2d90c8ee7\") " pod="openshift-ingress/e2e-praxis-b-f68c7e-openshift-default-db7d68949-c7q6l"
Nov 02 03:43:08.603986 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:08.603929    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/568d206f-e9d9-4f0a-b641-d7b2d90c8ee7-istio-podinfo\") pod \"e2e-praxis-b-f68c7e-openshift-default-db7d68949-c7q6l\" (UID: \"568d206f-e9d9-4f0a-b641-d7b2d90c8ee7\") " pod="openshift-ingress/e2e-praxis-b-f68c7e-openshift-default-db7d68949-c7q6l"
Nov 02 03:43:08.659116 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:08.659035    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrgq8\" (UniqueName: \"kubernetes.io/projected/568d206f-e9d9-4f0a-b641-d7b2d90c8ee7-kube-api-access-nrgq8\") pod \"e2e-praxis-b-f68c7e-openshift-default-db7d68949-c7q6l\" (UID: \"568d206f-e9d9-4f0a-b641-d7b2d90c8ee7\") " pod="openshift-ingress/e2e-praxis-b-f68c7e-openshift-default-db7d68949-c7q6l"
Nov 02 03:43:08.659321 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:08.659296    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/568d206f-e9d9-4f0a-b641-d7b2d90c8ee7-istio-token\") pod \"e2e-praxis-b-f68c7e-openshift-default-db7d68949-c7q6l\" (UID: \"568d206f-e9d9-4f0a-b641-d7b2d90c8ee7\") " pod="openshift-ingress/e2e-praxis-b-f68c7e-openshift-default-db7d68949-c7q6l"
Nov 02 03:43:08.701106 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:08.701057    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-f68c7e-openshift-default-db7d68949-c7q6l"
Nov 02 03:43:09.133981 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:09.133954    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-f68c7e-openshift-default-db7d68949-c7q6l"]
Nov 02 03:43:09.134916 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:43:09.134889    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod568d206f_e9d9_4f0a_b641_d7b2d90c8ee7.slice/crio-f97c73ff9a8dc0c9f28a842279d7831d101bb9ea9b44f91649321b5c0e391f62 WatchSource:0}: Error finding container f97c73ff9a8dc0c9f28a842279d7831d101bb9ea9b44f91649321b5c0e391f62: Status 404 returned error can't find the container with id f97c73ff9a8dc0c9f28a842279d7831d101bb9ea9b44f91649321b5c0e391f62
Nov 02 03:43:09.138121 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:09.138079    4496 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 03:43:09.138216 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:09.138164    4496 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 03:43:09.138216 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:09.138208    4496 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 03:43:09.609351 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:09.609313    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-f68c7e-openshift-default-db7d68949-c7q6l" event={"ID":"568d206f-e9d9-4f0a-b641-d7b2d90c8ee7","Type":"ContainerStarted","Data":"265541d06fa08c6ef738cae800ba6dff8c6d791c70bca23166a4ce599b068b92"}
Nov 02 03:43:09.609769 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:09.609383    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-f68c7e-openshift-default-db7d68949-c7q6l" event={"ID":"568d206f-e9d9-4f0a-b641-d7b2d90c8ee7","Type":"ContainerStarted","Data":"f97c73ff9a8dc0c9f28a842279d7831d101bb9ea9b44f91649321b5c0e391f62"}
Nov 02 03:43:09.702188 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:09.702151    4496 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-f68c7e-openshift-default-db7d68949-c7q6l"
Nov 02 03:43:09.731004 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:09.730949    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-f68c7e-openshift-default-db7d68949-c7q6l" podStartSLOduration=1.730935708 podStartE2EDuration="1.730935708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:43:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:43:09.729590841 +0000 UTC m=+3330.308198223" watchObservedRunningTime="2026-11-02 03:43:09.730935708 +0000 UTC m=+3330.309543087"
Nov 02 03:43:10.703299 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:10.703263    4496 patch_prober.go:29] interesting pod/e2e-praxis-b-f68c7e-openshift-default-db7d68949-c7q6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" start-of-body=
Nov 02 03:43:10.703727 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:10.703325    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-f68c7e-openshift-default-db7d68949-c7q6l" podUID="568d206f-e9d9-4f0a-b641-d7b2d90c8ee7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused"
Nov 02 03:43:11.701745 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:11.701708    4496 patch_prober.go:29] interesting pod/e2e-praxis-b-f68c7e-openshift-default-db7d68949-c7q6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" start-of-body=
Nov 02 03:43:11.701955 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:11.701767    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-f68c7e-openshift-default-db7d68949-c7q6l" podUID="568d206f-e9d9-4f0a-b641-d7b2d90c8ee7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused"
Nov 02 03:43:12.065706 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:12.065665    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-167a01-7dbfb6897f-6xfws"]
Nov 02 03:43:12.066198 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:12.065858    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-167a01-7dbfb6897f-6xfws" podUID="4a600497-7759-4a18-adcd-29c344cba773" containerName="maas-api" containerID="cri-o://95210ae1082b473faf043c5cf8b6c56cff9195c681e746140b7dde1432bc1794" gracePeriod=30
Nov 02 03:43:12.428810 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:12.428781    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-167a01-7dbfb6897f-6xfws"
Nov 02 03:43:12.469433 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:12.469400    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a600497-7759-4a18-adcd-29c344cba773-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a600497-7759-4a18-adcd-29c344cba773-maas-api-tls\") pod \"4a600497-7759-4a18-adcd-29c344cba773\" (UID: \"4a600497-7759-4a18-adcd-29c344cba773\") "
Nov 02 03:43:12.469633 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:12.469494    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a600497-7759-4a18-adcd-29c344cba773-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a600497-7759-4a18-adcd-29c344cba773-openshift-service-ca\") pod \"4a600497-7759-4a18-adcd-29c344cba773\" (UID: \"4a600497-7759-4a18-adcd-29c344cba773\") "
Nov 02 03:43:12.469633 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:12.469543    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a600497-7759-4a18-adcd-29c344cba773-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a600497-7759-4a18-adcd-29c344cba773-metrics-certs\") pod \"4a600497-7759-4a18-adcd-29c344cba773\" (UID: \"4a600497-7759-4a18-adcd-29c344cba773\") "
Nov 02 03:43:12.469758 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:12.469662    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a600497-7759-4a18-adcd-29c344cba773-kube-api-access-6nrj8\" (UniqueName: \"kubernetes.io/projected/4a600497-7759-4a18-adcd-29c344cba773-kube-api-access-6nrj8\") pod \"4a600497-7759-4a18-adcd-29c344cba773\" (UID: \"4a600497-7759-4a18-adcd-29c344cba773\") "
Nov 02 03:43:12.470120 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:12.470084    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a600497-7759-4a18-adcd-29c344cba773-openshift-service-ca" pod "4a600497-7759-4a18-adcd-29c344cba773" (UID: "4a600497-7759-4a18-adcd-29c344cba773"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:43:12.471436 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:12.471410    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a600497-7759-4a18-adcd-29c344cba773-maas-api-tls" pod "4a600497-7759-4a18-adcd-29c344cba773" (UID: "4a600497-7759-4a18-adcd-29c344cba773"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 03:43:12.471942 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:12.471916    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a600497-7759-4a18-adcd-29c344cba773-kube-api-access-6nrj8" pod "4a600497-7759-4a18-adcd-29c344cba773" (UID: "4a600497-7759-4a18-adcd-29c344cba773"). InnerVolumeSpecName "kube-api-access-6nrj8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:43:12.472059 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:12.472010    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a600497-7759-4a18-adcd-29c344cba773-metrics-certs" pod "4a600497-7759-4a18-adcd-29c344cba773" (UID: "4a600497-7759-4a18-adcd-29c344cba773"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 03:43:12.571360 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:12.571326    4496 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a600497-7759-4a18-adcd-29c344cba773-openshift-service-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:43:12.571360 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:12.571363    4496 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a600497-7759-4a18-adcd-29c344cba773-metrics-certs\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:43:12.571563 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:12.571380    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nrj8\" (UniqueName: \"kubernetes.io/projected/4a600497-7759-4a18-adcd-29c344cba773-kube-api-access-6nrj8\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:43:12.571563 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:12.571395    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a600497-7759-4a18-adcd-29c344cba773-maas-api-tls\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:43:12.630007 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:12.629974    4496 generic.go:356] "Generic (PLEG): container finished" podID="4a600497-7759-4a18-adcd-29c344cba773" containerID="95210ae1082b473faf043c5cf8b6c56cff9195c681e746140b7dde1432bc1794" exitCode=0
Nov 02 03:43:12.630199 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:12.630067    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-167a01-7dbfb6897f-6xfws" event={"ID":"4a600497-7759-4a18-adcd-29c344cba773","Type":"ContainerDied","Data":"95210ae1082b473faf043c5cf8b6c56cff9195c681e746140b7dde1432bc1794"}
Nov 02 03:43:12.630199 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:12.630126    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-167a01-7dbfb6897f-6xfws"
Nov 02 03:43:12.630199 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:12.630138    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-167a01-7dbfb6897f-6xfws" event={"ID":"4a600497-7759-4a18-adcd-29c344cba773","Type":"ContainerDied","Data":"9ad85a55d194c4e3b2c208e0e4fa8d355c1e2e9083cedd61313dc3b671264865"}
Nov 02 03:43:12.630199 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:12.630154    4496 scope.go:122] "RemoveContainer" containerID="95210ae1082b473faf043c5cf8b6c56cff9195c681e746140b7dde1432bc1794"
Nov 02 03:43:12.667146 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:12.667123    4496 scope.go:122] "RemoveContainer" containerID="95210ae1082b473faf043c5cf8b6c56cff9195c681e746140b7dde1432bc1794"
Nov 02 03:43:12.667620 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:43:12.667593    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95210ae1082b473faf043c5cf8b6c56cff9195c681e746140b7dde1432bc1794\": container with ID starting with 95210ae1082b473faf043c5cf8b6c56cff9195c681e746140b7dde1432bc1794 not found: ID does not exist" containerID="95210ae1082b473faf043c5cf8b6c56cff9195c681e746140b7dde1432bc1794"
Nov 02 03:43:12.667725 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:12.667629    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95210ae1082b473faf043c5cf8b6c56cff9195c681e746140b7dde1432bc1794"} err="failed to get container status \"95210ae1082b473faf043c5cf8b6c56cff9195c681e746140b7dde1432bc1794\": rpc error: code = NotFound desc = could not find container \"95210ae1082b473faf043c5cf8b6c56cff9195c681e746140b7dde1432bc1794\": container with ID starting with 95210ae1082b473faf043c5cf8b6c56cff9195c681e746140b7dde1432bc1794 not found: ID does not exist"
Nov 02 03:43:12.702345 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:12.702315    4496 patch_prober.go:29] interesting pod/e2e-praxis-b-f68c7e-openshift-default-db7d68949-c7q6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" start-of-body=
Nov 02 03:43:12.702458 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:12.702363    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-f68c7e-openshift-default-db7d68949-c7q6l" podUID="568d206f-e9d9-4f0a-b641-d7b2d90c8ee7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused"
Nov 02 03:43:12.801195 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:12.801157    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-167a01-7dbfb6897f-6xfws"]
Nov 02 03:43:12.806375 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:12.806339    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-167a01-7dbfb6897f-6xfws"]
Nov 02 03:43:13.574767 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:13.574730    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a600497-7759-4a18-adcd-29c344cba773" path="/var/lib/kubelet/pods/4a600497-7759-4a18-adcd-29c344cba773/volumes"
Nov 02 03:43:13.701704 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:13.701674    4496 patch_prober.go:29] interesting pod/e2e-praxis-b-f68c7e-openshift-default-db7d68949-c7q6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" start-of-body=
Nov 02 03:43:13.701878 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:13.701721    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-f68c7e-openshift-default-db7d68949-c7q6l" podUID="568d206f-e9d9-4f0a-b641-d7b2d90c8ee7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused"
Nov 02 03:43:14.702359 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:14.702324    4496 patch_prober.go:29] interesting pod/e2e-praxis-b-f68c7e-openshift-default-db7d68949-c7q6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" start-of-body=
Nov 02 03:43:14.702771 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:14.702400    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-f68c7e-openshift-default-db7d68949-c7q6l" podUID="568d206f-e9d9-4f0a-b641-d7b2d90c8ee7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused"
Nov 02 03:43:15.706216 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:15.706179    4496 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-f68c7e-openshift-default-db7d68949-c7q6l"
Nov 02 03:43:15.706712 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:15.706685    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-f68c7e-openshift-default-db7d68949-c7q6l"
Nov 02 03:43:15.707416 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:15.707393    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-f68c7e-openshift-default-db7d68949-c7q6l"
Nov 02 03:43:20.134112 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:20.134075    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-2912e6-5468f5d46f-czlhp"]
Nov 02 03:43:20.241348 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:20.241316    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-2912e6-66887f8d9f-fmjvw"]
Nov 02 03:43:25.134351 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:25.134302    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-2912e6-5468f5d46f-czlhp" podUID="3cf215ca-ff27-4a44-a3a7-7ce21695a0d5" containerName="payload-processing" containerID="cri-o://d6aad296c29cf6935d401db1cc093edbbb6cfe14422b0a35e7ebf93ade4d2571" gracePeriod=25
Nov 02 03:43:25.242130 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:25.242082    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2912e6-66887f8d9f-fmjvw" podUID="5e4bc2e9-17cc-4574-adbc-febeb3977823" containerName="payload-pre-processing" containerID="cri-o://edd8d80d28c66dc24f384e57b649722f54dfad34e5b8bcce807ea6073b1671c0" gracePeriod=25
Nov 02 03:43:25.442689 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:25.442656    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-2912e6-5468f5d46f-czlhp"
Nov 02 03:43:25.523662 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:25.523632    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3cf215ca-ff27-4a44-a3a7-7ce21695a0d5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cf215ca-ff27-4a44-a3a7-7ce21695a0d5-plugins-config-volume\") pod \"3cf215ca-ff27-4a44-a3a7-7ce21695a0d5\" (UID: \"3cf215ca-ff27-4a44-a3a7-7ce21695a0d5\") "
Nov 02 03:43:25.523816 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:25.523713    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cf215ca-ff27-4a44-a3a7-7ce21695a0d5-kube-api-access-9pshx\" (UniqueName: \"kubernetes.io/projected/3cf215ca-ff27-4a44-a3a7-7ce21695a0d5-kube-api-access-9pshx\") pod \"3cf215ca-ff27-4a44-a3a7-7ce21695a0d5\" (UID: \"3cf215ca-ff27-4a44-a3a7-7ce21695a0d5\") "
Nov 02 03:43:25.523967 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:25.523943    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cf215ca-ff27-4a44-a3a7-7ce21695a0d5-plugins-config-volume" pod "3cf215ca-ff27-4a44-a3a7-7ce21695a0d5" (UID: "3cf215ca-ff27-4a44-a3a7-7ce21695a0d5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:43:25.524141 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:25.524123    4496 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cf215ca-ff27-4a44-a3a7-7ce21695a0d5-plugins-config-volume\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:43:25.525696 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:25.525676    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cf215ca-ff27-4a44-a3a7-7ce21695a0d5-kube-api-access-9pshx" pod "3cf215ca-ff27-4a44-a3a7-7ce21695a0d5" (UID: "3cf215ca-ff27-4a44-a3a7-7ce21695a0d5"). InnerVolumeSpecName "kube-api-access-9pshx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:43:25.540937 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:25.540912    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2912e6-66887f8d9f-fmjvw"
Nov 02 03:43:25.624915 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:25.624883    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e4bc2e9-17cc-4574-adbc-febeb3977823-kube-api-access-qfnlw\" (UniqueName: \"kubernetes.io/projected/5e4bc2e9-17cc-4574-adbc-febeb3977823-kube-api-access-qfnlw\") pod \"5e4bc2e9-17cc-4574-adbc-febeb3977823\" (UID: \"5e4bc2e9-17cc-4574-adbc-febeb3977823\") "
Nov 02 03:43:25.625067 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:25.624922    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e4bc2e9-17cc-4574-adbc-febeb3977823-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e4bc2e9-17cc-4574-adbc-febeb3977823-plugins-config-volume\") pod \"5e4bc2e9-17cc-4574-adbc-febeb3977823\" (UID: \"5e4bc2e9-17cc-4574-adbc-febeb3977823\") "
Nov 02 03:43:25.625405 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:25.625378    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e4bc2e9-17cc-4574-adbc-febeb3977823-plugins-config-volume" pod "5e4bc2e9-17cc-4574-adbc-febeb3977823" (UID: "5e4bc2e9-17cc-4574-adbc-febeb3977823"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:43:25.625551 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:25.625513    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pshx\" (UniqueName: \"kubernetes.io/projected/3cf215ca-ff27-4a44-a3a7-7ce21695a0d5-kube-api-access-9pshx\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:43:25.625551 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:25.625537    4496 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e4bc2e9-17cc-4574-adbc-febeb3977823-plugins-config-volume\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:43:25.627078 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:25.627052    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e4bc2e9-17cc-4574-adbc-febeb3977823-kube-api-access-qfnlw" pod "5e4bc2e9-17cc-4574-adbc-febeb3977823" (UID: "5e4bc2e9-17cc-4574-adbc-febeb3977823"). InnerVolumeSpecName "kube-api-access-qfnlw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:43:25.727283 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:25.727245    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfnlw\" (UniqueName: \"kubernetes.io/projected/5e4bc2e9-17cc-4574-adbc-febeb3977823-kube-api-access-qfnlw\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:43:25.731777 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:25.731749    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-tpj5j"]
Nov 02 03:43:25.762354 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:25.762315    4496 generic.go:356] "Generic (PLEG): container finished" podID="5e4bc2e9-17cc-4574-adbc-febeb3977823" containerID="edd8d80d28c66dc24f384e57b649722f54dfad34e5b8bcce807ea6073b1671c0" exitCode=0
Nov 02 03:43:25.762538 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:25.762385    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2912e6-66887f8d9f-fmjvw"
Nov 02 03:43:25.762538 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:25.762405    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2912e6-66887f8d9f-fmjvw" event={"ID":"5e4bc2e9-17cc-4574-adbc-febeb3977823","Type":"ContainerDied","Data":"edd8d80d28c66dc24f384e57b649722f54dfad34e5b8bcce807ea6073b1671c0"}
Nov 02 03:43:25.762900 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:25.762873    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2912e6-66887f8d9f-fmjvw" event={"ID":"5e4bc2e9-17cc-4574-adbc-febeb3977823","Type":"ContainerDied","Data":"347379b6204450c5558a67d1208d94a0e90d6fbd17fbeabbb524e8a460e1b828"}
Nov 02 03:43:25.762982 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:25.762928    4496 scope.go:122] "RemoveContainer" containerID="edd8d80d28c66dc24f384e57b649722f54dfad34e5b8bcce807ea6073b1671c0"
Nov 02 03:43:25.764404 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:25.764100    4496 generic.go:356] "Generic (PLEG): container finished" podID="3cf215ca-ff27-4a44-a3a7-7ce21695a0d5" containerID="d6aad296c29cf6935d401db1cc093edbbb6cfe14422b0a35e7ebf93ade4d2571" exitCode=0
Nov 02 03:43:25.764404 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:25.764175    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-2912e6-5468f5d46f-czlhp"
Nov 02 03:43:25.764404 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:25.764272    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-2912e6-5468f5d46f-czlhp" event={"ID":"3cf215ca-ff27-4a44-a3a7-7ce21695a0d5","Type":"ContainerDied","Data":"d6aad296c29cf6935d401db1cc093edbbb6cfe14422b0a35e7ebf93ade4d2571"}
Nov 02 03:43:25.764404 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:25.764314    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-2912e6-5468f5d46f-czlhp" event={"ID":"3cf215ca-ff27-4a44-a3a7-7ce21695a0d5","Type":"ContainerDied","Data":"22b5e301809e9ca8eda75a6677518a8ba8a2ffcb7c82f5b672b249ddc8738441"}
Nov 02 03:43:25.764647 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:25.764554    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-tpj5j" podUID="8ce597fb-f95c-4569-9498-1dda5305706b" containerName="istio-proxy" containerID="cri-o://bf5fabd88371ac16996a9a47b2f5e345b449fa80635798c1fa0283481d4bb783" gracePeriod=30
Nov 02 03:43:25.777469 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:25.777448    4496 scope.go:122] "RemoveContainer" containerID="edd8d80d28c66dc24f384e57b649722f54dfad34e5b8bcce807ea6073b1671c0"
Nov 02 03:43:25.777780 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:43:25.777754    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edd8d80d28c66dc24f384e57b649722f54dfad34e5b8bcce807ea6073b1671c0\": container with ID starting with edd8d80d28c66dc24f384e57b649722f54dfad34e5b8bcce807ea6073b1671c0 not found: ID does not exist" containerID="edd8d80d28c66dc24f384e57b649722f54dfad34e5b8bcce807ea6073b1671c0"
Nov 02 03:43:25.777899 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:25.777787    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edd8d80d28c66dc24f384e57b649722f54dfad34e5b8bcce807ea6073b1671c0"} err="failed to get container status \"edd8d80d28c66dc24f384e57b649722f54dfad34e5b8bcce807ea6073b1671c0\": rpc error: code = NotFound desc = could not find container \"edd8d80d28c66dc24f384e57b649722f54dfad34e5b8bcce807ea6073b1671c0\": container with ID starting with edd8d80d28c66dc24f384e57b649722f54dfad34e5b8bcce807ea6073b1671c0 not found: ID does not exist"
Nov 02 03:43:25.777899 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:25.777809    4496 scope.go:122] "RemoveContainer" containerID="d6aad296c29cf6935d401db1cc093edbbb6cfe14422b0a35e7ebf93ade4d2571"
Nov 02 03:43:25.788243 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:25.788221    4496 scope.go:122] "RemoveContainer" containerID="d6aad296c29cf6935d401db1cc093edbbb6cfe14422b0a35e7ebf93ade4d2571"
Nov 02 03:43:25.788506 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:43:25.788484    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6aad296c29cf6935d401db1cc093edbbb6cfe14422b0a35e7ebf93ade4d2571\": container with ID starting with d6aad296c29cf6935d401db1cc093edbbb6cfe14422b0a35e7ebf93ade4d2571 not found: ID does not exist" containerID="d6aad296c29cf6935d401db1cc093edbbb6cfe14422b0a35e7ebf93ade4d2571"
Nov 02 03:43:25.788565 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:25.788512    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6aad296c29cf6935d401db1cc093edbbb6cfe14422b0a35e7ebf93ade4d2571"} err="failed to get container status \"d6aad296c29cf6935d401db1cc093edbbb6cfe14422b0a35e7ebf93ade4d2571\": rpc error: code = NotFound desc = could not find container \"d6aad296c29cf6935d401db1cc093edbbb6cfe14422b0a35e7ebf93ade4d2571\": container with ID starting with d6aad296c29cf6935d401db1cc093edbbb6cfe14422b0a35e7ebf93ade4d2571 not found: ID does not exist"
Nov 02 03:43:26.046517 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:26.045975    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-2912e6-66887f8d9f-fmjvw"]
Nov 02 03:43:26.050315 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:26.050286    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-2912e6-66887f8d9f-fmjvw"]
Nov 02 03:43:26.159755 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:26.159728    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-2912e6-5468f5d46f-czlhp"]
Nov 02 03:43:26.163557 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:26.163531    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-2912e6-5468f5d46f-czlhp"]
Nov 02 03:43:27.576596 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:27.576560    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3cf215ca-ff27-4a44-a3a7-7ce21695a0d5" path="/var/lib/kubelet/pods/3cf215ca-ff27-4a44-a3a7-7ce21695a0d5/volumes"
Nov 02 03:43:27.577123 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:27.576979    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e4bc2e9-17cc-4574-adbc-febeb3977823" path="/var/lib/kubelet/pods/5e4bc2e9-17cc-4574-adbc-febeb3977823/volumes"
Nov 02 03:43:31.051569 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:31.051507    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-tpj5j"
Nov 02 03:43:31.189978 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:31.189946    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ce597fb-f95c-4569-9498-1dda5305706b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ce597fb-f95c-4569-9498-1dda5305706b-workload-certs\") pod \"8ce597fb-f95c-4569-9498-1dda5305706b\" (UID: \"8ce597fb-f95c-4569-9498-1dda5305706b\") "
Nov 02 03:43:31.190179 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:31.189997    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ce597fb-f95c-4569-9498-1dda5305706b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ce597fb-f95c-4569-9498-1dda5305706b-istiod-ca-cert\") pod \"8ce597fb-f95c-4569-9498-1dda5305706b\" (UID: \"8ce597fb-f95c-4569-9498-1dda5305706b\") "
Nov 02 03:43:31.190179 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:31.190064    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ce597fb-f95c-4569-9498-1dda5305706b-istio-token\" (UniqueName: \"kubernetes.io/projected/8ce597fb-f95c-4569-9498-1dda5305706b-istio-token\") pod \"8ce597fb-f95c-4569-9498-1dda5305706b\" (UID: \"8ce597fb-f95c-4569-9498-1dda5305706b\") "
Nov 02 03:43:31.190179 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:31.190105    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ce597fb-f95c-4569-9498-1dda5305706b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ce597fb-f95c-4569-9498-1dda5305706b-workload-socket\") pod \"8ce597fb-f95c-4569-9498-1dda5305706b\" (UID: \"8ce597fb-f95c-4569-9498-1dda5305706b\") "
Nov 02 03:43:31.190179 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:31.190139    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ce597fb-f95c-4569-9498-1dda5305706b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ce597fb-f95c-4569-9498-1dda5305706b-istio-data\") pod \"8ce597fb-f95c-4569-9498-1dda5305706b\" (UID: \"8ce597fb-f95c-4569-9498-1dda5305706b\") "
Nov 02 03:43:31.190179 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:31.190163    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ce597fb-f95c-4569-9498-1dda5305706b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ce597fb-f95c-4569-9498-1dda5305706b-credential-socket\") pod \"8ce597fb-f95c-4569-9498-1dda5305706b\" (UID: \"8ce597fb-f95c-4569-9498-1dda5305706b\") "
Nov 02 03:43:31.190462 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:31.190182    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ce597fb-f95c-4569-9498-1dda5305706b-kube-api-access-t56jz\" (UniqueName: \"kubernetes.io/projected/8ce597fb-f95c-4569-9498-1dda5305706b-kube-api-access-t56jz\") pod \"8ce597fb-f95c-4569-9498-1dda5305706b\" (UID: \"8ce597fb-f95c-4569-9498-1dda5305706b\") "
Nov 02 03:43:31.190462 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:31.190249    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ce597fb-f95c-4569-9498-1dda5305706b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ce597fb-f95c-4569-9498-1dda5305706b-istio-envoy\") pod \"8ce597fb-f95c-4569-9498-1dda5305706b\" (UID: \"8ce597fb-f95c-4569-9498-1dda5305706b\") "
Nov 02 03:43:31.190462 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:31.190306    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ce597fb-f95c-4569-9498-1dda5305706b-workload-certs" pod "8ce597fb-f95c-4569-9498-1dda5305706b" (UID: "8ce597fb-f95c-4569-9498-1dda5305706b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:43:31.190462 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:31.190378    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8ce597fb-f95c-4569-9498-1dda5305706b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ce597fb-f95c-4569-9498-1dda5305706b-istio-podinfo\") pod \"8ce597fb-f95c-4569-9498-1dda5305706b\" (UID: \"8ce597fb-f95c-4569-9498-1dda5305706b\") "
Nov 02 03:43:31.190691 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:31.190524    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ce597fb-f95c-4569-9498-1dda5305706b-istio-data" pod "8ce597fb-f95c-4569-9498-1dda5305706b" (UID: "8ce597fb-f95c-4569-9498-1dda5305706b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:43:31.190691 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:31.190541    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ce597fb-f95c-4569-9498-1dda5305706b-istiod-ca-cert" pod "8ce597fb-f95c-4569-9498-1dda5305706b" (UID: "8ce597fb-f95c-4569-9498-1dda5305706b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:43:31.190691 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:31.190558    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ce597fb-f95c-4569-9498-1dda5305706b-workload-socket" pod "8ce597fb-f95c-4569-9498-1dda5305706b" (UID: "8ce597fb-f95c-4569-9498-1dda5305706b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:43:31.190691 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:31.190620    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ce597fb-f95c-4569-9498-1dda5305706b-credential-socket" pod "8ce597fb-f95c-4569-9498-1dda5305706b" (UID: "8ce597fb-f95c-4569-9498-1dda5305706b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:43:31.190949 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:31.190926    4496 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ce597fb-f95c-4569-9498-1dda5305706b-workload-certs\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:43:31.191043 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:31.190953    4496 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ce597fb-f95c-4569-9498-1dda5305706b-istiod-ca-cert\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:43:31.191043 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:31.190969    4496 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ce597fb-f95c-4569-9498-1dda5305706b-workload-socket\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:43:31.191043 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:31.190982    4496 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ce597fb-f95c-4569-9498-1dda5305706b-istio-data\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:43:31.191043 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:31.190995    4496 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ce597fb-f95c-4569-9498-1dda5305706b-credential-socket\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:43:31.192520 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:31.192496    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ce597fb-f95c-4569-9498-1dda5305706b-kube-api-access-t56jz" pod "8ce597fb-f95c-4569-9498-1dda5305706b" (UID: "8ce597fb-f95c-4569-9498-1dda5305706b"). InnerVolumeSpecName "kube-api-access-t56jz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:43:31.192624 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:31.192515    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ce597fb-f95c-4569-9498-1dda5305706b-istio-envoy" pod "8ce597fb-f95c-4569-9498-1dda5305706b" (UID: "8ce597fb-f95c-4569-9498-1dda5305706b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:43:31.192624 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:31.192529    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ce597fb-f95c-4569-9498-1dda5305706b-istio-token" pod "8ce597fb-f95c-4569-9498-1dda5305706b" (UID: "8ce597fb-f95c-4569-9498-1dda5305706b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:43:31.192624 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:31.192598    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8ce597fb-f95c-4569-9498-1dda5305706b-istio-podinfo" pod "8ce597fb-f95c-4569-9498-1dda5305706b" (UID: "8ce597fb-f95c-4569-9498-1dda5305706b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 03:43:31.292094 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:31.291992    4496 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ce597fb-f95c-4569-9498-1dda5305706b-istio-token\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:43:31.292094 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:31.292036    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t56jz\" (UniqueName: \"kubernetes.io/projected/8ce597fb-f95c-4569-9498-1dda5305706b-kube-api-access-t56jz\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:43:31.292094 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:31.292052    4496 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ce597fb-f95c-4569-9498-1dda5305706b-istio-envoy\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:43:31.292094 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:31.292064    4496 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ce597fb-f95c-4569-9498-1dda5305706b-istio-podinfo\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:43:31.812570 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:31.812535    4496 generic.go:356] "Generic (PLEG): container finished" podID="8ce597fb-f95c-4569-9498-1dda5305706b" containerID="bf5fabd88371ac16996a9a47b2f5e345b449fa80635798c1fa0283481d4bb783" exitCode=0
Nov 02 03:43:31.812759 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:31.812586    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-tpj5j"
Nov 02 03:43:31.812759 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:31.812614    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-tpj5j" event={"ID":"8ce597fb-f95c-4569-9498-1dda5305706b","Type":"ContainerDied","Data":"bf5fabd88371ac16996a9a47b2f5e345b449fa80635798c1fa0283481d4bb783"}
Nov 02 03:43:31.812759 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:31.812664    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-tpj5j" event={"ID":"8ce597fb-f95c-4569-9498-1dda5305706b","Type":"ContainerDied","Data":"83caa44067cd07e0070e9378d74b7b9d90d89671585194984088f51766f1cfef"}
Nov 02 03:43:31.812759 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:31.812685    4496 scope.go:122] "RemoveContainer" containerID="bf5fabd88371ac16996a9a47b2f5e345b449fa80635798c1fa0283481d4bb783"
Nov 02 03:43:31.822918 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:31.822899    4496 scope.go:122] "RemoveContainer" containerID="bf5fabd88371ac16996a9a47b2f5e345b449fa80635798c1fa0283481d4bb783"
Nov 02 03:43:31.823211 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:43:31.823187    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf5fabd88371ac16996a9a47b2f5e345b449fa80635798c1fa0283481d4bb783\": container with ID starting with bf5fabd88371ac16996a9a47b2f5e345b449fa80635798c1fa0283481d4bb783 not found: ID does not exist" containerID="bf5fabd88371ac16996a9a47b2f5e345b449fa80635798c1fa0283481d4bb783"
Nov 02 03:43:31.823325 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:31.823218    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf5fabd88371ac16996a9a47b2f5e345b449fa80635798c1fa0283481d4bb783"} err="failed to get container status \"bf5fabd88371ac16996a9a47b2f5e345b449fa80635798c1fa0283481d4bb783\": rpc error: code = NotFound desc = could not find container \"bf5fabd88371ac16996a9a47b2f5e345b449fa80635798c1fa0283481d4bb783\": container with ID starting with bf5fabd88371ac16996a9a47b2f5e345b449fa80635798c1fa0283481d4bb783 not found: ID does not exist"
Nov 02 03:43:32.030355 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:32.030324    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-tpj5j"]
Nov 02 03:43:32.081081 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:32.081045    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-tpj5j"]
Nov 02 03:43:33.526603 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:43:33.526569    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 03:43:33.574243 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:33.574213    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ce597fb-f95c-4569-9498-1dda5305706b" path="/var/lib/kubelet/pods/8ce597fb-f95c-4569-9498-1dda5305706b/volumes"
Nov 02 03:43:42.002821 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:42.002790    4496 scope.go:122] "RemoveContainer" containerID="e4e5e80a07d121ebbf791a58e464dd7e84a91c7395fdc447c3cd12492f308045"
Nov 02 03:43:51.865498 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:51.865459    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-praxis-a-1b1008/praxis-route-1b1008-kserve-86f7f4ccbf-w2gnx"]
Nov 02 03:43:51.866049 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:51.866033    4496 cpu_manager.go:414] "containerMap: removing container" podUID="5e4bc2e9-17cc-4574-adbc-febeb3977823" containerName="payload-pre-processing"
Nov 02 03:43:51.866049 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:51.866049    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e4bc2e9-17cc-4574-adbc-febeb3977823" containerName="payload-pre-processing"
Nov 02 03:43:51.866135 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:51.866072    4496 cpu_manager.go:414] "containerMap: removing container" podUID="3cf215ca-ff27-4a44-a3a7-7ce21695a0d5" containerName="payload-processing"
Nov 02 03:43:51.866135 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:51.866077    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3cf215ca-ff27-4a44-a3a7-7ce21695a0d5" containerName="payload-processing"
Nov 02 03:43:51.866135 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:51.866093    4496 cpu_manager.go:414] "containerMap: removing container" podUID="4a600497-7759-4a18-adcd-29c344cba773" containerName="maas-api"
Nov 02 03:43:51.866135 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:51.866098    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a600497-7759-4a18-adcd-29c344cba773" containerName="maas-api"
Nov 02 03:43:51.866135 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:51.866109    4496 cpu_manager.go:414] "containerMap: removing container" podUID="8ce597fb-f95c-4569-9498-1dda5305706b" containerName="istio-proxy"
Nov 02 03:43:51.866135 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:51.866114    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ce597fb-f95c-4569-9498-1dda5305706b" containerName="istio-proxy"
Nov 02 03:43:51.866373 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:51.866201    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="3cf215ca-ff27-4a44-a3a7-7ce21695a0d5" containerName="payload-processing"
Nov 02 03:43:51.866373 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:51.866212    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a600497-7759-4a18-adcd-29c344cba773" containerName="maas-api"
Nov 02 03:43:51.866373 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:51.866219    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ce597fb-f95c-4569-9498-1dda5305706b" containerName="istio-proxy"
Nov 02 03:43:51.866373 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:51.866229    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e4bc2e9-17cc-4574-adbc-febeb3977823" containerName="payload-pre-processing"
Nov 02 03:43:51.869334 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:51.869318    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-1b1008/praxis-route-1b1008-kserve-86f7f4ccbf-w2gnx"
Nov 02 03:43:51.907550 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:51.907513    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9c80fc8-7563-4f48-9ba7-0bf52c371d25-home\") pod \"praxis-route-1b1008-kserve-86f7f4ccbf-w2gnx\" (UID: \"e9c80fc8-7563-4f48-9ba7-0bf52c371d25\") " pod="ai-tenant-e2e-praxis-a-1b1008/praxis-route-1b1008-kserve-86f7f4ccbf-w2gnx"
Nov 02 03:43:51.907725 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:51.907568    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9c80fc8-7563-4f48-9ba7-0bf52c371d25-dshm\") pod \"praxis-route-1b1008-kserve-86f7f4ccbf-w2gnx\" (UID: \"e9c80fc8-7563-4f48-9ba7-0bf52c371d25\") " pod="ai-tenant-e2e-praxis-a-1b1008/praxis-route-1b1008-kserve-86f7f4ccbf-w2gnx"
Nov 02 03:43:51.907725 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:51.907655    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9c80fc8-7563-4f48-9ba7-0bf52c371d25-tls-certs\") pod \"praxis-route-1b1008-kserve-86f7f4ccbf-w2gnx\" (UID: \"e9c80fc8-7563-4f48-9ba7-0bf52c371d25\") " pod="ai-tenant-e2e-praxis-a-1b1008/praxis-route-1b1008-kserve-86f7f4ccbf-w2gnx"
Nov 02 03:43:51.907725 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:51.907697    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9c80fc8-7563-4f48-9ba7-0bf52c371d25-model-cache\") pod \"praxis-route-1b1008-kserve-86f7f4ccbf-w2gnx\" (UID: \"e9c80fc8-7563-4f48-9ba7-0bf52c371d25\") " pod="ai-tenant-e2e-praxis-a-1b1008/praxis-route-1b1008-kserve-86f7f4ccbf-w2gnx"
Nov 02 03:43:51.907925 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:51.907829    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lzff\" (UniqueName: \"kubernetes.io/projected/e9c80fc8-7563-4f48-9ba7-0bf52c371d25-kube-api-access-8lzff\") pod \"praxis-route-1b1008-kserve-86f7f4ccbf-w2gnx\" (UID: \"e9c80fc8-7563-4f48-9ba7-0bf52c371d25\") " pod="ai-tenant-e2e-praxis-a-1b1008/praxis-route-1b1008-kserve-86f7f4ccbf-w2gnx"
Nov 02 03:43:51.907925 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:51.907868    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9c80fc8-7563-4f48-9ba7-0bf52c371d25-tmp-dir\") pod \"praxis-route-1b1008-kserve-86f7f4ccbf-w2gnx\" (UID: \"e9c80fc8-7563-4f48-9ba7-0bf52c371d25\") " pod="ai-tenant-e2e-praxis-a-1b1008/praxis-route-1b1008-kserve-86f7f4ccbf-w2gnx"
Nov 02 03:43:51.925670 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:51.925644    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-1b1008/praxis-route-1b1008-kserve-86f7f4ccbf-w2gnx"]
Nov 02 03:43:51.972657 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:51.972626    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-1b1008\"/\"openshift-service-ca.crt\""
Nov 02 03:43:51.972840 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:51.972801    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-1b1008\"/\"kube-root-ca.crt\""
Nov 02 03:43:51.972933 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:51.972896    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-1b1008\"/\"praxis-route-1b1008-kserve-self-signed-certs\""
Nov 02 03:43:51.973817 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:51.973791    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-1b1008\"/\"default-dockercfg-jhf5b\""
Nov 02 03:43:52.008721 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:52.008682    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lzff\" (UniqueName: \"kubernetes.io/projected/e9c80fc8-7563-4f48-9ba7-0bf52c371d25-kube-api-access-8lzff\") pod \"praxis-route-1b1008-kserve-86f7f4ccbf-w2gnx\" (UID: \"e9c80fc8-7563-4f48-9ba7-0bf52c371d25\") " pod="ai-tenant-e2e-praxis-a-1b1008/praxis-route-1b1008-kserve-86f7f4ccbf-w2gnx"
Nov 02 03:43:52.008915 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:52.008775    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9c80fc8-7563-4f48-9ba7-0bf52c371d25-tmp-dir\") pod \"praxis-route-1b1008-kserve-86f7f4ccbf-w2gnx\" (UID: \"e9c80fc8-7563-4f48-9ba7-0bf52c371d25\") " pod="ai-tenant-e2e-praxis-a-1b1008/praxis-route-1b1008-kserve-86f7f4ccbf-w2gnx"
Nov 02 03:43:52.008915 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:52.008847    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9c80fc8-7563-4f48-9ba7-0bf52c371d25-home\") pod \"praxis-route-1b1008-kserve-86f7f4ccbf-w2gnx\" (UID: \"e9c80fc8-7563-4f48-9ba7-0bf52c371d25\") " pod="ai-tenant-e2e-praxis-a-1b1008/praxis-route-1b1008-kserve-86f7f4ccbf-w2gnx"
Nov 02 03:43:52.008915 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:52.008899    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9c80fc8-7563-4f48-9ba7-0bf52c371d25-dshm\") pod \"praxis-route-1b1008-kserve-86f7f4ccbf-w2gnx\" (UID: \"e9c80fc8-7563-4f48-9ba7-0bf52c371d25\") " pod="ai-tenant-e2e-praxis-a-1b1008/praxis-route-1b1008-kserve-86f7f4ccbf-w2gnx"
Nov 02 03:43:52.009157 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:52.008955    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9c80fc8-7563-4f48-9ba7-0bf52c371d25-tls-certs\") pod \"praxis-route-1b1008-kserve-86f7f4ccbf-w2gnx\" (UID: \"e9c80fc8-7563-4f48-9ba7-0bf52c371d25\") " pod="ai-tenant-e2e-praxis-a-1b1008/praxis-route-1b1008-kserve-86f7f4ccbf-w2gnx"
Nov 02 03:43:52.009157 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:52.009000    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9c80fc8-7563-4f48-9ba7-0bf52c371d25-model-cache\") pod \"praxis-route-1b1008-kserve-86f7f4ccbf-w2gnx\" (UID: \"e9c80fc8-7563-4f48-9ba7-0bf52c371d25\") " pod="ai-tenant-e2e-praxis-a-1b1008/praxis-route-1b1008-kserve-86f7f4ccbf-w2gnx"
Nov 02 03:43:52.009476 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:52.009449    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9c80fc8-7563-4f48-9ba7-0bf52c371d25-home\") pod \"praxis-route-1b1008-kserve-86f7f4ccbf-w2gnx\" (UID: \"e9c80fc8-7563-4f48-9ba7-0bf52c371d25\") " pod="ai-tenant-e2e-praxis-a-1b1008/praxis-route-1b1008-kserve-86f7f4ccbf-w2gnx"
Nov 02 03:43:52.009572 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:52.009547    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9c80fc8-7563-4f48-9ba7-0bf52c371d25-model-cache\") pod \"praxis-route-1b1008-kserve-86f7f4ccbf-w2gnx\" (UID: \"e9c80fc8-7563-4f48-9ba7-0bf52c371d25\") " pod="ai-tenant-e2e-praxis-a-1b1008/praxis-route-1b1008-kserve-86f7f4ccbf-w2gnx"
Nov 02 03:43:52.009618 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:52.009571    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9c80fc8-7563-4f48-9ba7-0bf52c371d25-tmp-dir\") pod \"praxis-route-1b1008-kserve-86f7f4ccbf-w2gnx\" (UID: \"e9c80fc8-7563-4f48-9ba7-0bf52c371d25\") " pod="ai-tenant-e2e-praxis-a-1b1008/praxis-route-1b1008-kserve-86f7f4ccbf-w2gnx"
Nov 02 03:43:52.012274 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:52.012250    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9c80fc8-7563-4f48-9ba7-0bf52c371d25-dshm\") pod \"praxis-route-1b1008-kserve-86f7f4ccbf-w2gnx\" (UID: \"e9c80fc8-7563-4f48-9ba7-0bf52c371d25\") " pod="ai-tenant-e2e-praxis-a-1b1008/praxis-route-1b1008-kserve-86f7f4ccbf-w2gnx"
Nov 02 03:43:52.012570 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:52.012540    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9c80fc8-7563-4f48-9ba7-0bf52c371d25-tls-certs\") pod \"praxis-route-1b1008-kserve-86f7f4ccbf-w2gnx\" (UID: \"e9c80fc8-7563-4f48-9ba7-0bf52c371d25\") " pod="ai-tenant-e2e-praxis-a-1b1008/praxis-route-1b1008-kserve-86f7f4ccbf-w2gnx"
Nov 02 03:43:52.068598 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:52.068564    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lzff\" (UniqueName: \"kubernetes.io/projected/e9c80fc8-7563-4f48-9ba7-0bf52c371d25-kube-api-access-8lzff\") pod \"praxis-route-1b1008-kserve-86f7f4ccbf-w2gnx\" (UID: \"e9c80fc8-7563-4f48-9ba7-0bf52c371d25\") " pod="ai-tenant-e2e-praxis-a-1b1008/praxis-route-1b1008-kserve-86f7f4ccbf-w2gnx"
Nov 02 03:43:52.182413 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:52.182321    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-1b1008/praxis-route-1b1008-kserve-86f7f4ccbf-w2gnx"
Nov 02 03:43:52.564679 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:52.564649    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-1b1008/praxis-route-1b1008-kserve-86f7f4ccbf-w2gnx"]
Nov 02 03:43:52.565328 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:43:52.565297    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode9c80fc8_7563_4f48_9ba7_0bf52c371d25.slice/crio-24e4b2622cda6a8049a0b31158a5df3ce548b00524be9a6f83c750b0b5d23fb6 WatchSource:0}: Error finding container 24e4b2622cda6a8049a0b31158a5df3ce548b00524be9a6f83c750b0b5d23fb6: Status 404 returned error can't find the container with id 24e4b2622cda6a8049a0b31158a5df3ce548b00524be9a6f83c750b0b5d23fb6
Nov 02 03:43:52.967142 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:52.967055    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-1b1008/praxis-route-1b1008-kserve-86f7f4ccbf-w2gnx" event={"ID":"e9c80fc8-7563-4f48-9ba7-0bf52c371d25","Type":"ContainerStarted","Data":"83d38862a811fef3724e2b67341c77fc4502c61e1b54efdc4f8026caa922969b"}
Nov 02 03:43:52.967142 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:52.967105    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-1b1008/praxis-route-1b1008-kserve-86f7f4ccbf-w2gnx" event={"ID":"e9c80fc8-7563-4f48-9ba7-0bf52c371d25","Type":"ContainerStarted","Data":"24e4b2622cda6a8049a0b31158a5df3ce548b00524be9a6f83c750b0b5d23fb6"}
Nov 02 03:43:53.087945 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:53.087884    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-praxis-a-1b1008/praxis-route-1b1008-kserve-86f7f4ccbf-w2gnx" podStartSLOduration=2.08786695 podStartE2EDuration="2.08786695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:43:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:43:53.085720336 +0000 UTC m=+3373.664327717" watchObservedRunningTime="2026-11-02 03:43:53.08786695 +0000 UTC m=+3373.666474330"
Nov 02 03:43:53.182794 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:53.182759    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-praxis-a-1b1008/praxis-route-1b1008-kserve-86f7f4ccbf-w2gnx"
Nov 02 03:43:53.299376 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:53.299342    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-2912e6-55ccd89d6-w6hh7"]
Nov 02 03:43:53.299565 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:53.299539    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2912e6-55ccd89d6-w6hh7" podUID="305183a6-4694-4195-98e1-a0a8a5deaf51" containerName="maas-api" containerID="cri-o://b68ed305e4757636be099f6d212f6933e9475b184cb01cf57b9abe9a58754f15" gracePeriod=30
Nov 02 03:43:53.600031 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:53.599991    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2912e6-55ccd89d6-w6hh7"
Nov 02 03:43:53.735993 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:53.735957    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/305183a6-4694-4195-98e1-a0a8a5deaf51-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/305183a6-4694-4195-98e1-a0a8a5deaf51-openshift-service-ca\") pod \"305183a6-4694-4195-98e1-a0a8a5deaf51\" (UID: \"305183a6-4694-4195-98e1-a0a8a5deaf51\") "
Nov 02 03:43:53.736237 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:53.736110    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/305183a6-4694-4195-98e1-a0a8a5deaf51-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/305183a6-4694-4195-98e1-a0a8a5deaf51-maas-api-tls\") pod \"305183a6-4694-4195-98e1-a0a8a5deaf51\" (UID: \"305183a6-4694-4195-98e1-a0a8a5deaf51\") "
Nov 02 03:43:53.737155 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:53.737124    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/305183a6-4694-4195-98e1-a0a8a5deaf51-openshift-service-ca" pod "305183a6-4694-4195-98e1-a0a8a5deaf51" (UID: "305183a6-4694-4195-98e1-a0a8a5deaf51"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:43:53.737350 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:53.736228    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/305183a6-4694-4195-98e1-a0a8a5deaf51-metrics-certs\" (UniqueName: \"kubernetes.io/secret/305183a6-4694-4195-98e1-a0a8a5deaf51-metrics-certs\") pod \"305183a6-4694-4195-98e1-a0a8a5deaf51\" (UID: \"305183a6-4694-4195-98e1-a0a8a5deaf51\") "
Nov 02 03:43:53.737502 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:53.737486    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/305183a6-4694-4195-98e1-a0a8a5deaf51-kube-api-access-7bn6s\" (UniqueName: \"kubernetes.io/projected/305183a6-4694-4195-98e1-a0a8a5deaf51-kube-api-access-7bn6s\") pod \"305183a6-4694-4195-98e1-a0a8a5deaf51\" (UID: \"305183a6-4694-4195-98e1-a0a8a5deaf51\") "
Nov 02 03:43:53.738157 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:53.738133    4496 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/305183a6-4694-4195-98e1-a0a8a5deaf51-openshift-service-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:43:53.744974 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:53.740934    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/305183a6-4694-4195-98e1-a0a8a5deaf51-kube-api-access-7bn6s" pod "305183a6-4694-4195-98e1-a0a8a5deaf51" (UID: "305183a6-4694-4195-98e1-a0a8a5deaf51"). InnerVolumeSpecName "kube-api-access-7bn6s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:43:53.744974 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:53.743221    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/305183a6-4694-4195-98e1-a0a8a5deaf51-maas-api-tls" pod "305183a6-4694-4195-98e1-a0a8a5deaf51" (UID: "305183a6-4694-4195-98e1-a0a8a5deaf51"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 03:43:53.746547 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:53.746515    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/305183a6-4694-4195-98e1-a0a8a5deaf51-metrics-certs" pod "305183a6-4694-4195-98e1-a0a8a5deaf51" (UID: "305183a6-4694-4195-98e1-a0a8a5deaf51"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 03:43:53.839245 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:53.839142    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/305183a6-4694-4195-98e1-a0a8a5deaf51-maas-api-tls\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:43:53.839245 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:53.839177    4496 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/305183a6-4694-4195-98e1-a0a8a5deaf51-metrics-certs\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:43:53.839245 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:53.839192    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bn6s\" (UniqueName: \"kubernetes.io/projected/305183a6-4694-4195-98e1-a0a8a5deaf51-kube-api-access-7bn6s\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:43:53.976788 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:53.976750    4496 generic.go:356] "Generic (PLEG): container finished" podID="305183a6-4694-4195-98e1-a0a8a5deaf51" containerID="b68ed305e4757636be099f6d212f6933e9475b184cb01cf57b9abe9a58754f15" exitCode=0
Nov 02 03:43:53.976788 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:53.976802    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2912e6-55ccd89d6-w6hh7"
Nov 02 03:43:53.977385 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:53.976842    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2912e6-55ccd89d6-w6hh7" event={"ID":"305183a6-4694-4195-98e1-a0a8a5deaf51","Type":"ContainerDied","Data":"b68ed305e4757636be099f6d212f6933e9475b184cb01cf57b9abe9a58754f15"}
Nov 02 03:43:53.977385 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:53.976910    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2912e6-55ccd89d6-w6hh7" event={"ID":"305183a6-4694-4195-98e1-a0a8a5deaf51","Type":"ContainerDied","Data":"e8ed5a48fd584a2d53b98aed44c989636ca768ba967ecc8a5ce3ed7bad5008c5"}
Nov 02 03:43:53.977385 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:53.976940    4496 scope.go:122] "RemoveContainer" containerID="b68ed305e4757636be099f6d212f6933e9475b184cb01cf57b9abe9a58754f15"
Nov 02 03:43:53.989250 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:53.989228    4496 scope.go:122] "RemoveContainer" containerID="b68ed305e4757636be099f6d212f6933e9475b184cb01cf57b9abe9a58754f15"
Nov 02 03:43:53.989552 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:43:53.989525    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b68ed305e4757636be099f6d212f6933e9475b184cb01cf57b9abe9a58754f15\": container with ID starting with b68ed305e4757636be099f6d212f6933e9475b184cb01cf57b9abe9a58754f15 not found: ID does not exist" containerID="b68ed305e4757636be099f6d212f6933e9475b184cb01cf57b9abe9a58754f15"
Nov 02 03:43:53.989630 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:53.989559    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b68ed305e4757636be099f6d212f6933e9475b184cb01cf57b9abe9a58754f15"} err="failed to get container status \"b68ed305e4757636be099f6d212f6933e9475b184cb01cf57b9abe9a58754f15\": rpc error: code = NotFound desc = could not find container \"b68ed305e4757636be099f6d212f6933e9475b184cb01cf57b9abe9a58754f15\": container with ID starting with b68ed305e4757636be099f6d212f6933e9475b184cb01cf57b9abe9a58754f15 not found: ID does not exist"
Nov 02 03:43:54.147662 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:54.147559    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-2912e6-55ccd89d6-w6hh7"]
Nov 02 03:43:54.152325 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:54.152292    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-2912e6-55ccd89d6-w6hh7"]
Nov 02 03:43:55.575128 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:43:55.575094    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="305183a6-4694-4195-98e1-a0a8a5deaf51" path="/var/lib/kubelet/pods/305183a6-4694-4195-98e1-a0a8a5deaf51/volumes"
Nov 02 03:44:02.182992 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:44:02.182953    4496 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-praxis-a-1b1008/praxis-route-1b1008-kserve-86f7f4ccbf-w2gnx"
Nov 02 03:44:02.195393 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:44:02.195365    4496 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-praxis-a-1b1008/praxis-route-1b1008-kserve-86f7f4ccbf-w2gnx"
Nov 02 03:44:03.058249 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:44:03.058211    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-praxis-a-1b1008/praxis-route-1b1008-kserve-86f7f4ccbf-w2gnx"
Nov 02 03:44:03.527045 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:44:03.526978    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 03:44:24.847340 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:44:24.847297    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-9b06f8-56944cd46c-rgw5t"]
Nov 02 03:44:24.847866 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:44:24.847535    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9b06f8-56944cd46c-rgw5t" podUID="7801d774-3dc0-4f41-ae02-ad732b765da7" containerName="maas-api" containerID="cri-o://fb6212fd410ac9e24c1bb509b001d867b16c7fb7672f770da176db0f7f9fadb1" gracePeriod=30
Nov 02 03:44:25.125008 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:44:25.124927    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9b06f8-56944cd46c-rgw5t"
Nov 02 03:44:25.198526 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:44:25.198494    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7801d774-3dc0-4f41-ae02-ad732b765da7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7801d774-3dc0-4f41-ae02-ad732b765da7-maas-api-tls\") pod \"7801d774-3dc0-4f41-ae02-ad732b765da7\" (UID: \"7801d774-3dc0-4f41-ae02-ad732b765da7\") "
Nov 02 03:44:25.198711 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:44:25.198562    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7801d774-3dc0-4f41-ae02-ad732b765da7-kube-api-access-rpxrn\" (UniqueName: \"kubernetes.io/projected/7801d774-3dc0-4f41-ae02-ad732b765da7-kube-api-access-rpxrn\") pod \"7801d774-3dc0-4f41-ae02-ad732b765da7\" (UID: \"7801d774-3dc0-4f41-ae02-ad732b765da7\") "
Nov 02 03:44:25.198711 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:44:25.198594    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7801d774-3dc0-4f41-ae02-ad732b765da7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7801d774-3dc0-4f41-ae02-ad732b765da7-metrics-certs\") pod \"7801d774-3dc0-4f41-ae02-ad732b765da7\" (UID: \"7801d774-3dc0-4f41-ae02-ad732b765da7\") "
Nov 02 03:44:25.198862 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:44:25.198734    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7801d774-3dc0-4f41-ae02-ad732b765da7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7801d774-3dc0-4f41-ae02-ad732b765da7-openshift-service-ca\") pod \"7801d774-3dc0-4f41-ae02-ad732b765da7\" (UID: \"7801d774-3dc0-4f41-ae02-ad732b765da7\") "
Nov 02 03:44:25.199224 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:44:25.199188    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7801d774-3dc0-4f41-ae02-ad732b765da7-openshift-service-ca" pod "7801d774-3dc0-4f41-ae02-ad732b765da7" (UID: "7801d774-3dc0-4f41-ae02-ad732b765da7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:44:25.200869 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:44:25.200841    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7801d774-3dc0-4f41-ae02-ad732b765da7-kube-api-access-rpxrn" pod "7801d774-3dc0-4f41-ae02-ad732b765da7" (UID: "7801d774-3dc0-4f41-ae02-ad732b765da7"). InnerVolumeSpecName "kube-api-access-rpxrn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:44:25.201253 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:44:25.201230    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7801d774-3dc0-4f41-ae02-ad732b765da7-maas-api-tls" pod "7801d774-3dc0-4f41-ae02-ad732b765da7" (UID: "7801d774-3dc0-4f41-ae02-ad732b765da7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 03:44:25.202138 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:44:25.202115    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7801d774-3dc0-4f41-ae02-ad732b765da7-metrics-certs" pod "7801d774-3dc0-4f41-ae02-ad732b765da7" (UID: "7801d774-3dc0-4f41-ae02-ad732b765da7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 03:44:25.203527 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:44:25.203505    4496 generic.go:356] "Generic (PLEG): container finished" podID="7801d774-3dc0-4f41-ae02-ad732b765da7" containerID="fb6212fd410ac9e24c1bb509b001d867b16c7fb7672f770da176db0f7f9fadb1" exitCode=0
Nov 02 03:44:25.203608 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:44:25.203559    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9b06f8-56944cd46c-rgw5t"
Nov 02 03:44:25.203608 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:44:25.203580    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9b06f8-56944cd46c-rgw5t" event={"ID":"7801d774-3dc0-4f41-ae02-ad732b765da7","Type":"ContainerDied","Data":"fb6212fd410ac9e24c1bb509b001d867b16c7fb7672f770da176db0f7f9fadb1"}
Nov 02 03:44:25.203685 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:44:25.203631    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9b06f8-56944cd46c-rgw5t" event={"ID":"7801d774-3dc0-4f41-ae02-ad732b765da7","Type":"ContainerDied","Data":"ecb7655dfa158969b27665bb13efd2de33ad11168d59dee429045aace9fdde90"}
Nov 02 03:44:25.203685 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:44:25.203653    4496 scope.go:122] "RemoveContainer" containerID="fb6212fd410ac9e24c1bb509b001d867b16c7fb7672f770da176db0f7f9fadb1"
Nov 02 03:44:25.234138 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:44:25.234111    4496 scope.go:122] "RemoveContainer" containerID="fb6212fd410ac9e24c1bb509b001d867b16c7fb7672f770da176db0f7f9fadb1"
Nov 02 03:44:25.234523 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:44:25.234493    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb6212fd410ac9e24c1bb509b001d867b16c7fb7672f770da176db0f7f9fadb1\": container with ID starting with fb6212fd410ac9e24c1bb509b001d867b16c7fb7672f770da176db0f7f9fadb1 not found: ID does not exist" containerID="fb6212fd410ac9e24c1bb509b001d867b16c7fb7672f770da176db0f7f9fadb1"
Nov 02 03:44:25.234722 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:44:25.234534    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb6212fd410ac9e24c1bb509b001d867b16c7fb7672f770da176db0f7f9fadb1"} err="failed to get container status \"fb6212fd410ac9e24c1bb509b001d867b16c7fb7672f770da176db0f7f9fadb1\": rpc error: code = NotFound desc = could not find container \"fb6212fd410ac9e24c1bb509b001d867b16c7fb7672f770da176db0f7f9fadb1\": container with ID starting with fb6212fd410ac9e24c1bb509b001d867b16c7fb7672f770da176db0f7f9fadb1 not found: ID does not exist"
Nov 02 03:44:25.300421 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:44:25.300388    4496 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7801d774-3dc0-4f41-ae02-ad732b765da7-openshift-service-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:44:25.300421 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:44:25.300415    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7801d774-3dc0-4f41-ae02-ad732b765da7-maas-api-tls\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:44:25.300421 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:44:25.300424    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpxrn\" (UniqueName: \"kubernetes.io/projected/7801d774-3dc0-4f41-ae02-ad732b765da7-kube-api-access-rpxrn\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:44:25.300660 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:44:25.300434    4496 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7801d774-3dc0-4f41-ae02-ad732b765da7-metrics-certs\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:44:25.354372 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:44:25.354315    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-9b06f8-56944cd46c-rgw5t"]
Nov 02 03:44:25.357237 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:44:25.357215    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-9b06f8-56944cd46c-rgw5t"]
Nov 02 03:44:25.574579 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:44:25.574543    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7801d774-3dc0-4f41-ae02-ad732b765da7" path="/var/lib/kubelet/pods/7801d774-3dc0-4f41-ae02-ad732b765da7/volumes"
Nov 02 03:44:33.526690 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:44:33.526644    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 03:44:42.056913 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:44:42.056873    4496 scope.go:122] "RemoveContainer" containerID="66861e8fe715e9eaa2899376320afe16e33c3b947bdf886aa8124486ef78362e"
Nov 02 03:44:42.067881 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:44:42.067858    4496 scope.go:122] "RemoveContainer" containerID="42da0b367f9a66076cf07827b82ce898f019f3ec6f0131275503cbe3cc9b05d7"
Nov 02 03:44:46.981093 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:44:46.981053    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xzskx"]
Nov 02 03:44:46.981856 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:44:46.981832    4496 cpu_manager.go:414] "containerMap: removing container" podUID="7801d774-3dc0-4f41-ae02-ad732b765da7" containerName="maas-api"
Nov 02 03:44:46.981961 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:44:46.981855    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7801d774-3dc0-4f41-ae02-ad732b765da7" containerName="maas-api"
Nov 02 03:44:46.981961 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:44:46.981877    4496 cpu_manager.go:414] "containerMap: removing container" podUID="305183a6-4694-4195-98e1-a0a8a5deaf51" containerName="maas-api"
Nov 02 03:44:46.981961 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:44:46.981884    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="305183a6-4694-4195-98e1-a0a8a5deaf51" containerName="maas-api"
Nov 02 03:44:46.982186 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:44:46.982065    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="7801d774-3dc0-4f41-ae02-ad732b765da7" containerName="maas-api"
Nov 02 03:44:46.982186 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:44:46.982084    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="305183a6-4694-4195-98e1-a0a8a5deaf51" containerName="maas-api"
Nov 02 03:44:46.985523 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:44:46.985500    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xzskx"
Nov 02 03:44:47.038140 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:44:47.038112    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19b510d7-4f25-4f4a-8c61-ce8a6013229a-workload-certs\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xzskx\" (UID: \"19b510d7-4f25-4f4a-8c61-ce8a6013229a\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xzskx"
Nov 02 03:44:47.038268 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:44:47.038152    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19b510d7-4f25-4f4a-8c61-ce8a6013229a-credential-socket\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xzskx\" (UID: \"19b510d7-4f25-4f4a-8c61-ce8a6013229a\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xzskx"
Nov 02 03:44:47.038268 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:44:47.038181    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19b510d7-4f25-4f4a-8c61-ce8a6013229a-istio-envoy\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xzskx\" (UID: \"19b510d7-4f25-4f4a-8c61-ce8a6013229a\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xzskx"
Nov 02 03:44:47.038367 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:44:47.038299    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqrl5\" (UniqueName: \"kubernetes.io/projected/19b510d7-4f25-4f4a-8c61-ce8a6013229a-kube-api-access-lqrl5\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xzskx\" (UID: \"19b510d7-4f25-4f4a-8c61-ce8a6013229a\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xzskx"
Nov 02 03:44:47.038367 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:44:47.038352    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19b510d7-4f25-4f4a-8c61-ce8a6013229a-istio-podinfo\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xzskx\" (UID: \"19b510d7-4f25-4f4a-8c61-ce8a6013229a\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xzskx"
Nov 02 03:44:47.038484 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:44:47.038459    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19b510d7-4f25-4f4a-8c61-ce8a6013229a-istiod-ca-cert\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xzskx\" (UID: \"19b510d7-4f25-4f4a-8c61-ce8a6013229a\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xzskx"
Nov 02 03:44:47.038565 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:44:47.038549    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19b510d7-4f25-4f4a-8c61-ce8a6013229a-workload-socket\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xzskx\" (UID: \"19b510d7-4f25-4f4a-8c61-ce8a6013229a\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xzskx"
Nov 02 03:44:47.038612 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:44:47.038579    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19b510d7-4f25-4f4a-8c61-ce8a6013229a-istio-data\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xzskx\" (UID: \"19b510d7-4f25-4f4a-8c61-ce8a6013229a\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xzskx"
Nov 02 03:44:47.038612 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:44:47.038595    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19b510d7-4f25-4f4a-8c61-ce8a6013229a-istio-token\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xzskx\" (UID: \"19b510d7-4f25-4f4a-8c61-ce8a6013229a\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xzskx"
Nov 02 03:44:47.045717 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:44:47.045690    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xzskx"]
Nov 02 03:44:47.091611 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:44:47.091583    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-5ac93f-openshift-default-dockercfg-cjnld\""
Nov 02 03:44:47.140009 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:44:47.139963    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19b510d7-4f25-4f4a-8c61-ce8a6013229a-istiod-ca-cert\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xzskx\" (UID: \"19b510d7-4f25-4f4a-8c61-ce8a6013229a\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xzskx"
Nov 02 03:44:47.140214 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:44:47.140062    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19b510d7-4f25-4f4a-8c61-ce8a6013229a-workload-socket\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xzskx\" (UID: \"19b510d7-4f25-4f4a-8c61-ce8a6013229a\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xzskx"
Nov 02 03:44:47.140214 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:44:47.140089    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19b510d7-4f25-4f4a-8c61-ce8a6013229a-istio-data\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xzskx\" (UID: \"19b510d7-4f25-4f4a-8c61-ce8a6013229a\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xzskx"
Nov 02 03:44:47.140214 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:44:47.140107    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19b510d7-4f25-4f4a-8c61-ce8a6013229a-istio-token\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xzskx\" (UID: \"19b510d7-4f25-4f4a-8c61-ce8a6013229a\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xzskx"
Nov 02 03:44:47.140214 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:44:47.140173    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19b510d7-4f25-4f4a-8c61-ce8a6013229a-workload-certs\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xzskx\" (UID: \"19b510d7-4f25-4f4a-8c61-ce8a6013229a\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xzskx"
Nov 02 03:44:47.140214 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:44:47.140208    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19b510d7-4f25-4f4a-8c61-ce8a6013229a-credential-socket\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xzskx\" (UID: \"19b510d7-4f25-4f4a-8c61-ce8a6013229a\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xzskx"
Nov 02 03:44:47.140489 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:44:47.140242    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19b510d7-4f25-4f4a-8c61-ce8a6013229a-istio-envoy\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xzskx\" (UID: \"19b510d7-4f25-4f4a-8c61-ce8a6013229a\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xzskx"
Nov 02 03:44:47.140489 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:44:47.140299    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqrl5\" (UniqueName: \"kubernetes.io/projected/19b510d7-4f25-4f4a-8c61-ce8a6013229a-kube-api-access-lqrl5\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xzskx\" (UID: \"19b510d7-4f25-4f4a-8c61-ce8a6013229a\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xzskx"
Nov 02 03:44:47.140489 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:44:47.140357    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19b510d7-4f25-4f4a-8c61-ce8a6013229a-istio-podinfo\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xzskx\" (UID: \"19b510d7-4f25-4f4a-8c61-ce8a6013229a\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xzskx"
Nov 02 03:44:47.140668 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:44:47.140494    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19b510d7-4f25-4f4a-8c61-ce8a6013229a-workload-socket\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xzskx\" (UID: \"19b510d7-4f25-4f4a-8c61-ce8a6013229a\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xzskx"
Nov 02 03:44:47.140746 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:44:47.140720    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19b510d7-4f25-4f4a-8c61-ce8a6013229a-istio-data\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xzskx\" (UID: \"19b510d7-4f25-4f4a-8c61-ce8a6013229a\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xzskx"
Nov 02 03:44:47.140808 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:44:47.140743    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19b510d7-4f25-4f4a-8c61-ce8a6013229a-workload-certs\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xzskx\" (UID: \"19b510d7-4f25-4f4a-8c61-ce8a6013229a\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xzskx"
Nov 02 03:44:47.140808 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:44:47.140783    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19b510d7-4f25-4f4a-8c61-ce8a6013229a-istiod-ca-cert\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xzskx\" (UID: \"19b510d7-4f25-4f4a-8c61-ce8a6013229a\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xzskx"
Nov 02 03:44:47.140930 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:44:47.140837    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19b510d7-4f25-4f4a-8c61-ce8a6013229a-credential-socket\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xzskx\" (UID: \"19b510d7-4f25-4f4a-8c61-ce8a6013229a\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xzskx"
Nov 02 03:44:47.142580 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:44:47.142552    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19b510d7-4f25-4f4a-8c61-ce8a6013229a-istio-envoy\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xzskx\" (UID: \"19b510d7-4f25-4f4a-8c61-ce8a6013229a\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xzskx"
Nov 02 03:44:47.142861 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:44:47.142843    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19b510d7-4f25-4f4a-8c61-ce8a6013229a-istio-podinfo\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xzskx\" (UID: \"19b510d7-4f25-4f4a-8c61-ce8a6013229a\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xzskx"
Nov 02 03:44:47.198468 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:44:47.198432    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19b510d7-4f25-4f4a-8c61-ce8a6013229a-istio-token\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xzskx\" (UID: \"19b510d7-4f25-4f4a-8c61-ce8a6013229a\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xzskx"
Nov 02 03:44:47.198620 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:44:47.198492    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqrl5\" (UniqueName: \"kubernetes.io/projected/19b510d7-4f25-4f4a-8c61-ce8a6013229a-kube-api-access-lqrl5\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xzskx\" (UID: \"19b510d7-4f25-4f4a-8c61-ce8a6013229a\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xzskx"
Nov 02 03:44:47.298758 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:44:47.298654    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xzskx"
Nov 02 03:44:47.728282 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:44:47.728214    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xzskx"]
Nov 02 03:44:47.729031 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:44:47.728993    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19b510d7_4f25_4f4a_8c61_ce8a6013229a.slice/crio-ca115b2434ad9b7f2e66e3a2faba29c70b695fc3d0324530103ee62b3a9fc44c WatchSource:0}: Error finding container ca115b2434ad9b7f2e66e3a2faba29c70b695fc3d0324530103ee62b3a9fc44c: Status 404 returned error can't find the container with id ca115b2434ad9b7f2e66e3a2faba29c70b695fc3d0324530103ee62b3a9fc44c
Nov 02 03:44:47.731464 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:44:47.731431    4496 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 03:44:47.731541 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:44:47.731499    4496 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 03:44:47.731541 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:44:47.731529    4496 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 03:44:48.364602 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:44:48.364562    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xzskx" event={"ID":"19b510d7-4f25-4f4a-8c61-ce8a6013229a","Type":"ContainerStarted","Data":"949d5401aba010bb77183a09e430b5ffe788b614f8dc1b2e9086b598df6ed780"}
Nov 02 03:44:48.365143 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:44:48.364645    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xzskx" event={"ID":"19b510d7-4f25-4f4a-8c61-ce8a6013229a","Type":"ContainerStarted","Data":"ca115b2434ad9b7f2e66e3a2faba29c70b695fc3d0324530103ee62b3a9fc44c"}
Nov 02 03:44:48.483573 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:44:48.483510    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xzskx" podStartSLOduration=2.4834911809999998 podStartE2EDuration="2.483491181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:44:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:44:48.482073445 +0000 UTC m=+3429.060680822" watchObservedRunningTime="2026-11-02 03:44:48.483491181 +0000 UTC m=+3429.062098562"
Nov 02 03:44:49.299722 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:44:49.299681    4496 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xzskx"
Nov 02 03:44:49.301486 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:44:49.301459    4496 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xzskx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.247:15021/healthz/ready\": dial tcp 10.134.0.247:15021: connect: connection refused" start-of-body=
Nov 02 03:44:49.301633 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:44:49.301508    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xzskx" podUID="19b510d7-4f25-4f4a-8c61-ce8a6013229a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.247:15021/healthz/ready\": dial tcp 10.134.0.247:15021: connect: connection refused"
Nov 02 03:44:50.299812 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:44:50.299768    4496 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xzskx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.247:15021/healthz/ready\": dial tcp 10.134.0.247:15021: connect: connection refused" start-of-body=
Nov 02 03:44:50.300257 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:44:50.299845    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xzskx" podUID="19b510d7-4f25-4f4a-8c61-ce8a6013229a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.247:15021/healthz/ready\": dial tcp 10.134.0.247:15021: connect: connection refused"
Nov 02 03:44:51.299399 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:44:51.299360    4496 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xzskx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.247:15021/healthz/ready\": dial tcp 10.134.0.247:15021: connect: connection refused" start-of-body=
Nov 02 03:44:51.299595 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:44:51.299425    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xzskx" podUID="19b510d7-4f25-4f4a-8c61-ce8a6013229a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.247:15021/healthz/ready\": dial tcp 10.134.0.247:15021: connect: connection refused"
Nov 02 03:44:52.298988 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:44:52.298950    4496 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xzskx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.247:15021/healthz/ready\": dial tcp 10.134.0.247:15021: connect: connection refused" start-of-body=
Nov 02 03:44:52.299414 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:44:52.299037    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xzskx" podUID="19b510d7-4f25-4f4a-8c61-ce8a6013229a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.247:15021/healthz/ready\": dial tcp 10.134.0.247:15021: connect: connection refused"
Nov 02 03:44:53.298992 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:44:53.298956    4496 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xzskx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.247:15021/healthz/ready\": dial tcp 10.134.0.247:15021: connect: connection refused" start-of-body=
Nov 02 03:44:53.299435 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:44:53.299048    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xzskx" podUID="19b510d7-4f25-4f4a-8c61-ce8a6013229a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.247:15021/healthz/ready\": dial tcp 10.134.0.247:15021: connect: connection refused"
Nov 02 03:44:54.299448 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:44:54.299399    4496 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xzskx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.247:15021/healthz/ready\": dial tcp 10.134.0.247:15021: connect: connection refused" start-of-body=
Nov 02 03:44:54.299823 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:44:54.299470    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xzskx" podUID="19b510d7-4f25-4f4a-8c61-ce8a6013229a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.247:15021/healthz/ready\": dial tcp 10.134.0.247:15021: connect: connection refused"
Nov 02 03:44:55.142007 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:44:55.141968    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-ad96f4-5487969c88-xt82w"]
Nov 02 03:44:55.142324 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:44:55.142273    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ad96f4-5487969c88-xt82w" podUID="b335eb4b-7295-45f3-9dd4-637e3c7c9c58" containerName="maas-api" containerID="cri-o://75ebd9cdace4ab091697af66db848dc3813d027ce4655d706658839f2406d4b2" gracePeriod=30
Nov 02 03:44:55.303658 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:44:55.303509    4496 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xzskx"
Nov 02 03:44:55.304305 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:44:55.304252    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xzskx"
Nov 02 03:44:55.305757 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:44:55.304893    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xzskx"
Nov 02 03:44:55.398244 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:44:55.398179    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ad96f4-5487969c88-xt82w"
Nov 02 03:44:55.419652 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:44:55.419611    4496 generic.go:356] "Generic (PLEG): container finished" podID="b335eb4b-7295-45f3-9dd4-637e3c7c9c58" containerID="75ebd9cdace4ab091697af66db848dc3813d027ce4655d706658839f2406d4b2" exitCode=0
Nov 02 03:44:55.420067 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:44:55.420043    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ad96f4-5487969c88-xt82w"
Nov 02 03:44:55.420208 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:44:55.420176    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ad96f4-5487969c88-xt82w" event={"ID":"b335eb4b-7295-45f3-9dd4-637e3c7c9c58","Type":"ContainerDied","Data":"75ebd9cdace4ab091697af66db848dc3813d027ce4655d706658839f2406d4b2"}
Nov 02 03:44:55.420286 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:44:55.420219    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ad96f4-5487969c88-xt82w" event={"ID":"b335eb4b-7295-45f3-9dd4-637e3c7c9c58","Type":"ContainerDied","Data":"49cf61ee1bbf096e3d857df03d94ac613332232769c15b38aaa1c1fef7fa2c68"}
Nov 02 03:44:55.420286 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:44:55.420250    4496 scope.go:122] "RemoveContainer" containerID="75ebd9cdace4ab091697af66db848dc3813d027ce4655d706658839f2406d4b2"
Nov 02 03:44:55.431331 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:44:55.431309    4496 scope.go:122] "RemoveContainer" containerID="75ebd9cdace4ab091697af66db848dc3813d027ce4655d706658839f2406d4b2"
Nov 02 03:44:55.431630 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:44:55.431601    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75ebd9cdace4ab091697af66db848dc3813d027ce4655d706658839f2406d4b2\": container with ID starting with 75ebd9cdace4ab091697af66db848dc3813d027ce4655d706658839f2406d4b2 not found: ID does not exist" containerID="75ebd9cdace4ab091697af66db848dc3813d027ce4655d706658839f2406d4b2"
Nov 02 03:44:55.431697 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:44:55.431633    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75ebd9cdace4ab091697af66db848dc3813d027ce4655d706658839f2406d4b2"} err="failed to get container status \"75ebd9cdace4ab091697af66db848dc3813d027ce4655d706658839f2406d4b2\": rpc error: code = NotFound desc = could not find container \"75ebd9cdace4ab091697af66db848dc3813d027ce4655d706658839f2406d4b2\": container with ID starting with 75ebd9cdace4ab091697af66db848dc3813d027ce4655d706658839f2406d4b2 not found: ID does not exist"
Nov 02 03:44:55.450477 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:44:55.450456    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b335eb4b-7295-45f3-9dd4-637e3c7c9c58-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b335eb4b-7295-45f3-9dd4-637e3c7c9c58-metrics-certs\") pod \"b335eb4b-7295-45f3-9dd4-637e3c7c9c58\" (UID: \"b335eb4b-7295-45f3-9dd4-637e3c7c9c58\") "
Nov 02 03:44:55.450574 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:44:55.450554    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b335eb4b-7295-45f3-9dd4-637e3c7c9c58-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b335eb4b-7295-45f3-9dd4-637e3c7c9c58-openshift-service-ca\") pod \"b335eb4b-7295-45f3-9dd4-637e3c7c9c58\" (UID: \"b335eb4b-7295-45f3-9dd4-637e3c7c9c58\") "
Nov 02 03:44:55.450622 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:44:55.450577    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b335eb4b-7295-45f3-9dd4-637e3c7c9c58-kube-api-access-q2zx8\" (UniqueName: \"kubernetes.io/projected/b335eb4b-7295-45f3-9dd4-637e3c7c9c58-kube-api-access-q2zx8\") pod \"b335eb4b-7295-45f3-9dd4-637e3c7c9c58\" (UID: \"b335eb4b-7295-45f3-9dd4-637e3c7c9c58\") "
Nov 02 03:44:55.450662 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:44:55.450629    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b335eb4b-7295-45f3-9dd4-637e3c7c9c58-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b335eb4b-7295-45f3-9dd4-637e3c7c9c58-maas-api-tls\") pod \"b335eb4b-7295-45f3-9dd4-637e3c7c9c58\" (UID: \"b335eb4b-7295-45f3-9dd4-637e3c7c9c58\") "
Nov 02 03:44:55.451076 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:44:55.450948    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b335eb4b-7295-45f3-9dd4-637e3c7c9c58-openshift-service-ca" pod "b335eb4b-7295-45f3-9dd4-637e3c7c9c58" (UID: "b335eb4b-7295-45f3-9dd4-637e3c7c9c58"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:44:55.451180 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:44:55.451154    4496 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b335eb4b-7295-45f3-9dd4-637e3c7c9c58-openshift-service-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:44:55.452640 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:44:55.452617    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b335eb4b-7295-45f3-9dd4-637e3c7c9c58-kube-api-access-q2zx8" pod "b335eb4b-7295-45f3-9dd4-637e3c7c9c58" (UID: "b335eb4b-7295-45f3-9dd4-637e3c7c9c58"). InnerVolumeSpecName "kube-api-access-q2zx8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:44:55.452763 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:44:55.452746    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b335eb4b-7295-45f3-9dd4-637e3c7c9c58-metrics-certs" pod "b335eb4b-7295-45f3-9dd4-637e3c7c9c58" (UID: "b335eb4b-7295-45f3-9dd4-637e3c7c9c58"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 03:44:55.452812 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:44:55.452770    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b335eb4b-7295-45f3-9dd4-637e3c7c9c58-maas-api-tls" pod "b335eb4b-7295-45f3-9dd4-637e3c7c9c58" (UID: "b335eb4b-7295-45f3-9dd4-637e3c7c9c58"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 03:44:55.552668 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:44:55.552631    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b335eb4b-7295-45f3-9dd4-637e3c7c9c58-maas-api-tls\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:44:55.552668 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:44:55.552668    4496 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b335eb4b-7295-45f3-9dd4-637e3c7c9c58-metrics-certs\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:44:55.552886 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:44:55.552703    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2zx8\" (UniqueName: \"kubernetes.io/projected/b335eb4b-7295-45f3-9dd4-637e3c7c9c58-kube-api-access-q2zx8\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:44:55.837714 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:44:55.837680    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-ad96f4-5487969c88-xt82w"]
Nov 02 03:44:55.840813 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:44:55.840788    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-ad96f4-5487969c88-xt82w"]
Nov 02 03:44:57.573514 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:44:57.573480    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b335eb4b-7295-45f3-9dd4-637e3c7c9c58" path="/var/lib/kubelet/pods/b335eb4b-7295-45f3-9dd4-637e3c7c9c58/volumes"
Nov 02 03:45:00.161066 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:00.161029    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-1b1008-29893185-p6m52"]
Nov 02 03:45:00.161942 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:00.161920    4496 cpu_manager.go:414] "containerMap: removing container" podUID="b335eb4b-7295-45f3-9dd4-637e3c7c9c58" containerName="maas-api"
Nov 02 03:45:00.161990 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:00.161944    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b335eb4b-7295-45f3-9dd4-637e3c7c9c58" containerName="maas-api"
Nov 02 03:45:00.162148 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:00.162136    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="b335eb4b-7295-45f3-9dd4-637e3c7c9c58" containerName="maas-api"
Nov 02 03:45:00.167465 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:00.167442    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f68c7e-29893185-v2tf6"]
Nov 02 03:45:00.167572 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:00.167556    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-1b1008-29893185-p6m52"
Nov 02 03:45:00.171761 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:00.171737    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f68c7e-29893185-v2tf6"
Nov 02 03:45:00.179638 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:00.179612    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-398828-29893185-qx6s5"]
Nov 02 03:45:00.185009 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:00.184990    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-398828-29893185-qx6s5"
Nov 02 03:45:00.211654 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:00.211625    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893185-xfs9t"]
Nov 02 03:45:00.215919 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:00.215894    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-858236-29893185-swkgb"]
Nov 02 03:45:00.216092 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:00.216011    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893185-xfs9t"
Nov 02 03:45:00.219922 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:00.219903    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-858236-29893185-swkgb"
Nov 02 03:45:00.226994 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:00.226975    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9z5bk\""
Nov 02 03:45:00.264611 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:00.264581    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-1b1008-29893185-p6m52"]
Nov 02 03:45:00.269445 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:00.269419    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f68c7e-29893185-v2tf6"]
Nov 02 03:45:00.308140 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:00.307785    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfljp\" (UniqueName: \"kubernetes.io/projected/97a0dd5f-e357-46d3-9084-d698fecba5e6-kube-api-access-cfljp\") pod \"maas-api-key-cleanup-e2e-worker-w3-858236-29893185-swkgb\" (UID: \"97a0dd5f-e357-46d3-9084-d698fecba5e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-858236-29893185-swkgb"
Nov 02 03:45:00.308140 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:00.307842    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ecfde53-c331-4730-b873-397029daeae1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-1b1008-29893185-p6m52\" (UID: \"7ecfde53-c331-4730-b873-397029daeae1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-1b1008-29893185-p6m52"
Nov 02 03:45:00.308140 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:00.307884    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9mq9\" (UniqueName: \"kubernetes.io/projected/340fca13-ed7e-4312-b0c6-1ee609ab38d4-kube-api-access-c9mq9\") pod \"maas-api-key-cleanup-e2e-praxis-b-f68c7e-29893185-v2tf6\" (UID: \"340fca13-ed7e-4312-b0c6-1ee609ab38d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f68c7e-29893185-v2tf6"
Nov 02 03:45:00.308140 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:00.307927    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/340fca13-ed7e-4312-b0c6-1ee609ab38d4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-f68c7e-29893185-v2tf6\" (UID: \"340fca13-ed7e-4312-b0c6-1ee609ab38d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f68c7e-29893185-v2tf6"
Nov 02 03:45:00.308140 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:00.308032    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f8344a7-8c6e-471d-9509-7fa59b1701de-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-398828-29893185-qx6s5\" (UID: \"6f8344a7-8c6e-471d-9509-7fa59b1701de\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-398828-29893185-qx6s5"
Nov 02 03:45:00.308140 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:00.308072    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97a0dd5f-e357-46d3-9084-d698fecba5e6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-858236-29893185-swkgb\" (UID: \"97a0dd5f-e357-46d3-9084-d698fecba5e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-858236-29893185-swkgb"
Nov 02 03:45:00.316302 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:00.308178    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwxhl\" (UniqueName: \"kubernetes.io/projected/7ecfde53-c331-4730-b873-397029daeae1-kube-api-access-pwxhl\") pod \"maas-api-key-cleanup-e2e-praxis-a-1b1008-29893185-p6m52\" (UID: \"7ecfde53-c331-4730-b873-397029daeae1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-1b1008-29893185-p6m52"
Nov 02 03:45:00.316302 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:00.308223    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e8d8ba0-989d-45b4-8fbc-abe58b26c48c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893185-xfs9t\" (UID: \"8e8d8ba0-989d-45b4-8fbc-abe58b26c48c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893185-xfs9t"
Nov 02 03:45:00.316302 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:00.308255    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4qgn\" (UniqueName: \"kubernetes.io/projected/6f8344a7-8c6e-471d-9509-7fa59b1701de-kube-api-access-f4qgn\") pod \"maas-api-key-cleanup-e2e-worker-w4-398828-29893185-qx6s5\" (UID: \"6f8344a7-8c6e-471d-9509-7fa59b1701de\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-398828-29893185-qx6s5"
Nov 02 03:45:00.316302 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:00.308288    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ksh5\" (UniqueName: \"kubernetes.io/projected/8e8d8ba0-989d-45b4-8fbc-abe58b26c48c-kube-api-access-2ksh5\") pod \"maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893185-xfs9t\" (UID: \"8e8d8ba0-989d-45b4-8fbc-abe58b26c48c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893185-xfs9t"
Nov 02 03:45:00.318158 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:00.318125    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-398828-29893185-qx6s5"]
Nov 02 03:45:00.319283 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:00.319258    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893185-xfs9t"]
Nov 02 03:45:00.355510 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:00.355471    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-858236-29893185-swkgb"]
Nov 02 03:45:00.409347 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:00.409307    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwxhl\" (UniqueName: \"kubernetes.io/projected/7ecfde53-c331-4730-b873-397029daeae1-kube-api-access-pwxhl\") pod \"maas-api-key-cleanup-e2e-praxis-a-1b1008-29893185-p6m52\" (UID: \"7ecfde53-c331-4730-b873-397029daeae1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-1b1008-29893185-p6m52"
Nov 02 03:45:00.409529 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:00.409458    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e8d8ba0-989d-45b4-8fbc-abe58b26c48c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893185-xfs9t\" (UID: \"8e8d8ba0-989d-45b4-8fbc-abe58b26c48c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893185-xfs9t"
Nov 02 03:45:00.409529 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:00.409493    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4qgn\" (UniqueName: \"kubernetes.io/projected/6f8344a7-8c6e-471d-9509-7fa59b1701de-kube-api-access-f4qgn\") pod \"maas-api-key-cleanup-e2e-worker-w4-398828-29893185-qx6s5\" (UID: \"6f8344a7-8c6e-471d-9509-7fa59b1701de\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-398828-29893185-qx6s5"
Nov 02 03:45:00.409529 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:00.409523    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ksh5\" (UniqueName: \"kubernetes.io/projected/8e8d8ba0-989d-45b4-8fbc-abe58b26c48c-kube-api-access-2ksh5\") pod \"maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893185-xfs9t\" (UID: \"8e8d8ba0-989d-45b4-8fbc-abe58b26c48c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893185-xfs9t"
Nov 02 03:45:00.409700 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:00.409573    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfljp\" (UniqueName: \"kubernetes.io/projected/97a0dd5f-e357-46d3-9084-d698fecba5e6-kube-api-access-cfljp\") pod \"maas-api-key-cleanup-e2e-worker-w3-858236-29893185-swkgb\" (UID: \"97a0dd5f-e357-46d3-9084-d698fecba5e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-858236-29893185-swkgb"
Nov 02 03:45:00.409700 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:00.409603    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ecfde53-c331-4730-b873-397029daeae1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-1b1008-29893185-p6m52\" (UID: \"7ecfde53-c331-4730-b873-397029daeae1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-1b1008-29893185-p6m52"
Nov 02 03:45:00.409700 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:00.409640    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9mq9\" (UniqueName: \"kubernetes.io/projected/340fca13-ed7e-4312-b0c6-1ee609ab38d4-kube-api-access-c9mq9\") pod \"maas-api-key-cleanup-e2e-praxis-b-f68c7e-29893185-v2tf6\" (UID: \"340fca13-ed7e-4312-b0c6-1ee609ab38d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f68c7e-29893185-v2tf6"
Nov 02 03:45:00.409700 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:00.409688    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/340fca13-ed7e-4312-b0c6-1ee609ab38d4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-f68c7e-29893185-v2tf6\" (UID: \"340fca13-ed7e-4312-b0c6-1ee609ab38d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f68c7e-29893185-v2tf6"
Nov 02 03:45:00.409930 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:00.409776    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f8344a7-8c6e-471d-9509-7fa59b1701de-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-398828-29893185-qx6s5\" (UID: \"6f8344a7-8c6e-471d-9509-7fa59b1701de\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-398828-29893185-qx6s5"
Nov 02 03:45:00.409930 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:00.409817    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97a0dd5f-e357-46d3-9084-d698fecba5e6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-858236-29893185-swkgb\" (UID: \"97a0dd5f-e357-46d3-9084-d698fecba5e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-858236-29893185-swkgb"
Nov 02 03:45:00.410439 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:00.410410    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e8d8ba0-989d-45b4-8fbc-abe58b26c48c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893185-xfs9t\" (UID: \"8e8d8ba0-989d-45b4-8fbc-abe58b26c48c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893185-xfs9t"
Nov 02 03:45:00.410549 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:00.410448    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ecfde53-c331-4730-b873-397029daeae1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-1b1008-29893185-p6m52\" (UID: \"7ecfde53-c331-4730-b873-397029daeae1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-1b1008-29893185-p6m52"
Nov 02 03:45:00.410549 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:00.410479    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/340fca13-ed7e-4312-b0c6-1ee609ab38d4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-f68c7e-29893185-v2tf6\" (UID: \"340fca13-ed7e-4312-b0c6-1ee609ab38d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f68c7e-29893185-v2tf6"
Nov 02 03:45:00.410549 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:00.410522    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f8344a7-8c6e-471d-9509-7fa59b1701de-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-398828-29893185-qx6s5\" (UID: \"6f8344a7-8c6e-471d-9509-7fa59b1701de\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-398828-29893185-qx6s5"
Nov 02 03:45:00.410669 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:00.410579    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97a0dd5f-e357-46d3-9084-d698fecba5e6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-858236-29893185-swkgb\" (UID: \"97a0dd5f-e357-46d3-9084-d698fecba5e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-858236-29893185-swkgb"
Nov 02 03:45:00.468261 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:00.468179    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4qgn\" (UniqueName: \"kubernetes.io/projected/6f8344a7-8c6e-471d-9509-7fa59b1701de-kube-api-access-f4qgn\") pod \"maas-api-key-cleanup-e2e-worker-w4-398828-29893185-qx6s5\" (UID: \"6f8344a7-8c6e-471d-9509-7fa59b1701de\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-398828-29893185-qx6s5"
Nov 02 03:45:00.468484 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:00.468461    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ksh5\" (UniqueName: \"kubernetes.io/projected/8e8d8ba0-989d-45b4-8fbc-abe58b26c48c-kube-api-access-2ksh5\") pod \"maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893185-xfs9t\" (UID: \"8e8d8ba0-989d-45b4-8fbc-abe58b26c48c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893185-xfs9t"
Nov 02 03:45:00.468642 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:00.468619    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfljp\" (UniqueName: \"kubernetes.io/projected/97a0dd5f-e357-46d3-9084-d698fecba5e6-kube-api-access-cfljp\") pod \"maas-api-key-cleanup-e2e-worker-w3-858236-29893185-swkgb\" (UID: \"97a0dd5f-e357-46d3-9084-d698fecba5e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-858236-29893185-swkgb"
Nov 02 03:45:00.468732 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:00.468714    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9mq9\" (UniqueName: \"kubernetes.io/projected/340fca13-ed7e-4312-b0c6-1ee609ab38d4-kube-api-access-c9mq9\") pod \"maas-api-key-cleanup-e2e-praxis-b-f68c7e-29893185-v2tf6\" (UID: \"340fca13-ed7e-4312-b0c6-1ee609ab38d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f68c7e-29893185-v2tf6"
Nov 02 03:45:00.469002 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:00.468982    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwxhl\" (UniqueName: \"kubernetes.io/projected/7ecfde53-c331-4730-b873-397029daeae1-kube-api-access-pwxhl\") pod \"maas-api-key-cleanup-e2e-praxis-a-1b1008-29893185-p6m52\" (UID: \"7ecfde53-c331-4730-b873-397029daeae1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-1b1008-29893185-p6m52"
Nov 02 03:45:00.484293 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:00.484272    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-1b1008-29893185-p6m52"
Nov 02 03:45:00.489744 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:00.489704    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f68c7e-29893185-v2tf6"
Nov 02 03:45:00.499255 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:00.499234    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-398828-29893185-qx6s5"
Nov 02 03:45:00.530560 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:00.530527    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893185-xfs9t"
Nov 02 03:45:00.538930 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:00.538903    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-858236-29893185-swkgb"
Nov 02 03:45:00.887612 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:00.887581    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f68c7e-29893185-v2tf6"]
Nov 02 03:45:00.887879 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:45:00.887848    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod340fca13_ed7e_4312_b0c6_1ee609ab38d4.slice/crio-36589fb7971786491b195fa33848410e648bc4f0797e2d85b83ef235befa1c8e WatchSource:0}: Error finding container 36589fb7971786491b195fa33848410e648bc4f0797e2d85b83ef235befa1c8e: Status 404 returned error can't find the container with id 36589fb7971786491b195fa33848410e648bc4f0797e2d85b83ef235befa1c8e
Nov 02 03:45:00.938649 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:00.938623    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-1b1008-29893185-p6m52"]
Nov 02 03:45:00.938846 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:45:00.938822    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ecfde53_c331_4730_b873_397029daeae1.slice/crio-a7da7582763b951b73a744a95d51837277c3cb10c304d5c53f6d3a53c1b7d7d4 WatchSource:0}: Error finding container a7da7582763b951b73a744a95d51837277c3cb10c304d5c53f6d3a53c1b7d7d4: Status 404 returned error can't find the container with id a7da7582763b951b73a744a95d51837277c3cb10c304d5c53f6d3a53c1b7d7d4
Nov 02 03:45:00.939988 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:45:00.939964    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f8344a7_8c6e_471d_9509_7fa59b1701de.slice/crio-18a5f05ac64c2813eced4858c7bd5e8d5dd516d2a5e585233bc958efa975aa17 WatchSource:0}: Error finding container 18a5f05ac64c2813eced4858c7bd5e8d5dd516d2a5e585233bc958efa975aa17: Status 404 returned error can't find the container with id 18a5f05ac64c2813eced4858c7bd5e8d5dd516d2a5e585233bc958efa975aa17
Nov 02 03:45:00.943557 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:00.943504    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-398828-29893185-qx6s5"]
Nov 02 03:45:00.946168 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:45:00.946143    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e8d8ba0_989d_45b4_8fbc_abe58b26c48c.slice/crio-898ff95f6a218e9dae69273fc39fb04d89e6b7fc7d2f9ad5d024b4138ebffd66 WatchSource:0}: Error finding container 898ff95f6a218e9dae69273fc39fb04d89e6b7fc7d2f9ad5d024b4138ebffd66: Status 404 returned error can't find the container with id 898ff95f6a218e9dae69273fc39fb04d89e6b7fc7d2f9ad5d024b4138ebffd66
Nov 02 03:45:00.946438 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:00.946419    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893185-xfs9t"]
Nov 02 03:45:00.987872 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:00.987843    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-858236-29893185-swkgb"]
Nov 02 03:45:00.988187 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:45:00.988166    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97a0dd5f_e357_46d3_9084_d698fecba5e6.slice/crio-32952f9aa64bbe1c73b1a8807afa83365a75e4a3ea708ae423836770f192a048 WatchSource:0}: Error finding container 32952f9aa64bbe1c73b1a8807afa83365a75e4a3ea708ae423836770f192a048: Status 404 returned error can't find the container with id 32952f9aa64bbe1c73b1a8807afa83365a75e4a3ea708ae423836770f192a048
Nov 02 03:45:01.474357 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:01.474319    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f68c7e-29893185-v2tf6" event={"ID":"340fca13-ed7e-4312-b0c6-1ee609ab38d4","Type":"ContainerStarted","Data":"f81aff47edcf68b8b79068823010dc2343c596c944179ddd9e3291ae75169c88"}
Nov 02 03:45:01.474728 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:01.474386    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f68c7e-29893185-v2tf6" event={"ID":"340fca13-ed7e-4312-b0c6-1ee609ab38d4","Type":"ContainerStarted","Data":"36589fb7971786491b195fa33848410e648bc4f0797e2d85b83ef235befa1c8e"}
Nov 02 03:45:01.476139 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:01.476111    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-1b1008-29893185-p6m52" event={"ID":"7ecfde53-c331-4730-b873-397029daeae1","Type":"ContainerStarted","Data":"328a15c843a454a98d64119b8e54b458d27c2823f3b2bbfc3c7869424d3a5261"}
Nov 02 03:45:01.476269 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:01.476163    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-1b1008-29893185-p6m52" event={"ID":"7ecfde53-c331-4730-b873-397029daeae1","Type":"ContainerStarted","Data":"a7da7582763b951b73a744a95d51837277c3cb10c304d5c53f6d3a53c1b7d7d4"}
Nov 02 03:45:01.485043 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:01.478009    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-398828-29893185-qx6s5" event={"ID":"6f8344a7-8c6e-471d-9509-7fa59b1701de","Type":"ContainerStarted","Data":"b688058a0e4b22b08fb81fee0579bfd0cda5e5af8e12355c05aac3d6feb7633b"}
Nov 02 03:45:01.485043 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:01.478081    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-398828-29893185-qx6s5" event={"ID":"6f8344a7-8c6e-471d-9509-7fa59b1701de","Type":"ContainerStarted","Data":"18a5f05ac64c2813eced4858c7bd5e8d5dd516d2a5e585233bc958efa975aa17"}
Nov 02 03:45:01.485043 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:01.480002    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-858236-29893185-swkgb" event={"ID":"97a0dd5f-e357-46d3-9084-d698fecba5e6","Type":"ContainerStarted","Data":"db88d55caeb1bf8a6ee2ba81713a655493644796f49a79f2c827fb1c0d51c6e2"}
Nov 02 03:45:01.485043 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:01.480064    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-858236-29893185-swkgb" event={"ID":"97a0dd5f-e357-46d3-9084-d698fecba5e6","Type":"ContainerStarted","Data":"32952f9aa64bbe1c73b1a8807afa83365a75e4a3ea708ae423836770f192a048"}
Nov 02 03:45:01.485043 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:01.482064    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893185-xfs9t" event={"ID":"8e8d8ba0-989d-45b4-8fbc-abe58b26c48c","Type":"ContainerStarted","Data":"ff864ff090cd91288f3f3997836ebc3d719e8d31e0adf3bf9c840f79fc8af03f"}
Nov 02 03:45:01.485043 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:01.482119    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893185-xfs9t" event={"ID":"8e8d8ba0-989d-45b4-8fbc-abe58b26c48c","Type":"ContainerStarted","Data":"898ff95f6a218e9dae69273fc39fb04d89e6b7fc7d2f9ad5d024b4138ebffd66"}
Nov 02 03:45:01.590219 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:01.590163    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f68c7e-29893185-v2tf6" podStartSLOduration=1.590149071 podStartE2EDuration="1.590149071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:45:01.589186548 +0000 UTC m=+3442.167793932" watchObservedRunningTime="2026-11-02 03:45:01.590149071 +0000 UTC m=+3442.168756450"
Nov 02 03:45:01.705478 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:01.705377    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-1b1008-29893185-p6m52" podStartSLOduration=1.7053632250000001 podStartE2EDuration="1.705363225s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:45:01.703058147 +0000 UTC m=+3442.281665526" watchObservedRunningTime="2026-11-02 03:45:01.705363225 +0000 UTC m=+3442.283970605"
Nov 02 03:45:01.818740 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:01.818681    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-398828-29893185-qx6s5" podStartSLOduration=1.8186673500000001 podStartE2EDuration="1.81866735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:45:01.815660397 +0000 UTC m=+3442.394267777" watchObservedRunningTime="2026-11-02 03:45:01.81866735 +0000 UTC m=+3442.397274727"
Nov 02 03:45:01.932749 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:01.932682    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-858236-29893185-swkgb" podStartSLOduration=1.9326629789999998 podStartE2EDuration="1.932662979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:45:01.929089556 +0000 UTC m=+3442.507696938" watchObservedRunningTime="2026-11-02 03:45:01.932662979 +0000 UTC m=+3442.511270359"
Nov 02 03:45:02.049035 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:02.047626    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893185-xfs9t" podStartSLOduration=2.047607376 podStartE2EDuration="2.047607376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:45:02.044900834 +0000 UTC m=+3442.623508214" watchObservedRunningTime="2026-11-02 03:45:02.047607376 +0000 UTC m=+3442.626214760"
Nov 02 03:45:02.492110 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:02.492073    4496 generic.go:356] "Generic (PLEG): container finished" podID="6f8344a7-8c6e-471d-9509-7fa59b1701de" containerID="b688058a0e4b22b08fb81fee0579bfd0cda5e5af8e12355c05aac3d6feb7633b" exitCode=60
Nov 02 03:45:02.492555 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:02.492228    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-398828-29893185-qx6s5" event={"ID":"6f8344a7-8c6e-471d-9509-7fa59b1701de","Type":"ContainerDied","Data":"b688058a0e4b22b08fb81fee0579bfd0cda5e5af8e12355c05aac3d6feb7633b"}
Nov 02 03:45:02.492555 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:02.492392    4496 scope.go:122] "RemoveContainer" containerID="b688058a0e4b22b08fb81fee0579bfd0cda5e5af8e12355c05aac3d6feb7633b"
Nov 02 03:45:02.494059 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:02.494033    4496 generic.go:356] "Generic (PLEG): container finished" podID="97a0dd5f-e357-46d3-9084-d698fecba5e6" containerID="db88d55caeb1bf8a6ee2ba81713a655493644796f49a79f2c827fb1c0d51c6e2" exitCode=60
Nov 02 03:45:02.494199 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:02.494114    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-858236-29893185-swkgb" event={"ID":"97a0dd5f-e357-46d3-9084-d698fecba5e6","Type":"ContainerDied","Data":"db88d55caeb1bf8a6ee2ba81713a655493644796f49a79f2c827fb1c0d51c6e2"}
Nov 02 03:45:02.494343 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:02.494324    4496 scope.go:122] "RemoveContainer" containerID="db88d55caeb1bf8a6ee2ba81713a655493644796f49a79f2c827fb1c0d51c6e2"
Nov 02 03:45:02.496175 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:02.496004    4496 generic.go:356] "Generic (PLEG): container finished" podID="8e8d8ba0-989d-45b4-8fbc-abe58b26c48c" containerID="ff864ff090cd91288f3f3997836ebc3d719e8d31e0adf3bf9c840f79fc8af03f" exitCode=60
Nov 02 03:45:02.496175 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:02.496060    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893185-xfs9t" event={"ID":"8e8d8ba0-989d-45b4-8fbc-abe58b26c48c","Type":"ContainerDied","Data":"ff864ff090cd91288f3f3997836ebc3d719e8d31e0adf3bf9c840f79fc8af03f"}
Nov 02 03:45:02.496351 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:02.496313    4496 scope.go:122] "RemoveContainer" containerID="ff864ff090cd91288f3f3997836ebc3d719e8d31e0adf3bf9c840f79fc8af03f"
Nov 02 03:45:02.498241 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:02.498220    4496 generic.go:356] "Generic (PLEG): container finished" podID="340fca13-ed7e-4312-b0c6-1ee609ab38d4" containerID="f81aff47edcf68b8b79068823010dc2343c596c944179ddd9e3291ae75169c88" exitCode=60
Nov 02 03:45:02.498351 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:02.498303    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f68c7e-29893185-v2tf6" event={"ID":"340fca13-ed7e-4312-b0c6-1ee609ab38d4","Type":"ContainerDied","Data":"f81aff47edcf68b8b79068823010dc2343c596c944179ddd9e3291ae75169c88"}
Nov 02 03:45:02.498479 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:02.498465    4496 scope.go:122] "RemoveContainer" containerID="f81aff47edcf68b8b79068823010dc2343c596c944179ddd9e3291ae75169c88"
Nov 02 03:45:02.500073 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:02.500052    4496 generic.go:356] "Generic (PLEG): container finished" podID="7ecfde53-c331-4730-b873-397029daeae1" containerID="328a15c843a454a98d64119b8e54b458d27c2823f3b2bbfc3c7869424d3a5261" exitCode=60
Nov 02 03:45:02.500182 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:02.500083    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-1b1008-29893185-p6m52" event={"ID":"7ecfde53-c331-4730-b873-397029daeae1","Type":"ContainerDied","Data":"328a15c843a454a98d64119b8e54b458d27c2823f3b2bbfc3c7869424d3a5261"}
Nov 02 03:45:02.500314 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:02.500298    4496 scope.go:122] "RemoveContainer" containerID="328a15c843a454a98d64119b8e54b458d27c2823f3b2bbfc3c7869424d3a5261"
Nov 02 03:45:03.509856 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:03.509807    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-1b1008-29893185-p6m52" event={"ID":"7ecfde53-c331-4730-b873-397029daeae1","Type":"ContainerStarted","Data":"865369ea4f898779ce9a064ca83d581d7bcc5d589b788ce5f6f4d751f4143ba2"}
Nov 02 03:45:03.512232 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:03.512196    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-398828-29893185-qx6s5" event={"ID":"6f8344a7-8c6e-471d-9509-7fa59b1701de","Type":"ContainerStarted","Data":"7d9202e833a9cc6b230dd5995aa68e43ffde01f0ea7cac7f8e654bc4e0958f3e"}
Nov 02 03:45:03.514488 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:03.514452    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-858236-29893185-swkgb" event={"ID":"97a0dd5f-e357-46d3-9084-d698fecba5e6","Type":"ContainerStarted","Data":"5940d4f82bccc43662d6a2734611c5d68633f64185bd741da63f3352060e5fa4"}
Nov 02 03:45:03.516891 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:03.516866    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893185-xfs9t" event={"ID":"8e8d8ba0-989d-45b4-8fbc-abe58b26c48c","Type":"ContainerStarted","Data":"59922be9a35efd2898c61e7193d2f36747aaec29f5b00b125aa599056aa045eb"}
Nov 02 03:45:03.518966 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:03.518943    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f68c7e-29893185-v2tf6" event={"ID":"340fca13-ed7e-4312-b0c6-1ee609ab38d4","Type":"ContainerStarted","Data":"96a0f4357395aa23839f7b54d9cf7d3837b49b32548290b6fe531b7207dc39c0"}
Nov 02 03:45:03.529593 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:45:03.529566    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 03:45:04.529494 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:04.529459    4496 generic.go:356] "Generic (PLEG): container finished" podID="97a0dd5f-e357-46d3-9084-d698fecba5e6" containerID="5940d4f82bccc43662d6a2734611c5d68633f64185bd741da63f3352060e5fa4" exitCode=60
Nov 02 03:45:04.529943 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:04.529569    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-858236-29893185-swkgb" event={"ID":"97a0dd5f-e357-46d3-9084-d698fecba5e6","Type":"ContainerDied","Data":"5940d4f82bccc43662d6a2734611c5d68633f64185bd741da63f3352060e5fa4"}
Nov 02 03:45:04.529943 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:04.529638    4496 scope.go:122] "RemoveContainer" containerID="db88d55caeb1bf8a6ee2ba81713a655493644796f49a79f2c827fb1c0d51c6e2"
Nov 02 03:45:04.529943 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:04.529854    4496 scope.go:122] "RemoveContainer" containerID="5940d4f82bccc43662d6a2734611c5d68633f64185bd741da63f3352060e5fa4"
Nov 02 03:45:04.530253 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:45:04.530222    4496 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-858236-29893185-swkgb_odh-ai-gateway-infra(97a0dd5f-e357-46d3-9084-d698fecba5e6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-858236-29893185-swkgb" podUID="97a0dd5f-e357-46d3-9084-d698fecba5e6"
Nov 02 03:45:04.531882 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:04.531860    4496 generic.go:356] "Generic (PLEG): container finished" podID="8e8d8ba0-989d-45b4-8fbc-abe58b26c48c" containerID="59922be9a35efd2898c61e7193d2f36747aaec29f5b00b125aa599056aa045eb" exitCode=60
Nov 02 03:45:04.531980 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:04.531912    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893185-xfs9t" event={"ID":"8e8d8ba0-989d-45b4-8fbc-abe58b26c48c","Type":"ContainerDied","Data":"59922be9a35efd2898c61e7193d2f36747aaec29f5b00b125aa599056aa045eb"}
Nov 02 03:45:04.532228 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:04.532212    4496 scope.go:122] "RemoveContainer" containerID="59922be9a35efd2898c61e7193d2f36747aaec29f5b00b125aa599056aa045eb"
Nov 02 03:45:04.532527 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:45:04.532504    4496 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893185-xfs9t_odh-ai-gateway-infra(8e8d8ba0-989d-45b4-8fbc-abe58b26c48c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893185-xfs9t" podUID="8e8d8ba0-989d-45b4-8fbc-abe58b26c48c"
Nov 02 03:45:04.533979 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:04.533958    4496 generic.go:356] "Generic (PLEG): container finished" podID="340fca13-ed7e-4312-b0c6-1ee609ab38d4" containerID="96a0f4357395aa23839f7b54d9cf7d3837b49b32548290b6fe531b7207dc39c0" exitCode=60
Nov 02 03:45:04.534082 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:04.534044    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f68c7e-29893185-v2tf6" event={"ID":"340fca13-ed7e-4312-b0c6-1ee609ab38d4","Type":"ContainerDied","Data":"96a0f4357395aa23839f7b54d9cf7d3837b49b32548290b6fe531b7207dc39c0"}
Nov 02 03:45:04.534281 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:04.534267    4496 scope.go:122] "RemoveContainer" containerID="96a0f4357395aa23839f7b54d9cf7d3837b49b32548290b6fe531b7207dc39c0"
Nov 02 03:45:04.534551 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:45:04.534526    4496 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-f68c7e-29893185-v2tf6_odh-ai-gateway-infra(340fca13-ed7e-4312-b0c6-1ee609ab38d4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f68c7e-29893185-v2tf6" podUID="340fca13-ed7e-4312-b0c6-1ee609ab38d4"
Nov 02 03:45:04.535956 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:04.535928    4496 generic.go:356] "Generic (PLEG): container finished" podID="7ecfde53-c331-4730-b873-397029daeae1" containerID="865369ea4f898779ce9a064ca83d581d7bcc5d589b788ce5f6f4d751f4143ba2" exitCode=60
Nov 02 03:45:04.536280 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:04.536048    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-1b1008-29893185-p6m52" event={"ID":"7ecfde53-c331-4730-b873-397029daeae1","Type":"ContainerDied","Data":"865369ea4f898779ce9a064ca83d581d7bcc5d589b788ce5f6f4d751f4143ba2"}
Nov 02 03:45:04.536280 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:04.536195    4496 scope.go:122] "RemoveContainer" containerID="865369ea4f898779ce9a064ca83d581d7bcc5d589b788ce5f6f4d751f4143ba2"
Nov 02 03:45:04.536462 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:45:04.536438    4496 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-1b1008-29893185-p6m52_odh-ai-gateway-infra(7ecfde53-c331-4730-b873-397029daeae1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-1b1008-29893185-p6m52" podUID="7ecfde53-c331-4730-b873-397029daeae1"
Nov 02 03:45:04.538295 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:04.538274    4496 generic.go:356] "Generic (PLEG): container finished" podID="6f8344a7-8c6e-471d-9509-7fa59b1701de" containerID="7d9202e833a9cc6b230dd5995aa68e43ffde01f0ea7cac7f8e654bc4e0958f3e" exitCode=60
Nov 02 03:45:04.538380 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:04.538318    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-398828-29893185-qx6s5" event={"ID":"6f8344a7-8c6e-471d-9509-7fa59b1701de","Type":"ContainerDied","Data":"7d9202e833a9cc6b230dd5995aa68e43ffde01f0ea7cac7f8e654bc4e0958f3e"}
Nov 02 03:45:04.538583 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:04.538566    4496 scope.go:122] "RemoveContainer" containerID="7d9202e833a9cc6b230dd5995aa68e43ffde01f0ea7cac7f8e654bc4e0958f3e"
Nov 02 03:45:04.538828 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:45:04.538806    4496 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-398828-29893185-qx6s5_odh-ai-gateway-infra(6f8344a7-8c6e-471d-9509-7fa59b1701de)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-398828-29893185-qx6s5" podUID="6f8344a7-8c6e-471d-9509-7fa59b1701de"
Nov 02 03:45:04.546009 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:04.545927    4496 scope.go:122] "RemoveContainer" containerID="ff864ff090cd91288f3f3997836ebc3d719e8d31e0adf3bf9c840f79fc8af03f"
Nov 02 03:45:04.561628 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:04.561609    4496 scope.go:122] "RemoveContainer" containerID="f81aff47edcf68b8b79068823010dc2343c596c944179ddd9e3291ae75169c88"
Nov 02 03:45:04.574460 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:04.574443    4496 scope.go:122] "RemoveContainer" containerID="328a15c843a454a98d64119b8e54b458d27c2823f3b2bbfc3c7869424d3a5261"
Nov 02 03:45:04.586961 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:04.586941    4496 scope.go:122] "RemoveContainer" containerID="b688058a0e4b22b08fb81fee0579bfd0cda5e5af8e12355c05aac3d6feb7633b"
Nov 02 03:45:05.549243 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:05.549199    4496 scope.go:122] "RemoveContainer" containerID="865369ea4f898779ce9a064ca83d581d7bcc5d589b788ce5f6f4d751f4143ba2"
Nov 02 03:45:05.549862 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:45:05.549564    4496 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-1b1008-29893185-p6m52_odh-ai-gateway-infra(7ecfde53-c331-4730-b873-397029daeae1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-1b1008-29893185-p6m52" podUID="7ecfde53-c331-4730-b873-397029daeae1"
Nov 02 03:45:05.551311 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:05.551291    4496 scope.go:122] "RemoveContainer" containerID="7d9202e833a9cc6b230dd5995aa68e43ffde01f0ea7cac7f8e654bc4e0958f3e"
Nov 02 03:45:05.551600 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:45:05.551579    4496 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-398828-29893185-qx6s5_odh-ai-gateway-infra(6f8344a7-8c6e-471d-9509-7fa59b1701de)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-398828-29893185-qx6s5" podUID="6f8344a7-8c6e-471d-9509-7fa59b1701de"
Nov 02 03:45:05.553160 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:05.553140    4496 scope.go:122] "RemoveContainer" containerID="5940d4f82bccc43662d6a2734611c5d68633f64185bd741da63f3352060e5fa4"
Nov 02 03:45:05.553494 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:45:05.553452    4496 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-858236-29893185-swkgb_odh-ai-gateway-infra(97a0dd5f-e357-46d3-9084-d698fecba5e6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-858236-29893185-swkgb" podUID="97a0dd5f-e357-46d3-9084-d698fecba5e6"
Nov 02 03:45:05.555231 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:05.555212    4496 scope.go:122] "RemoveContainer" containerID="59922be9a35efd2898c61e7193d2f36747aaec29f5b00b125aa599056aa045eb"
Nov 02 03:45:05.555459 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:45:05.555431    4496 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893185-xfs9t_odh-ai-gateway-infra(8e8d8ba0-989d-45b4-8fbc-abe58b26c48c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893185-xfs9t" podUID="8e8d8ba0-989d-45b4-8fbc-abe58b26c48c"
Nov 02 03:45:05.557343 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:05.557325    4496 scope.go:122] "RemoveContainer" containerID="96a0f4357395aa23839f7b54d9cf7d3837b49b32548290b6fe531b7207dc39c0"
Nov 02 03:45:05.557567 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:45:05.557538    4496 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-f68c7e-29893185-v2tf6_odh-ai-gateway-infra(340fca13-ed7e-4312-b0c6-1ee609ab38d4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f68c7e-29893185-v2tf6" podUID="340fca13-ed7e-4312-b0c6-1ee609ab38d4"
Nov 02 03:45:15.574536 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:15.574510    4496 scope.go:122] "RemoveContainer" containerID="96a0f4357395aa23839f7b54d9cf7d3837b49b32548290b6fe531b7207dc39c0"
Nov 02 03:45:15.574963 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:15.574642    4496 scope.go:122] "RemoveContainer" containerID="59922be9a35efd2898c61e7193d2f36747aaec29f5b00b125aa599056aa045eb"
Nov 02 03:45:15.574963 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:15.574675    4496 scope.go:122] "RemoveContainer" containerID="865369ea4f898779ce9a064ca83d581d7bcc5d589b788ce5f6f4d751f4143ba2"
Nov 02 03:45:16.565964 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:16.565932    4496 scope.go:122] "RemoveContainer" containerID="5940d4f82bccc43662d6a2734611c5d68633f64185bd741da63f3352060e5fa4"
Nov 02 03:45:16.566165 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:16.566109    4496 scope.go:122] "RemoveContainer" containerID="7d9202e833a9cc6b230dd5995aa68e43ffde01f0ea7cac7f8e654bc4e0958f3e"
Nov 02 03:45:16.646063 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:16.645999    4496 generic.go:356] "Generic (PLEG): container finished" podID="8e8d8ba0-989d-45b4-8fbc-abe58b26c48c" containerID="43fa52f1822fcc6689fb72f9a135ccf9c964116f3a2bb42a239781bb4ba96125" exitCode=60
Nov 02 03:45:16.646063 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:16.646059    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893185-xfs9t" event={"ID":"8e8d8ba0-989d-45b4-8fbc-abe58b26c48c","Type":"ContainerDied","Data":"43fa52f1822fcc6689fb72f9a135ccf9c964116f3a2bb42a239781bb4ba96125"}
Nov 02 03:45:16.646595 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:16.646123    4496 scope.go:122] "RemoveContainer" containerID="59922be9a35efd2898c61e7193d2f36747aaec29f5b00b125aa599056aa045eb"
Nov 02 03:45:16.646595 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:16.646500    4496 scope.go:122] "RemoveContainer" containerID="43fa52f1822fcc6689fb72f9a135ccf9c964116f3a2bb42a239781bb4ba96125"
Nov 02 03:45:16.646830 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:45:16.646804    4496 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893185-xfs9t_odh-ai-gateway-infra(8e8d8ba0-989d-45b4-8fbc-abe58b26c48c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893185-xfs9t" podUID="8e8d8ba0-989d-45b4-8fbc-abe58b26c48c"
Nov 02 03:45:16.648501 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:16.648478    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f68c7e-29893185-v2tf6" event={"ID":"340fca13-ed7e-4312-b0c6-1ee609ab38d4","Type":"ContainerStarted","Data":"672dfe3cd998a64c5b936b359cff54f6a8c3d9a38a58098bf968d71ede9081d0"}
Nov 02 03:45:16.650430 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:16.650404    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-1b1008-29893185-p6m52" event={"ID":"7ecfde53-c331-4730-b873-397029daeae1","Type":"ContainerStarted","Data":"6396b482668bb2a59135dfd95c784e907c2102bba900415deadbd7fc0e826c92"}
Nov 02 03:45:17.661629 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:17.661597    4496 generic.go:356] "Generic (PLEG): container finished" podID="97a0dd5f-e357-46d3-9084-d698fecba5e6" containerID="91e4e4b27e309f757e2e6e333789f2ef68dc6f23d18f6770b01a5738007825c4" exitCode=60
Nov 02 03:45:17.662084 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:17.661668    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-858236-29893185-swkgb" event={"ID":"97a0dd5f-e357-46d3-9084-d698fecba5e6","Type":"ContainerDied","Data":"91e4e4b27e309f757e2e6e333789f2ef68dc6f23d18f6770b01a5738007825c4"}
Nov 02 03:45:17.662084 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:17.661731    4496 scope.go:122] "RemoveContainer" containerID="5940d4f82bccc43662d6a2734611c5d68633f64185bd741da63f3352060e5fa4"
Nov 02 03:45:17.662221 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:17.662170    4496 scope.go:122] "RemoveContainer" containerID="91e4e4b27e309f757e2e6e333789f2ef68dc6f23d18f6770b01a5738007825c4"
Nov 02 03:45:17.662506 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:45:17.662469    4496 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-858236-29893185-swkgb_odh-ai-gateway-infra(97a0dd5f-e357-46d3-9084-d698fecba5e6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-858236-29893185-swkgb" podUID="97a0dd5f-e357-46d3-9084-d698fecba5e6"
Nov 02 03:45:17.666293 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:17.666268    4496 generic.go:356] "Generic (PLEG): container finished" podID="340fca13-ed7e-4312-b0c6-1ee609ab38d4" containerID="672dfe3cd998a64c5b936b359cff54f6a8c3d9a38a58098bf968d71ede9081d0" exitCode=60
Nov 02 03:45:17.666407 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:17.666297    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f68c7e-29893185-v2tf6" event={"ID":"340fca13-ed7e-4312-b0c6-1ee609ab38d4","Type":"ContainerDied","Data":"672dfe3cd998a64c5b936b359cff54f6a8c3d9a38a58098bf968d71ede9081d0"}
Nov 02 03:45:17.666688 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:17.666667    4496 scope.go:122] "RemoveContainer" containerID="672dfe3cd998a64c5b936b359cff54f6a8c3d9a38a58098bf968d71ede9081d0"
Nov 02 03:45:17.666985 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:45:17.666963    4496 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-f68c7e-29893185-v2tf6_odh-ai-gateway-infra(340fca13-ed7e-4312-b0c6-1ee609ab38d4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f68c7e-29893185-v2tf6" podUID="340fca13-ed7e-4312-b0c6-1ee609ab38d4"
Nov 02 03:45:17.668546 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:17.668525    4496 generic.go:356] "Generic (PLEG): container finished" podID="7ecfde53-c331-4730-b873-397029daeae1" containerID="6396b482668bb2a59135dfd95c784e907c2102bba900415deadbd7fc0e826c92" exitCode=60
Nov 02 03:45:17.668653 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:17.668588    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-1b1008-29893185-p6m52" event={"ID":"7ecfde53-c331-4730-b873-397029daeae1","Type":"ContainerDied","Data":"6396b482668bb2a59135dfd95c784e907c2102bba900415deadbd7fc0e826c92"}
Nov 02 03:45:17.668963 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:17.668941    4496 scope.go:122] "RemoveContainer" containerID="6396b482668bb2a59135dfd95c784e907c2102bba900415deadbd7fc0e826c92"
Nov 02 03:45:17.669240 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:45:17.669210    4496 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-1b1008-29893185-p6m52_odh-ai-gateway-infra(7ecfde53-c331-4730-b873-397029daeae1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-1b1008-29893185-p6m52" podUID="7ecfde53-c331-4730-b873-397029daeae1"
Nov 02 03:45:17.670917 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:17.670897    4496 generic.go:356] "Generic (PLEG): container finished" podID="6f8344a7-8c6e-471d-9509-7fa59b1701de" containerID="dc60ca881fd7b62ca06c8762217fc7dd3763b811b72d7d6209a6338dbe9f0a67" exitCode=60
Nov 02 03:45:17.671035 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:17.670970    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-398828-29893185-qx6s5" event={"ID":"6f8344a7-8c6e-471d-9509-7fa59b1701de","Type":"ContainerDied","Data":"dc60ca881fd7b62ca06c8762217fc7dd3763b811b72d7d6209a6338dbe9f0a67"}
Nov 02 03:45:17.671365 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:17.671349    4496 scope.go:122] "RemoveContainer" containerID="dc60ca881fd7b62ca06c8762217fc7dd3763b811b72d7d6209a6338dbe9f0a67"
Nov 02 03:45:17.671613 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:45:17.671593    4496 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-398828-29893185-qx6s5_odh-ai-gateway-infra(6f8344a7-8c6e-471d-9509-7fa59b1701de)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-398828-29893185-qx6s5" podUID="6f8344a7-8c6e-471d-9509-7fa59b1701de"
Nov 02 03:45:17.676136 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:17.676111    4496 scope.go:122] "RemoveContainer" containerID="96a0f4357395aa23839f7b54d9cf7d3837b49b32548290b6fe531b7207dc39c0"
Nov 02 03:45:17.689397 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:17.689377    4496 scope.go:122] "RemoveContainer" containerID="865369ea4f898779ce9a064ca83d581d7bcc5d589b788ce5f6f4d751f4143ba2"
Nov 02 03:45:17.702189 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:17.702172    4496 scope.go:122] "RemoveContainer" containerID="7d9202e833a9cc6b230dd5995aa68e43ffde01f0ea7cac7f8e654bc4e0958f3e"
Nov 02 03:45:17.771650 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:17.771609    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893185-xfs9t"]
Nov 02 03:45:17.886963 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:17.886921    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f68c7e-29893185-v2tf6"]
Nov 02 03:45:17.931934 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:17.931910    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893185-xfs9t"
Nov 02 03:45:18.000100 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:18.000065    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-1b1008-29893185-p6m52"]
Nov 02 03:45:18.032366 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:18.032335    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e8d8ba0-989d-45b4-8fbc-abe58b26c48c-kube-api-access-2ksh5\" (UniqueName: \"kubernetes.io/projected/8e8d8ba0-989d-45b4-8fbc-abe58b26c48c-kube-api-access-2ksh5\") pod \"8e8d8ba0-989d-45b4-8fbc-abe58b26c48c\" (UID: \"8e8d8ba0-989d-45b4-8fbc-abe58b26c48c\") "
Nov 02 03:45:18.032551 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:18.032444    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e8d8ba0-989d-45b4-8fbc-abe58b26c48c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e8d8ba0-989d-45b4-8fbc-abe58b26c48c-maas-api-service-ca\") pod \"8e8d8ba0-989d-45b4-8fbc-abe58b26c48c\" (UID: \"8e8d8ba0-989d-45b4-8fbc-abe58b26c48c\") "
Nov 02 03:45:18.032827 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:18.032799    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e8d8ba0-989d-45b4-8fbc-abe58b26c48c-maas-api-service-ca" pod "8e8d8ba0-989d-45b4-8fbc-abe58b26c48c" (UID: "8e8d8ba0-989d-45b4-8fbc-abe58b26c48c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:45:18.034671 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:18.034641    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e8d8ba0-989d-45b4-8fbc-abe58b26c48c-kube-api-access-2ksh5" pod "8e8d8ba0-989d-45b4-8fbc-abe58b26c48c" (UID: "8e8d8ba0-989d-45b4-8fbc-abe58b26c48c"). InnerVolumeSpecName "kube-api-access-2ksh5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:45:18.133960 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:18.133919    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ksh5\" (UniqueName: \"kubernetes.io/projected/8e8d8ba0-989d-45b4-8fbc-abe58b26c48c-kube-api-access-2ksh5\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:45:18.133960 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:18.133958    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e8d8ba0-989d-45b4-8fbc-abe58b26c48c-maas-api-service-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:45:18.681634 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:18.681595    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893185-xfs9t" event={"ID":"8e8d8ba0-989d-45b4-8fbc-abe58b26c48c","Type":"ContainerDied","Data":"898ff95f6a218e9dae69273fc39fb04d89e6b7fc7d2f9ad5d024b4138ebffd66"}
Nov 02 03:45:18.681634 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:18.681621    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893185-xfs9t"
Nov 02 03:45:18.682235 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:18.681666    4496 scope.go:122] "RemoveContainer" containerID="43fa52f1822fcc6689fb72f9a135ccf9c964116f3a2bb42a239781bb4ba96125"
Nov 02 03:45:18.794536 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:18.794506    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-858236-29893185-swkgb"]
Nov 02 03:45:18.850445 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:18.850163    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-1b1008-29893185-p6m52"
Nov 02 03:45:18.870954 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:18.870928    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f68c7e-29893185-v2tf6"
Nov 02 03:45:18.905763 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:18.905342    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-398828-29893185-qx6s5"]
Nov 02 03:45:18.943844 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:18.943755    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/340fca13-ed7e-4312-b0c6-1ee609ab38d4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/340fca13-ed7e-4312-b0c6-1ee609ab38d4-maas-api-service-ca\") pod \"340fca13-ed7e-4312-b0c6-1ee609ab38d4\" (UID: \"340fca13-ed7e-4312-b0c6-1ee609ab38d4\") "
Nov 02 03:45:18.943992 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:18.943879    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/340fca13-ed7e-4312-b0c6-1ee609ab38d4-kube-api-access-c9mq9\" (UniqueName: \"kubernetes.io/projected/340fca13-ed7e-4312-b0c6-1ee609ab38d4-kube-api-access-c9mq9\") pod \"340fca13-ed7e-4312-b0c6-1ee609ab38d4\" (UID: \"340fca13-ed7e-4312-b0c6-1ee609ab38d4\") "
Nov 02 03:45:18.943992 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:18.943941    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ecfde53-c331-4730-b873-397029daeae1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ecfde53-c331-4730-b873-397029daeae1-maas-api-service-ca\") pod \"7ecfde53-c331-4730-b873-397029daeae1\" (UID: \"7ecfde53-c331-4730-b873-397029daeae1\") "
Nov 02 03:45:18.944083 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:18.943992    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ecfde53-c331-4730-b873-397029daeae1-kube-api-access-pwxhl\" (UniqueName: \"kubernetes.io/projected/7ecfde53-c331-4730-b873-397029daeae1-kube-api-access-pwxhl\") pod \"7ecfde53-c331-4730-b873-397029daeae1\" (UID: \"7ecfde53-c331-4730-b873-397029daeae1\") "
Nov 02 03:45:18.944721 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:18.944688    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ecfde53-c331-4730-b873-397029daeae1-maas-api-service-ca" pod "7ecfde53-c331-4730-b873-397029daeae1" (UID: "7ecfde53-c331-4730-b873-397029daeae1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:45:18.945079 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:18.945009    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/340fca13-ed7e-4312-b0c6-1ee609ab38d4-maas-api-service-ca" pod "340fca13-ed7e-4312-b0c6-1ee609ab38d4" (UID: "340fca13-ed7e-4312-b0c6-1ee609ab38d4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:45:18.947482 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:18.947445    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ecfde53-c331-4730-b873-397029daeae1-kube-api-access-pwxhl" pod "7ecfde53-c331-4730-b873-397029daeae1" (UID: "7ecfde53-c331-4730-b873-397029daeae1"). InnerVolumeSpecName "kube-api-access-pwxhl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:45:18.947780 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:18.947757    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/340fca13-ed7e-4312-b0c6-1ee609ab38d4-kube-api-access-c9mq9" pod "340fca13-ed7e-4312-b0c6-1ee609ab38d4" (UID: "340fca13-ed7e-4312-b0c6-1ee609ab38d4"). InnerVolumeSpecName "kube-api-access-c9mq9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:45:18.969743 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:18.969600    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893185-xfs9t"]
Nov 02 03:45:18.973125 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:18.973101    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893185-xfs9t"]
Nov 02 03:45:18.982237 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:18.982216    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-858236-29893185-swkgb"
Nov 02 03:45:19.045808 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:19.045773    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97a0dd5f-e357-46d3-9084-d698fecba5e6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97a0dd5f-e357-46d3-9084-d698fecba5e6-maas-api-service-ca\") pod \"97a0dd5f-e357-46d3-9084-d698fecba5e6\" (UID: \"97a0dd5f-e357-46d3-9084-d698fecba5e6\") "
Nov 02 03:45:19.046051 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:19.045814    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97a0dd5f-e357-46d3-9084-d698fecba5e6-kube-api-access-cfljp\" (UniqueName: \"kubernetes.io/projected/97a0dd5f-e357-46d3-9084-d698fecba5e6-kube-api-access-cfljp\") pod \"97a0dd5f-e357-46d3-9084-d698fecba5e6\" (UID: \"97a0dd5f-e357-46d3-9084-d698fecba5e6\") "
Nov 02 03:45:19.046216 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:19.046186    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97a0dd5f-e357-46d3-9084-d698fecba5e6-maas-api-service-ca" pod "97a0dd5f-e357-46d3-9084-d698fecba5e6" (UID: "97a0dd5f-e357-46d3-9084-d698fecba5e6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:45:19.047563 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:19.047534    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ecfde53-c331-4730-b873-397029daeae1-maas-api-service-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:45:19.047563 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:19.047564    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97a0dd5f-e357-46d3-9084-d698fecba5e6-maas-api-service-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:45:19.047760 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:19.047579    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwxhl\" (UniqueName: \"kubernetes.io/projected/7ecfde53-c331-4730-b873-397029daeae1-kube-api-access-pwxhl\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:45:19.047760 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:19.047593    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/340fca13-ed7e-4312-b0c6-1ee609ab38d4-maas-api-service-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:45:19.047760 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:19.047607    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9mq9\" (UniqueName: \"kubernetes.io/projected/340fca13-ed7e-4312-b0c6-1ee609ab38d4-kube-api-access-c9mq9\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:45:19.048863 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:19.048838    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97a0dd5f-e357-46d3-9084-d698fecba5e6-kube-api-access-cfljp" pod "97a0dd5f-e357-46d3-9084-d698fecba5e6" (UID: "97a0dd5f-e357-46d3-9084-d698fecba5e6"). InnerVolumeSpecName "kube-api-access-cfljp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:45:19.093342 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:19.093316    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-398828-29893185-qx6s5"
Nov 02 03:45:19.148478 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:19.148445    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f8344a7-8c6e-471d-9509-7fa59b1701de-kube-api-access-f4qgn\" (UniqueName: \"kubernetes.io/projected/6f8344a7-8c6e-471d-9509-7fa59b1701de-kube-api-access-f4qgn\") pod \"6f8344a7-8c6e-471d-9509-7fa59b1701de\" (UID: \"6f8344a7-8c6e-471d-9509-7fa59b1701de\") "
Nov 02 03:45:19.148651 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:19.148633    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f8344a7-8c6e-471d-9509-7fa59b1701de-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f8344a7-8c6e-471d-9509-7fa59b1701de-maas-api-service-ca\") pod \"6f8344a7-8c6e-471d-9509-7fa59b1701de\" (UID: \"6f8344a7-8c6e-471d-9509-7fa59b1701de\") "
Nov 02 03:45:19.148999 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:19.148974    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f8344a7-8c6e-471d-9509-7fa59b1701de-maas-api-service-ca" pod "6f8344a7-8c6e-471d-9509-7fa59b1701de" (UID: "6f8344a7-8c6e-471d-9509-7fa59b1701de"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:45:19.149102 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:19.149051    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfljp\" (UniqueName: \"kubernetes.io/projected/97a0dd5f-e357-46d3-9084-d698fecba5e6-kube-api-access-cfljp\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:45:19.150536 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:19.150519    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f8344a7-8c6e-471d-9509-7fa59b1701de-kube-api-access-f4qgn" pod "6f8344a7-8c6e-471d-9509-7fa59b1701de" (UID: "6f8344a7-8c6e-471d-9509-7fa59b1701de"). InnerVolumeSpecName "kube-api-access-f4qgn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:45:19.250458 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:19.250399    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f8344a7-8c6e-471d-9509-7fa59b1701de-maas-api-service-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:45:19.250458 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:19.250438    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4qgn\" (UniqueName: \"kubernetes.io/projected/6f8344a7-8c6e-471d-9509-7fa59b1701de-kube-api-access-f4qgn\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:45:19.576431 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:19.576306    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e8d8ba0-989d-45b4-8fbc-abe58b26c48c" path="/var/lib/kubelet/pods/8e8d8ba0-989d-45b4-8fbc-abe58b26c48c/volumes"
Nov 02 03:45:19.696871 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:19.696838    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-858236-29893185-swkgb"
Nov 02 03:45:19.697357 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:19.696848    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-858236-29893185-swkgb" event={"ID":"97a0dd5f-e357-46d3-9084-d698fecba5e6","Type":"ContainerDied","Data":"32952f9aa64bbe1c73b1a8807afa83365a75e4a3ea708ae423836770f192a048"}
Nov 02 03:45:19.697357 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:19.697052    4496 scope.go:122] "RemoveContainer" containerID="91e4e4b27e309f757e2e6e333789f2ef68dc6f23d18f6770b01a5738007825c4"
Nov 02 03:45:19.700328 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:19.700299    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f68c7e-29893185-v2tf6" event={"ID":"340fca13-ed7e-4312-b0c6-1ee609ab38d4","Type":"ContainerDied","Data":"36589fb7971786491b195fa33848410e648bc4f0797e2d85b83ef235befa1c8e"}
Nov 02 03:45:19.700451 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:19.700306    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f68c7e-29893185-v2tf6"
Nov 02 03:45:19.702347 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:19.702327    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-1b1008-29893185-p6m52"
Nov 02 03:45:19.702472 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:19.702351    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-1b1008-29893185-p6m52" event={"ID":"7ecfde53-c331-4730-b873-397029daeae1","Type":"ContainerDied","Data":"a7da7582763b951b73a744a95d51837277c3cb10c304d5c53f6d3a53c1b7d7d4"}
Nov 02 03:45:19.703819 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:19.703799    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-398828-29893185-qx6s5"
Nov 02 03:45:19.703945 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:19.703819    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-398828-29893185-qx6s5" event={"ID":"6f8344a7-8c6e-471d-9509-7fa59b1701de","Type":"ContainerDied","Data":"18a5f05ac64c2813eced4858c7bd5e8d5dd516d2a5e585233bc958efa975aa17"}
Nov 02 03:45:19.709299 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:19.709271    4496 scope.go:122] "RemoveContainer" containerID="672dfe3cd998a64c5b936b359cff54f6a8c3d9a38a58098bf968d71ede9081d0"
Nov 02 03:45:19.718637 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:19.718612    4496 scope.go:122] "RemoveContainer" containerID="6396b482668bb2a59135dfd95c784e907c2102bba900415deadbd7fc0e826c92"
Nov 02 03:45:19.727958 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:19.727939    4496 scope.go:122] "RemoveContainer" containerID="dc60ca881fd7b62ca06c8762217fc7dd3763b811b72d7d6209a6338dbe9f0a67"
Nov 02 03:45:19.865512 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:19.865425    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-398828-29893185-qx6s5"]
Nov 02 03:45:19.869538 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:19.869502    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-398828-29893185-qx6s5"]
Nov 02 03:45:19.983575 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:19.983544    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-858236-29893185-swkgb"]
Nov 02 03:45:19.989484 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:19.989452    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-858236-29893185-swkgb"]
Nov 02 03:45:20.100875 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:20.100836    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f68c7e-29893185-v2tf6"]
Nov 02 03:45:20.104721 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:20.104694    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f68c7e-29893185-v2tf6"]
Nov 02 03:45:20.215561 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:20.215485    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-1b1008-29893185-p6m52"]
Nov 02 03:45:20.226697 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:20.226666    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-1b1008-29893185-p6m52"]
Nov 02 03:45:21.575525 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:21.575489    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="340fca13-ed7e-4312-b0c6-1ee609ab38d4" path="/var/lib/kubelet/pods/340fca13-ed7e-4312-b0c6-1ee609ab38d4/volumes"
Nov 02 03:45:21.576126 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:21.576104    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f8344a7-8c6e-471d-9509-7fa59b1701de" path="/var/lib/kubelet/pods/6f8344a7-8c6e-471d-9509-7fa59b1701de/volumes"
Nov 02 03:45:21.576678 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:21.576659    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ecfde53-c331-4730-b873-397029daeae1" path="/var/lib/kubelet/pods/7ecfde53-c331-4730-b873-397029daeae1/volumes"
Nov 02 03:45:21.577670 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:21.577650    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97a0dd5f-e357-46d3-9084-d698fecba5e6" path="/var/lib/kubelet/pods/97a0dd5f-e357-46d3-9084-d698fecba5e6/volumes"
Nov 02 03:45:32.733131 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:32.733053    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ee0fed-66c7bc7579-6b2zf"]
Nov 02 03:45:32.733630 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:32.733615    4496 cpu_manager.go:414] "containerMap: removing container" podUID="6f8344a7-8c6e-471d-9509-7fa59b1701de" containerName="cleanup"
Nov 02 03:45:32.733677 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:32.733631    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f8344a7-8c6e-471d-9509-7fa59b1701de" containerName="cleanup"
Nov 02 03:45:32.733677 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:32.733646    4496 cpu_manager.go:414] "containerMap: removing container" podUID="97a0dd5f-e357-46d3-9084-d698fecba5e6" containerName="cleanup"
Nov 02 03:45:32.733677 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:32.733652    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97a0dd5f-e357-46d3-9084-d698fecba5e6" containerName="cleanup"
Nov 02 03:45:32.733677 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:32.733663    4496 cpu_manager.go:414] "containerMap: removing container" podUID="7ecfde53-c331-4730-b873-397029daeae1" containerName="cleanup"
Nov 02 03:45:32.733677 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:32.733668    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ecfde53-c331-4730-b873-397029daeae1" containerName="cleanup"
Nov 02 03:45:32.733677 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:32.733676    4496 cpu_manager.go:414] "containerMap: removing container" podUID="7ecfde53-c331-4730-b873-397029daeae1" containerName="cleanup"
Nov 02 03:45:32.733865 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:32.733681    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ecfde53-c331-4730-b873-397029daeae1" containerName="cleanup"
Nov 02 03:45:32.733865 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:32.733691    4496 cpu_manager.go:414] "containerMap: removing container" podUID="8e8d8ba0-989d-45b4-8fbc-abe58b26c48c" containerName="cleanup"
Nov 02 03:45:32.733865 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:32.733695    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e8d8ba0-989d-45b4-8fbc-abe58b26c48c" containerName="cleanup"
Nov 02 03:45:32.733865 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:32.733703    4496 cpu_manager.go:414] "containerMap: removing container" podUID="8e8d8ba0-989d-45b4-8fbc-abe58b26c48c" containerName="cleanup"
Nov 02 03:45:32.733865 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:32.733707    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e8d8ba0-989d-45b4-8fbc-abe58b26c48c" containerName="cleanup"
Nov 02 03:45:32.733865 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:32.733718    4496 cpu_manager.go:414] "containerMap: removing container" podUID="97a0dd5f-e357-46d3-9084-d698fecba5e6" containerName="cleanup"
Nov 02 03:45:32.733865 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:32.733723    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97a0dd5f-e357-46d3-9084-d698fecba5e6" containerName="cleanup"
Nov 02 03:45:32.733865 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:32.733734    4496 cpu_manager.go:414] "containerMap: removing container" podUID="7ecfde53-c331-4730-b873-397029daeae1" containerName="cleanup"
Nov 02 03:45:32.733865 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:32.733738    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ecfde53-c331-4730-b873-397029daeae1" containerName="cleanup"
Nov 02 03:45:32.733865 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:32.733746    4496 cpu_manager.go:414] "containerMap: removing container" podUID="8e8d8ba0-989d-45b4-8fbc-abe58b26c48c" containerName="cleanup"
Nov 02 03:45:32.733865 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:32.733750    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e8d8ba0-989d-45b4-8fbc-abe58b26c48c" containerName="cleanup"
Nov 02 03:45:32.733865 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:32.733762    4496 cpu_manager.go:414] "containerMap: removing container" podUID="6f8344a7-8c6e-471d-9509-7fa59b1701de" containerName="cleanup"
Nov 02 03:45:32.733865 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:32.733767    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f8344a7-8c6e-471d-9509-7fa59b1701de" containerName="cleanup"
Nov 02 03:45:32.733865 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:32.733778    4496 cpu_manager.go:414] "containerMap: removing container" podUID="340fca13-ed7e-4312-b0c6-1ee609ab38d4" containerName="cleanup"
Nov 02 03:45:32.733865 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:32.733782    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="340fca13-ed7e-4312-b0c6-1ee609ab38d4" containerName="cleanup"
Nov 02 03:45:32.733865 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:32.733790    4496 cpu_manager.go:414] "containerMap: removing container" podUID="340fca13-ed7e-4312-b0c6-1ee609ab38d4" containerName="cleanup"
Nov 02 03:45:32.733865 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:32.733794    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="340fca13-ed7e-4312-b0c6-1ee609ab38d4" containerName="cleanup"
Nov 02 03:45:32.733865 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:32.733802    4496 cpu_manager.go:414] "containerMap: removing container" podUID="340fca13-ed7e-4312-b0c6-1ee609ab38d4" containerName="cleanup"
Nov 02 03:45:32.733865 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:32.733806    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="340fca13-ed7e-4312-b0c6-1ee609ab38d4" containerName="cleanup"
Nov 02 03:45:32.734661 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:32.733881    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="97a0dd5f-e357-46d3-9084-d698fecba5e6" containerName="cleanup"
Nov 02 03:45:32.734661 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:32.733891    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="97a0dd5f-e357-46d3-9084-d698fecba5e6" containerName="cleanup"
Nov 02 03:45:32.734661 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:32.733900    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ecfde53-c331-4730-b873-397029daeae1" containerName="cleanup"
Nov 02 03:45:32.734661 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:32.733905    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f8344a7-8c6e-471d-9509-7fa59b1701de" containerName="cleanup"
Nov 02 03:45:32.734661 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:32.733911    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e8d8ba0-989d-45b4-8fbc-abe58b26c48c" containerName="cleanup"
Nov 02 03:45:32.734661 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:32.733917    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f8344a7-8c6e-471d-9509-7fa59b1701de" containerName="cleanup"
Nov 02 03:45:32.734661 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:32.733925    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="340fca13-ed7e-4312-b0c6-1ee609ab38d4" containerName="cleanup"
Nov 02 03:45:32.734661 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:32.733929    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="340fca13-ed7e-4312-b0c6-1ee609ab38d4" containerName="cleanup"
Nov 02 03:45:32.734661 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:32.733934    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="340fca13-ed7e-4312-b0c6-1ee609ab38d4" containerName="cleanup"
Nov 02 03:45:32.734661 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:32.733939    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f8344a7-8c6e-471d-9509-7fa59b1701de" containerName="cleanup"
Nov 02 03:45:32.734661 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:32.733946    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ecfde53-c331-4730-b873-397029daeae1" containerName="cleanup"
Nov 02 03:45:32.734661 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:32.733950    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ecfde53-c331-4730-b873-397029daeae1" containerName="cleanup"
Nov 02 03:45:32.734661 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:32.733955    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e8d8ba0-989d-45b4-8fbc-abe58b26c48c" containerName="cleanup"
Nov 02 03:45:32.734661 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:32.733961    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e8d8ba0-989d-45b4-8fbc-abe58b26c48c" containerName="cleanup"
Nov 02 03:45:32.738875 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:32.738855    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ee0fed-66c7bc7579-6b2zf"
Nov 02 03:45:32.796237 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:32.796200    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ee0fed-66c7bc7579-6b2zf"]
Nov 02 03:45:32.809468 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:32.809426    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1955f7a8-3b12-45cd-8059-7d701db7ac37-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-ee0fed-66c7bc7579-6b2zf\" (UID: \"1955f7a8-3b12-45cd-8059-7d701db7ac37\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ee0fed-66c7bc7579-6b2zf"
Nov 02 03:45:32.809632 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:32.809545    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mmkm\" (UniqueName: \"kubernetes.io/projected/1955f7a8-3b12-45cd-8059-7d701db7ac37-kube-api-access-9mmkm\") pod \"payload-pre-processing-e2e-shared-a-w0-ee0fed-66c7bc7579-6b2zf\" (UID: \"1955f7a8-3b12-45cd-8059-7d701db7ac37\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ee0fed-66c7bc7579-6b2zf"
Nov 02 03:45:32.840725 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:32.840686    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-ee0fed\""
Nov 02 03:45:32.849730 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:32.849694    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-ee0fed-c75c4d6c9-fbd9s"]
Nov 02 03:45:32.850704 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:32.850670    4496 cpu_manager.go:414] "containerMap: removing container" podUID="97a0dd5f-e357-46d3-9084-d698fecba5e6" containerName="cleanup"
Nov 02 03:45:32.850704 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:32.850695    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97a0dd5f-e357-46d3-9084-d698fecba5e6" containerName="cleanup"
Nov 02 03:45:32.850919 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:32.850730    4496 cpu_manager.go:414] "containerMap: removing container" podUID="6f8344a7-8c6e-471d-9509-7fa59b1701de" containerName="cleanup"
Nov 02 03:45:32.850919 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:32.850738    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f8344a7-8c6e-471d-9509-7fa59b1701de" containerName="cleanup"
Nov 02 03:45:32.850919 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:32.850906    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="97a0dd5f-e357-46d3-9084-d698fecba5e6" containerName="cleanup"
Nov 02 03:45:32.855912 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:32.855888    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ee0fed-c75c4d6c9-fbd9s"
Nov 02 03:45:32.873527 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:32.873500    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-ee0fed-dockercfg-6pnzx\""
Nov 02 03:45:32.908917 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:32.908872    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-ee0fed-c75c4d6c9-fbd9s"]
Nov 02 03:45:32.910908 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:32.910878    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1955f7a8-3b12-45cd-8059-7d701db7ac37-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-ee0fed-66c7bc7579-6b2zf\" (UID: \"1955f7a8-3b12-45cd-8059-7d701db7ac37\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ee0fed-66c7bc7579-6b2zf"
Nov 02 03:45:32.911041 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:32.910944    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mmkm\" (UniqueName: \"kubernetes.io/projected/1955f7a8-3b12-45cd-8059-7d701db7ac37-kube-api-access-9mmkm\") pod \"payload-pre-processing-e2e-shared-a-w0-ee0fed-66c7bc7579-6b2zf\" (UID: \"1955f7a8-3b12-45cd-8059-7d701db7ac37\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ee0fed-66c7bc7579-6b2zf"
Nov 02 03:45:32.911789 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:32.911765    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1955f7a8-3b12-45cd-8059-7d701db7ac37-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-ee0fed-66c7bc7579-6b2zf\" (UID: \"1955f7a8-3b12-45cd-8059-7d701db7ac37\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ee0fed-66c7bc7579-6b2zf"
Nov 02 03:45:32.969418 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:32.969387    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mmkm\" (UniqueName: \"kubernetes.io/projected/1955f7a8-3b12-45cd-8059-7d701db7ac37-kube-api-access-9mmkm\") pod \"payload-pre-processing-e2e-shared-a-w0-ee0fed-66c7bc7579-6b2zf\" (UID: \"1955f7a8-3b12-45cd-8059-7d701db7ac37\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ee0fed-66c7bc7579-6b2zf"
Nov 02 03:45:33.011743 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:33.011663    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9887dae0-cba1-4148-bce6-936ff305a1a9-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-ee0fed-c75c4d6c9-fbd9s\" (UID: \"9887dae0-cba1-4148-bce6-936ff305a1a9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ee0fed-c75c4d6c9-fbd9s"
Nov 02 03:45:33.011743 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:33.011740    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jl57p\" (UniqueName: \"kubernetes.io/projected/9887dae0-cba1-4148-bce6-936ff305a1a9-kube-api-access-jl57p\") pod \"payload-processing-e2e-shared-a-w0-ee0fed-c75c4d6c9-fbd9s\" (UID: \"9887dae0-cba1-4148-bce6-936ff305a1a9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ee0fed-c75c4d6c9-fbd9s"
Nov 02 03:45:33.057945 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:33.057912    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ee0fed-66c7bc7579-6b2zf"
Nov 02 03:45:33.112639 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:33.112604    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jl57p\" (UniqueName: \"kubernetes.io/projected/9887dae0-cba1-4148-bce6-936ff305a1a9-kube-api-access-jl57p\") pod \"payload-processing-e2e-shared-a-w0-ee0fed-c75c4d6c9-fbd9s\" (UID: \"9887dae0-cba1-4148-bce6-936ff305a1a9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ee0fed-c75c4d6c9-fbd9s"
Nov 02 03:45:33.112821 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:33.112756    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9887dae0-cba1-4148-bce6-936ff305a1a9-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-ee0fed-c75c4d6c9-fbd9s\" (UID: \"9887dae0-cba1-4148-bce6-936ff305a1a9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ee0fed-c75c4d6c9-fbd9s"
Nov 02 03:45:33.113682 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:33.113651    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9887dae0-cba1-4148-bce6-936ff305a1a9-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-ee0fed-c75c4d6c9-fbd9s\" (UID: \"9887dae0-cba1-4148-bce6-936ff305a1a9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ee0fed-c75c4d6c9-fbd9s"
Nov 02 03:45:33.143795 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:33.143760    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-aad578-openshift-default-688f6b55cc-qxqmg"]
Nov 02 03:45:33.149320 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:33.149298    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-aad578-openshift-default-688f6b55cc-qxqmg"
Nov 02 03:45:33.170138 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:33.170106    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jl57p\" (UniqueName: \"kubernetes.io/projected/9887dae0-cba1-4148-bce6-936ff305a1a9-kube-api-access-jl57p\") pod \"payload-processing-e2e-shared-a-w0-ee0fed-c75c4d6c9-fbd9s\" (UID: \"9887dae0-cba1-4148-bce6-936ff305a1a9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ee0fed-c75c4d6c9-fbd9s"
Nov 02 03:45:33.179376 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:33.179110    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ee0fed-c75c4d6c9-fbd9s"
Nov 02 03:45:33.232513 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:33.230851    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-aad578-openshift-default-dockercfg-46rdq\""
Nov 02 03:45:33.237304 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:33.237171    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-aad578-openshift-default-688f6b55cc-qxqmg"]
Nov 02 03:45:33.334525 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:33.334362    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e754d50-d985-4886-9184-102561eb17a8-istio-token\") pod \"e2e-shared-a-w3-aad578-openshift-default-688f6b55cc-qxqmg\" (UID: \"1e754d50-d985-4886-9184-102561eb17a8\") " pod="openshift-ingress/e2e-shared-a-w3-aad578-openshift-default-688f6b55cc-qxqmg"
Nov 02 03:45:33.334525 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:33.334469    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e754d50-d985-4886-9184-102561eb17a8-istiod-ca-cert\") pod \"e2e-shared-a-w3-aad578-openshift-default-688f6b55cc-qxqmg\" (UID: \"1e754d50-d985-4886-9184-102561eb17a8\") " pod="openshift-ingress/e2e-shared-a-w3-aad578-openshift-default-688f6b55cc-qxqmg"
Nov 02 03:45:33.334768 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:33.334564    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e754d50-d985-4886-9184-102561eb17a8-workload-socket\") pod \"e2e-shared-a-w3-aad578-openshift-default-688f6b55cc-qxqmg\" (UID: \"1e754d50-d985-4886-9184-102561eb17a8\") " pod="openshift-ingress/e2e-shared-a-w3-aad578-openshift-default-688f6b55cc-qxqmg"
Nov 02 03:45:33.334810 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:33.334766    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e754d50-d985-4886-9184-102561eb17a8-credential-socket\") pod \"e2e-shared-a-w3-aad578-openshift-default-688f6b55cc-qxqmg\" (UID: \"1e754d50-d985-4886-9184-102561eb17a8\") " pod="openshift-ingress/e2e-shared-a-w3-aad578-openshift-default-688f6b55cc-qxqmg"
Nov 02 03:45:33.334864 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:33.334809    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e754d50-d985-4886-9184-102561eb17a8-istio-data\") pod \"e2e-shared-a-w3-aad578-openshift-default-688f6b55cc-qxqmg\" (UID: \"1e754d50-d985-4886-9184-102561eb17a8\") " pod="openshift-ingress/e2e-shared-a-w3-aad578-openshift-default-688f6b55cc-qxqmg"
Nov 02 03:45:33.334864 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:33.334837    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e754d50-d985-4886-9184-102561eb17a8-istio-podinfo\") pod \"e2e-shared-a-w3-aad578-openshift-default-688f6b55cc-qxqmg\" (UID: \"1e754d50-d985-4886-9184-102561eb17a8\") " pod="openshift-ingress/e2e-shared-a-w3-aad578-openshift-default-688f6b55cc-qxqmg"
Nov 02 03:45:33.334968 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:33.334901    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e754d50-d985-4886-9184-102561eb17a8-istio-envoy\") pod \"e2e-shared-a-w3-aad578-openshift-default-688f6b55cc-qxqmg\" (UID: \"1e754d50-d985-4886-9184-102561eb17a8\") " pod="openshift-ingress/e2e-shared-a-w3-aad578-openshift-default-688f6b55cc-qxqmg"
Nov 02 03:45:33.335063 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:33.335008    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e754d50-d985-4886-9184-102561eb17a8-workload-certs\") pod \"e2e-shared-a-w3-aad578-openshift-default-688f6b55cc-qxqmg\" (UID: \"1e754d50-d985-4886-9184-102561eb17a8\") " pod="openshift-ingress/e2e-shared-a-w3-aad578-openshift-default-688f6b55cc-qxqmg"
Nov 02 03:45:33.335175 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:33.335141    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdjwg\" (UniqueName: \"kubernetes.io/projected/1e754d50-d985-4886-9184-102561eb17a8-kube-api-access-tdjwg\") pod \"e2e-shared-a-w3-aad578-openshift-default-688f6b55cc-qxqmg\" (UID: \"1e754d50-d985-4886-9184-102561eb17a8\") " pod="openshift-ingress/e2e-shared-a-w3-aad578-openshift-default-688f6b55cc-qxqmg"
Nov 02 03:45:33.436807 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:33.436772    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e754d50-d985-4886-9184-102561eb17a8-workload-socket\") pod \"e2e-shared-a-w3-aad578-openshift-default-688f6b55cc-qxqmg\" (UID: \"1e754d50-d985-4886-9184-102561eb17a8\") " pod="openshift-ingress/e2e-shared-a-w3-aad578-openshift-default-688f6b55cc-qxqmg"
Nov 02 03:45:33.436992 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:33.436872    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e754d50-d985-4886-9184-102561eb17a8-credential-socket\") pod \"e2e-shared-a-w3-aad578-openshift-default-688f6b55cc-qxqmg\" (UID: \"1e754d50-d985-4886-9184-102561eb17a8\") " pod="openshift-ingress/e2e-shared-a-w3-aad578-openshift-default-688f6b55cc-qxqmg"
Nov 02 03:45:33.436992 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:33.436900    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e754d50-d985-4886-9184-102561eb17a8-istio-data\") pod \"e2e-shared-a-w3-aad578-openshift-default-688f6b55cc-qxqmg\" (UID: \"1e754d50-d985-4886-9184-102561eb17a8\") " pod="openshift-ingress/e2e-shared-a-w3-aad578-openshift-default-688f6b55cc-qxqmg"
Nov 02 03:45:33.436992 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:33.436921    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e754d50-d985-4886-9184-102561eb17a8-istio-podinfo\") pod \"e2e-shared-a-w3-aad578-openshift-default-688f6b55cc-qxqmg\" (UID: \"1e754d50-d985-4886-9184-102561eb17a8\") " pod="openshift-ingress/e2e-shared-a-w3-aad578-openshift-default-688f6b55cc-qxqmg"
Nov 02 03:45:33.436992 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:33.436961    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e754d50-d985-4886-9184-102561eb17a8-istio-envoy\") pod \"e2e-shared-a-w3-aad578-openshift-default-688f6b55cc-qxqmg\" (UID: \"1e754d50-d985-4886-9184-102561eb17a8\") " pod="openshift-ingress/e2e-shared-a-w3-aad578-openshift-default-688f6b55cc-qxqmg"
Nov 02 03:45:33.436992 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:33.436986    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e754d50-d985-4886-9184-102561eb17a8-workload-certs\") pod \"e2e-shared-a-w3-aad578-openshift-default-688f6b55cc-qxqmg\" (UID: \"1e754d50-d985-4886-9184-102561eb17a8\") " pod="openshift-ingress/e2e-shared-a-w3-aad578-openshift-default-688f6b55cc-qxqmg"
Nov 02 03:45:33.437343 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:33.437074    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdjwg\" (UniqueName: \"kubernetes.io/projected/1e754d50-d985-4886-9184-102561eb17a8-kube-api-access-tdjwg\") pod \"e2e-shared-a-w3-aad578-openshift-default-688f6b55cc-qxqmg\" (UID: \"1e754d50-d985-4886-9184-102561eb17a8\") " pod="openshift-ingress/e2e-shared-a-w3-aad578-openshift-default-688f6b55cc-qxqmg"
Nov 02 03:45:33.437426 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:33.437356    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e754d50-d985-4886-9184-102561eb17a8-workload-socket\") pod \"e2e-shared-a-w3-aad578-openshift-default-688f6b55cc-qxqmg\" (UID: \"1e754d50-d985-4886-9184-102561eb17a8\") " pod="openshift-ingress/e2e-shared-a-w3-aad578-openshift-default-688f6b55cc-qxqmg"
Nov 02 03:45:33.437496 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:33.437433    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e754d50-d985-4886-9184-102561eb17a8-istio-token\") pod \"e2e-shared-a-w3-aad578-openshift-default-688f6b55cc-qxqmg\" (UID: \"1e754d50-d985-4886-9184-102561eb17a8\") " pod="openshift-ingress/e2e-shared-a-w3-aad578-openshift-default-688f6b55cc-qxqmg"
Nov 02 03:45:33.437496 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:33.437444    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e754d50-d985-4886-9184-102561eb17a8-credential-socket\") pod \"e2e-shared-a-w3-aad578-openshift-default-688f6b55cc-qxqmg\" (UID: \"1e754d50-d985-4886-9184-102561eb17a8\") " pod="openshift-ingress/e2e-shared-a-w3-aad578-openshift-default-688f6b55cc-qxqmg"
Nov 02 03:45:33.437637 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:33.437532    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e754d50-d985-4886-9184-102561eb17a8-istiod-ca-cert\") pod \"e2e-shared-a-w3-aad578-openshift-default-688f6b55cc-qxqmg\" (UID: \"1e754d50-d985-4886-9184-102561eb17a8\") " pod="openshift-ingress/e2e-shared-a-w3-aad578-openshift-default-688f6b55cc-qxqmg"
Nov 02 03:45:33.437732 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:33.437712    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e754d50-d985-4886-9184-102561eb17a8-workload-certs\") pod \"e2e-shared-a-w3-aad578-openshift-default-688f6b55cc-qxqmg\" (UID: \"1e754d50-d985-4886-9184-102561eb17a8\") " pod="openshift-ingress/e2e-shared-a-w3-aad578-openshift-default-688f6b55cc-qxqmg"
Nov 02 03:45:33.438114 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:33.438087    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e754d50-d985-4886-9184-102561eb17a8-istio-data\") pod \"e2e-shared-a-w3-aad578-openshift-default-688f6b55cc-qxqmg\" (UID: \"1e754d50-d985-4886-9184-102561eb17a8\") " pod="openshift-ingress/e2e-shared-a-w3-aad578-openshift-default-688f6b55cc-qxqmg"
Nov 02 03:45:33.438377 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:33.438356    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e754d50-d985-4886-9184-102561eb17a8-istiod-ca-cert\") pod \"e2e-shared-a-w3-aad578-openshift-default-688f6b55cc-qxqmg\" (UID: \"1e754d50-d985-4886-9184-102561eb17a8\") " pod="openshift-ingress/e2e-shared-a-w3-aad578-openshift-default-688f6b55cc-qxqmg"
Nov 02 03:45:33.443083 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:33.440691    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e754d50-d985-4886-9184-102561eb17a8-istio-envoy\") pod \"e2e-shared-a-w3-aad578-openshift-default-688f6b55cc-qxqmg\" (UID: \"1e754d50-d985-4886-9184-102561eb17a8\") " pod="openshift-ingress/e2e-shared-a-w3-aad578-openshift-default-688f6b55cc-qxqmg"
Nov 02 03:45:33.443083 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:33.442689    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e754d50-d985-4886-9184-102561eb17a8-istio-podinfo\") pod \"e2e-shared-a-w3-aad578-openshift-default-688f6b55cc-qxqmg\" (UID: \"1e754d50-d985-4886-9184-102561eb17a8\") " pod="openshift-ingress/e2e-shared-a-w3-aad578-openshift-default-688f6b55cc-qxqmg"
Nov 02 03:45:33.446483 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:33.446454    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ee0fed-66c7bc7579-6b2zf"]
Nov 02 03:45:33.448669 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:45:33.448645    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1955f7a8_3b12_45cd_8059_7d701db7ac37.slice/crio-5c302a0091a677bc658523fd3bb14bbea1f7507ffe1767af5a7fadd2b2de0545 WatchSource:0}: Error finding container 5c302a0091a677bc658523fd3bb14bbea1f7507ffe1767af5a7fadd2b2de0545: Status 404 returned error can't find the container with id 5c302a0091a677bc658523fd3bb14bbea1f7507ffe1767af5a7fadd2b2de0545
Nov 02 03:45:33.500223 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:33.500199    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e754d50-d985-4886-9184-102561eb17a8-istio-token\") pod \"e2e-shared-a-w3-aad578-openshift-default-688f6b55cc-qxqmg\" (UID: \"1e754d50-d985-4886-9184-102561eb17a8\") " pod="openshift-ingress/e2e-shared-a-w3-aad578-openshift-default-688f6b55cc-qxqmg"
Nov 02 03:45:33.500349 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:33.500267    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdjwg\" (UniqueName: \"kubernetes.io/projected/1e754d50-d985-4886-9184-102561eb17a8-kube-api-access-tdjwg\") pod \"e2e-shared-a-w3-aad578-openshift-default-688f6b55cc-qxqmg\" (UID: \"1e754d50-d985-4886-9184-102561eb17a8\") " pod="openshift-ingress/e2e-shared-a-w3-aad578-openshift-default-688f6b55cc-qxqmg"
Nov 02 03:45:33.531362 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:45:33.531332    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 03:45:33.618842 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:45:33.618789    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9887dae0_cba1_4148_bce6_936ff305a1a9.slice/crio-9aea61bbe1fdf2aea02c4481fa446d916ca7b2970326f04c8e74bab72752f8f3 WatchSource:0}: Error finding container 9aea61bbe1fdf2aea02c4481fa446d916ca7b2970326f04c8e74bab72752f8f3: Status 404 returned error can't find the container with id 9aea61bbe1fdf2aea02c4481fa446d916ca7b2970326f04c8e74bab72752f8f3
Nov 02 03:45:33.619367 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:33.619343    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-ee0fed-c75c4d6c9-fbd9s"]
Nov 02 03:45:33.764233 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:33.764206    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-aad578-openshift-default-688f6b55cc-qxqmg"
Nov 02 03:45:33.806075 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:33.805998    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ee0fed-c75c4d6c9-fbd9s" event={"ID":"9887dae0-cba1-4148-bce6-936ff305a1a9","Type":"ContainerStarted","Data":"f67649bdc1ce376990d78ef0569898b047ef25db7d1e55cc7797cc1def170f5b"}
Nov 02 03:45:33.806247 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:33.806108    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ee0fed-c75c4d6c9-fbd9s" event={"ID":"9887dae0-cba1-4148-bce6-936ff305a1a9","Type":"ContainerStarted","Data":"9aea61bbe1fdf2aea02c4481fa446d916ca7b2970326f04c8e74bab72752f8f3"}
Nov 02 03:45:33.806311 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:33.806249    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ee0fed-c75c4d6c9-fbd9s"
Nov 02 03:45:33.808950 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:33.808927    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ee0fed-66c7bc7579-6b2zf" event={"ID":"1955f7a8-3b12-45cd-8059-7d701db7ac37","Type":"ContainerStarted","Data":"016ed7e6f9aab48ab13f89b14adc5e8bc05d6f92fff4e21d68b86fd473c48e3e"}
Nov 02 03:45:33.809085 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:33.808964    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ee0fed-66c7bc7579-6b2zf" event={"ID":"1955f7a8-3b12-45cd-8059-7d701db7ac37","Type":"ContainerStarted","Data":"5c302a0091a677bc658523fd3bb14bbea1f7507ffe1767af5a7fadd2b2de0545"}
Nov 02 03:45:33.809226 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:33.809210    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ee0fed-66c7bc7579-6b2zf"
Nov 02 03:45:33.926842 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:33.926723    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ee0fed-c75c4d6c9-fbd9s" podStartSLOduration=1.926705787 podStartE2EDuration="1.926705787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:45:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:45:33.92224509 +0000 UTC m=+3474.500852470" watchObservedRunningTime="2026-11-02 03:45:33.926705787 +0000 UTC m=+3474.505313167"
Nov 02 03:45:34.036973 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:34.036912    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ee0fed-66c7bc7579-6b2zf" podStartSLOduration=2.036897325 podStartE2EDuration="2.036897325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:45:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:45:34.036412897 +0000 UTC m=+3474.615020276" watchObservedRunningTime="2026-11-02 03:45:34.036897325 +0000 UTC m=+3474.615504705"
Nov 02 03:45:34.199732 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:34.199702    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-aad578-openshift-default-688f6b55cc-qxqmg"]
Nov 02 03:45:34.200826 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:45:34.200799    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e754d50_d985_4886_9184_102561eb17a8.slice/crio-6e3d85ec18d018c0a5ea3b0254aa9f77c456a7b3838bc834ab776dd0ee18bd75 WatchSource:0}: Error finding container 6e3d85ec18d018c0a5ea3b0254aa9f77c456a7b3838bc834ab776dd0ee18bd75: Status 404 returned error can't find the container with id 6e3d85ec18d018c0a5ea3b0254aa9f77c456a7b3838bc834ab776dd0ee18bd75
Nov 02 03:45:34.203613 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:34.203575    4496 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 03:45:34.203699 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:34.203653    4496 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 03:45:34.203699 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:34.203694    4496 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 03:45:34.816412 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:34.816375    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-aad578-openshift-default-688f6b55cc-qxqmg" event={"ID":"1e754d50-d985-4886-9184-102561eb17a8","Type":"ContainerStarted","Data":"2c8eac79e4c5e23da4285d294e4910ece44f2aca0053a9a741c669b4c5e04a6b"}
Nov 02 03:45:34.816880 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:34.816431    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-aad578-openshift-default-688f6b55cc-qxqmg" event={"ID":"1e754d50-d985-4886-9184-102561eb17a8","Type":"ContainerStarted","Data":"6e3d85ec18d018c0a5ea3b0254aa9f77c456a7b3838bc834ab776dd0ee18bd75"}
Nov 02 03:45:34.936741 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:34.936670    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-aad578-openshift-default-688f6b55cc-qxqmg" podStartSLOduration=1.936649767 podStartE2EDuration="1.936649767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:45:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:45:34.934549176 +0000 UTC m=+3475.513156565" watchObservedRunningTime="2026-11-02 03:45:34.936649767 +0000 UTC m=+3475.515257148"
Nov 02 03:45:35.765259 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:35.765216    4496 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-aad578-openshift-default-688f6b55cc-qxqmg"
Nov 02 03:45:35.771924 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:35.771897    4496 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-aad578-openshift-default-688f6b55cc-qxqmg"
Nov 02 03:45:35.824394 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:35.824368    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-aad578-openshift-default-688f6b55cc-qxqmg"
Nov 02 03:45:35.825218 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:35.825198    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-aad578-openshift-default-688f6b55cc-qxqmg"
Nov 02 03:45:37.542410 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:37.542373    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a7cf93-7b78677459-fvrpv"]
Nov 02 03:45:37.547807 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:37.547780    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a7cf93-7b78677459-fvrpv"
Nov 02 03:45:37.586422 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:37.586386    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9lsx\" (UniqueName: \"kubernetes.io/projected/8ea63372-147c-486f-99d4-5d2a5390db50-kube-api-access-n9lsx\") pod \"payload-pre-processing-e2e-shared-a-w4-a7cf93-7b78677459-fvrpv\" (UID: \"8ea63372-147c-486f-99d4-5d2a5390db50\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a7cf93-7b78677459-fvrpv"
Nov 02 03:45:37.586574 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:37.586431    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ea63372-147c-486f-99d4-5d2a5390db50-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-a7cf93-7b78677459-fvrpv\" (UID: \"8ea63372-147c-486f-99d4-5d2a5390db50\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a7cf93-7b78677459-fvrpv"
Nov 02 03:45:37.599547 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:37.599520    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a7cf93-7b78677459-fvrpv"]
Nov 02 03:45:37.646489 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:37.646451    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-a7cf93\""
Nov 02 03:45:37.646703 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:37.646584    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-a7cf93-dockercfg-4hg92\""
Nov 02 03:45:37.687873 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:37.687833    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9lsx\" (UniqueName: \"kubernetes.io/projected/8ea63372-147c-486f-99d4-5d2a5390db50-kube-api-access-n9lsx\") pod \"payload-pre-processing-e2e-shared-a-w4-a7cf93-7b78677459-fvrpv\" (UID: \"8ea63372-147c-486f-99d4-5d2a5390db50\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a7cf93-7b78677459-fvrpv"
Nov 02 03:45:37.688108 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:37.687887    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ea63372-147c-486f-99d4-5d2a5390db50-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-a7cf93-7b78677459-fvrpv\" (UID: \"8ea63372-147c-486f-99d4-5d2a5390db50\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a7cf93-7b78677459-fvrpv"
Nov 02 03:45:37.688672 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:37.688646    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ea63372-147c-486f-99d4-5d2a5390db50-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-a7cf93-7b78677459-fvrpv\" (UID: \"8ea63372-147c-486f-99d4-5d2a5390db50\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a7cf93-7b78677459-fvrpv"
Nov 02 03:45:37.695614 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:37.695584    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-a7cf93-859f5d856c-qgchk"]
Nov 02 03:45:37.701545 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:37.701517    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-a7cf93-859f5d856c-qgchk"
Nov 02 03:45:37.745150 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:37.745119    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9lsx\" (UniqueName: \"kubernetes.io/projected/8ea63372-147c-486f-99d4-5d2a5390db50-kube-api-access-n9lsx\") pod \"payload-pre-processing-e2e-shared-a-w4-a7cf93-7b78677459-fvrpv\" (UID: \"8ea63372-147c-486f-99d4-5d2a5390db50\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a7cf93-7b78677459-fvrpv"
Nov 02 03:45:37.755061 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:37.754989    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-a7cf93-859f5d856c-qgchk"]
Nov 02 03:45:37.789218 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:37.789186    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rb89v\" (UniqueName: \"kubernetes.io/projected/ab5ecfc3-592a-47ca-8acd-000d982544d5-kube-api-access-rb89v\") pod \"payload-processing-e2e-shared-a-w4-a7cf93-859f5d856c-qgchk\" (UID: \"ab5ecfc3-592a-47ca-8acd-000d982544d5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-a7cf93-859f5d856c-qgchk"
Nov 02 03:45:37.789377 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:37.789229    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab5ecfc3-592a-47ca-8acd-000d982544d5-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-a7cf93-859f5d856c-qgchk\" (UID: \"ab5ecfc3-592a-47ca-8acd-000d982544d5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-a7cf93-859f5d856c-qgchk"
Nov 02 03:45:37.863973 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:37.863892    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a7cf93-7b78677459-fvrpv"
Nov 02 03:45:37.891097 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:37.891062    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rb89v\" (UniqueName: \"kubernetes.io/projected/ab5ecfc3-592a-47ca-8acd-000d982544d5-kube-api-access-rb89v\") pod \"payload-processing-e2e-shared-a-w4-a7cf93-859f5d856c-qgchk\" (UID: \"ab5ecfc3-592a-47ca-8acd-000d982544d5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-a7cf93-859f5d856c-qgchk"
Nov 02 03:45:37.891256 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:37.891143    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab5ecfc3-592a-47ca-8acd-000d982544d5-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-a7cf93-859f5d856c-qgchk\" (UID: \"ab5ecfc3-592a-47ca-8acd-000d982544d5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-a7cf93-859f5d856c-qgchk"
Nov 02 03:45:37.891757 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:37.891739    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab5ecfc3-592a-47ca-8acd-000d982544d5-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-a7cf93-859f5d856c-qgchk\" (UID: \"ab5ecfc3-592a-47ca-8acd-000d982544d5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-a7cf93-859f5d856c-qgchk"
Nov 02 03:45:37.948732 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:37.948695    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rb89v\" (UniqueName: \"kubernetes.io/projected/ab5ecfc3-592a-47ca-8acd-000d982544d5-kube-api-access-rb89v\") pod \"payload-processing-e2e-shared-a-w4-a7cf93-859f5d856c-qgchk\" (UID: \"ab5ecfc3-592a-47ca-8acd-000d982544d5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-a7cf93-859f5d856c-qgchk"
Nov 02 03:45:38.016367 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:38.016331    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-a7cf93-859f5d856c-qgchk"
Nov 02 03:45:38.241573 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:45:38.241530    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ea63372_147c_486f_99d4_5d2a5390db50.slice/crio-a227278b5878dd053e4e454021268e845a9851de77b382f513aec54fa930bf01 WatchSource:0}: Error finding container a227278b5878dd053e4e454021268e845a9851de77b382f513aec54fa930bf01: Status 404 returned error can't find the container with id a227278b5878dd053e4e454021268e845a9851de77b382f513aec54fa930bf01
Nov 02 03:45:38.242842 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:38.242810    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a7cf93-7b78677459-fvrpv"]
Nov 02 03:45:38.399188 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:45:38.399158    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab5ecfc3_592a_47ca_8acd_000d982544d5.slice/crio-b8c178eac4a11f3cf48d4b852928b5a5d4001bf22962eb913d68d80a97765aea WatchSource:0}: Error finding container b8c178eac4a11f3cf48d4b852928b5a5d4001bf22962eb913d68d80a97765aea: Status 404 returned error can't find the container with id b8c178eac4a11f3cf48d4b852928b5a5d4001bf22962eb913d68d80a97765aea
Nov 02 03:45:38.399302 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:38.399242    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-a7cf93-859f5d856c-qgchk"]
Nov 02 03:45:38.848615 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:38.848575    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a7cf93-7b78677459-fvrpv" event={"ID":"8ea63372-147c-486f-99d4-5d2a5390db50","Type":"ContainerStarted","Data":"90c9b4d908742ce3d2ba56515696ae5d598541b0e6fe318587bead83fc34efff"}
Nov 02 03:45:38.849079 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:38.848675    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a7cf93-7b78677459-fvrpv" event={"ID":"8ea63372-147c-486f-99d4-5d2a5390db50","Type":"ContainerStarted","Data":"a227278b5878dd053e4e454021268e845a9851de77b382f513aec54fa930bf01"}
Nov 02 03:45:38.849079 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:38.848711    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a7cf93-7b78677459-fvrpv"
Nov 02 03:45:38.850599 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:38.850567    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-a7cf93-859f5d856c-qgchk" event={"ID":"ab5ecfc3-592a-47ca-8acd-000d982544d5","Type":"ContainerStarted","Data":"26fea63cd9c0d4c790fc0e2f4cabc35cfd11907f32dc291eb15159ada15992f3"}
Nov 02 03:45:38.850730 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:38.850615    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-a7cf93-859f5d856c-qgchk" event={"ID":"ab5ecfc3-592a-47ca-8acd-000d982544d5","Type":"ContainerStarted","Data":"b8c178eac4a11f3cf48d4b852928b5a5d4001bf22962eb913d68d80a97765aea"}
Nov 02 03:45:38.850730 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:38.850656    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-a7cf93-859f5d856c-qgchk"
Nov 02 03:45:38.966667 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:38.966612    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a7cf93-7b78677459-fvrpv" podStartSLOduration=1.966597432 podStartE2EDuration="1.966597432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:45:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:45:38.963626241 +0000 UTC m=+3479.542233624" watchObservedRunningTime="2026-11-02 03:45:38.966597432 +0000 UTC m=+3479.545204813"
Nov 02 03:45:39.081380 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:39.081314    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-a7cf93-859f5d856c-qgchk" podStartSLOduration=2.081295351 podStartE2EDuration="2.081295351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:45:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:45:39.077767155 +0000 UTC m=+3479.656374534" watchObservedRunningTime="2026-11-02 03:45:39.081295351 +0000 UTC m=+3479.659902733"
Nov 02 03:45:44.569214 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:44.569174    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7c1bee-694764dc5-zrthf"]
Nov 02 03:45:44.575589 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:44.575567    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7c1bee-694764dc5-zrthf"
Nov 02 03:45:44.629410 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:44.629373    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7c1bee-694764dc5-zrthf"]
Nov 02 03:45:44.679459 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:44.679397    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-7c1bee\""
Nov 02 03:45:44.679670 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:44.679486    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-7c1bee-dockercfg-gvw84\""
Nov 02 03:45:44.681330 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:44.680982    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2046cb9f-964f-45f4-9f8a-e427b5818ffc-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-7c1bee-694764dc5-zrthf\" (UID: \"2046cb9f-964f-45f4-9f8a-e427b5818ffc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7c1bee-694764dc5-zrthf"
Nov 02 03:45:44.681330 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:44.681155    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d755f\" (UniqueName: \"kubernetes.io/projected/2046cb9f-964f-45f4-9f8a-e427b5818ffc-kube-api-access-d755f\") pod \"payload-pre-processing-e2e-shared-a-w1-7c1bee-694764dc5-zrthf\" (UID: \"2046cb9f-964f-45f4-9f8a-e427b5818ffc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7c1bee-694764dc5-zrthf"
Nov 02 03:45:44.733711 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:44.733650    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-7c1bee-84b77485bf-9bk69"]
Nov 02 03:45:44.737906 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:44.737887    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-7c1bee-84b77485bf-9bk69"
Nov 02 03:45:44.782282 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:44.782248    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d147ea36-e9b2-42de-ac31-de7c5e03fe51-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-7c1bee-84b77485bf-9bk69\" (UID: \"d147ea36-e9b2-42de-ac31-de7c5e03fe51\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-7c1bee-84b77485bf-9bk69"
Nov 02 03:45:44.782431 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:44.782303    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d755f\" (UniqueName: \"kubernetes.io/projected/2046cb9f-964f-45f4-9f8a-e427b5818ffc-kube-api-access-d755f\") pod \"payload-pre-processing-e2e-shared-a-w1-7c1bee-694764dc5-zrthf\" (UID: \"2046cb9f-964f-45f4-9f8a-e427b5818ffc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7c1bee-694764dc5-zrthf"
Nov 02 03:45:44.782549 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:44.782516    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2046cb9f-964f-45f4-9f8a-e427b5818ffc-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-7c1bee-694764dc5-zrthf\" (UID: \"2046cb9f-964f-45f4-9f8a-e427b5818ffc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7c1bee-694764dc5-zrthf"
Nov 02 03:45:44.782673 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:44.782566    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4lss\" (UniqueName: \"kubernetes.io/projected/d147ea36-e9b2-42de-ac31-de7c5e03fe51-kube-api-access-w4lss\") pod \"payload-processing-e2e-shared-a-w1-7c1bee-84b77485bf-9bk69\" (UID: \"d147ea36-e9b2-42de-ac31-de7c5e03fe51\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-7c1bee-84b77485bf-9bk69"
Nov 02 03:45:44.783927 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:44.783881    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2046cb9f-964f-45f4-9f8a-e427b5818ffc-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-7c1bee-694764dc5-zrthf\" (UID: \"2046cb9f-964f-45f4-9f8a-e427b5818ffc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7c1bee-694764dc5-zrthf"
Nov 02 03:45:44.786449 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:44.786424    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-7c1bee-84b77485bf-9bk69"]
Nov 02 03:45:44.818947 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:44.818929    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ee0fed-66c7bc7579-6b2zf"
Nov 02 03:45:44.819081 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:44.818982    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ee0fed-c75c4d6c9-fbd9s"
Nov 02 03:45:44.840511 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:44.840442    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d755f\" (UniqueName: \"kubernetes.io/projected/2046cb9f-964f-45f4-9f8a-e427b5818ffc-kube-api-access-d755f\") pod \"payload-pre-processing-e2e-shared-a-w1-7c1bee-694764dc5-zrthf\" (UID: \"2046cb9f-964f-45f4-9f8a-e427b5818ffc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7c1bee-694764dc5-zrthf"
Nov 02 03:45:44.883386 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:44.883359    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4lss\" (UniqueName: \"kubernetes.io/projected/d147ea36-e9b2-42de-ac31-de7c5e03fe51-kube-api-access-w4lss\") pod \"payload-processing-e2e-shared-a-w1-7c1bee-84b77485bf-9bk69\" (UID: \"d147ea36-e9b2-42de-ac31-de7c5e03fe51\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-7c1bee-84b77485bf-9bk69"
Nov 02 03:45:44.883707 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:44.883675    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d147ea36-e9b2-42de-ac31-de7c5e03fe51-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-7c1bee-84b77485bf-9bk69\" (UID: \"d147ea36-e9b2-42de-ac31-de7c5e03fe51\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-7c1bee-84b77485bf-9bk69"
Nov 02 03:45:44.884335 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:44.884311    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d147ea36-e9b2-42de-ac31-de7c5e03fe51-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-7c1bee-84b77485bf-9bk69\" (UID: \"d147ea36-e9b2-42de-ac31-de7c5e03fe51\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-7c1bee-84b77485bf-9bk69"
Nov 02 03:45:44.889054 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:44.889007    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7c1bee-694764dc5-zrthf"
Nov 02 03:45:44.941553 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:44.941520    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4lss\" (UniqueName: \"kubernetes.io/projected/d147ea36-e9b2-42de-ac31-de7c5e03fe51-kube-api-access-w4lss\") pod \"payload-processing-e2e-shared-a-w1-7c1bee-84b77485bf-9bk69\" (UID: \"d147ea36-e9b2-42de-ac31-de7c5e03fe51\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-7c1bee-84b77485bf-9bk69"
Nov 02 03:45:45.052452 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:45.052406    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-7c1bee-84b77485bf-9bk69"
Nov 02 03:45:45.274520 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:45.274351    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7c1bee-694764dc5-zrthf"]
Nov 02 03:45:45.277622 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:45:45.277592    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2046cb9f_964f_45f4_9f8a_e427b5818ffc.slice/crio-59b7f28ae6ec6681da5789fca6635b388417b370e87085c607b566f63089d392 WatchSource:0}: Error finding container 59b7f28ae6ec6681da5789fca6635b388417b370e87085c607b566f63089d392: Status 404 returned error can't find the container with id 59b7f28ae6ec6681da5789fca6635b388417b370e87085c607b566f63089d392
Nov 02 03:45:45.431779 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:45.431744    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-7c1bee-84b77485bf-9bk69"]
Nov 02 03:45:45.432951 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:45:45.432912    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd147ea36_e9b2_42de_ac31_de7c5e03fe51.slice/crio-ffc55630fa4142853775e386bce25e5e8b7423082327f7471dea9f8fb44286be WatchSource:0}: Error finding container ffc55630fa4142853775e386bce25e5e8b7423082327f7471dea9f8fb44286be: Status 404 returned error can't find the container with id ffc55630fa4142853775e386bce25e5e8b7423082327f7471dea9f8fb44286be
Nov 02 03:45:45.905181 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:45.905145    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-7c1bee-84b77485bf-9bk69" event={"ID":"d147ea36-e9b2-42de-ac31-de7c5e03fe51","Type":"ContainerStarted","Data":"033f92905c6e23818d5fd5699de333f7c23cd44452c99a0e5cf3f03f30d8bbd3"}
Nov 02 03:45:45.905634 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:45.905210    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-7c1bee-84b77485bf-9bk69"
Nov 02 03:45:45.905634 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:45.905236    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-7c1bee-84b77485bf-9bk69" event={"ID":"d147ea36-e9b2-42de-ac31-de7c5e03fe51","Type":"ContainerStarted","Data":"ffc55630fa4142853775e386bce25e5e8b7423082327f7471dea9f8fb44286be"}
Nov 02 03:45:45.906965 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:45.906933    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7c1bee-694764dc5-zrthf" event={"ID":"2046cb9f-964f-45f4-9f8a-e427b5818ffc","Type":"ContainerStarted","Data":"82cb4bb8139daaf6a20153fc19c23fc0243490b51055519ba1344df219fbd0d5"}
Nov 02 03:45:45.907103 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:45.906987    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7c1bee-694764dc5-zrthf" event={"ID":"2046cb9f-964f-45f4-9f8a-e427b5818ffc","Type":"ContainerStarted","Data":"59b7f28ae6ec6681da5789fca6635b388417b370e87085c607b566f63089d392"}
Nov 02 03:45:45.907103 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:45.907036    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7c1bee-694764dc5-zrthf"
Nov 02 03:45:46.023609 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:46.023555    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-7c1bee-84b77485bf-9bk69" podStartSLOduration=2.023540975 podStartE2EDuration="2.023540975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:45:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:45:46.021212146 +0000 UTC m=+3486.599819526" watchObservedRunningTime="2026-11-02 03:45:46.023540975 +0000 UTC m=+3486.602148355"
Nov 02 03:45:46.139059 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:46.138967    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7c1bee-694764dc5-zrthf" podStartSLOduration=2.13894907 podStartE2EDuration="2.13894907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:45:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:45:46.134625051 +0000 UTC m=+3486.713232432" watchObservedRunningTime="2026-11-02 03:45:46.13894907 +0000 UTC m=+3486.717556450"
Nov 02 03:45:47.443184 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:47.443143    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-t8dbv"]
Nov 02 03:45:47.447582 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:47.447556    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-t8dbv"
Nov 02 03:45:47.500591 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:47.500518    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-858236-openshift-default-dockercfg-79wxp\""
Nov 02 03:45:47.515285 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:47.515254    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e8b61fe-99dc-4705-ab37-cde3bfad0451-credential-socket\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-t8dbv\" (UID: \"4e8b61fe-99dc-4705-ab37-cde3bfad0451\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-t8dbv"
Nov 02 03:45:47.515450 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:47.515434    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e8b61fe-99dc-4705-ab37-cde3bfad0451-workload-certs\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-t8dbv\" (UID: \"4e8b61fe-99dc-4705-ab37-cde3bfad0451\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-t8dbv"
Nov 02 03:45:47.515504 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:47.515476    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e8b61fe-99dc-4705-ab37-cde3bfad0451-istio-podinfo\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-t8dbv\" (UID: \"4e8b61fe-99dc-4705-ab37-cde3bfad0451\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-t8dbv"
Nov 02 03:45:47.515629 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:47.515608    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e8b61fe-99dc-4705-ab37-cde3bfad0451-istiod-ca-cert\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-t8dbv\" (UID: \"4e8b61fe-99dc-4705-ab37-cde3bfad0451\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-t8dbv"
Nov 02 03:45:47.515696 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:47.515650    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e8b61fe-99dc-4705-ab37-cde3bfad0451-istio-token\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-t8dbv\" (UID: \"4e8b61fe-99dc-4705-ab37-cde3bfad0451\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-t8dbv"
Nov 02 03:45:47.515765 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:47.515726    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e8b61fe-99dc-4705-ab37-cde3bfad0451-istio-envoy\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-t8dbv\" (UID: \"4e8b61fe-99dc-4705-ab37-cde3bfad0451\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-t8dbv"
Nov 02 03:45:47.515811 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:47.515773    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e8b61fe-99dc-4705-ab37-cde3bfad0451-workload-socket\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-t8dbv\" (UID: \"4e8b61fe-99dc-4705-ab37-cde3bfad0451\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-t8dbv"
Nov 02 03:45:47.515811 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:47.515798    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e8b61fe-99dc-4705-ab37-cde3bfad0451-istio-data\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-t8dbv\" (UID: \"4e8b61fe-99dc-4705-ab37-cde3bfad0451\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-t8dbv"
Nov 02 03:45:47.515891 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:47.515830    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5b9s\" (UniqueName: \"kubernetes.io/projected/4e8b61fe-99dc-4705-ab37-cde3bfad0451-kube-api-access-k5b9s\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-t8dbv\" (UID: \"4e8b61fe-99dc-4705-ab37-cde3bfad0451\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-t8dbv"
Nov 02 03:45:47.579713 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:47.579672    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-t8dbv"]
Nov 02 03:45:47.617381 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:47.617326    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e8b61fe-99dc-4705-ab37-cde3bfad0451-workload-certs\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-t8dbv\" (UID: \"4e8b61fe-99dc-4705-ab37-cde3bfad0451\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-t8dbv"
Nov 02 03:45:47.617592 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:47.617439    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e8b61fe-99dc-4705-ab37-cde3bfad0451-istio-podinfo\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-t8dbv\" (UID: \"4e8b61fe-99dc-4705-ab37-cde3bfad0451\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-t8dbv"
Nov 02 03:45:47.617592 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:47.617507    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e8b61fe-99dc-4705-ab37-cde3bfad0451-istiod-ca-cert\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-t8dbv\" (UID: \"4e8b61fe-99dc-4705-ab37-cde3bfad0451\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-t8dbv"
Nov 02 03:45:47.617592 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:47.617555    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e8b61fe-99dc-4705-ab37-cde3bfad0451-istio-token\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-t8dbv\" (UID: \"4e8b61fe-99dc-4705-ab37-cde3bfad0451\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-t8dbv"
Nov 02 03:45:47.617772 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:47.617593    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e8b61fe-99dc-4705-ab37-cde3bfad0451-istio-envoy\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-t8dbv\" (UID: \"4e8b61fe-99dc-4705-ab37-cde3bfad0451\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-t8dbv"
Nov 02 03:45:47.617772 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:47.617623    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e8b61fe-99dc-4705-ab37-cde3bfad0451-workload-socket\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-t8dbv\" (UID: \"4e8b61fe-99dc-4705-ab37-cde3bfad0451\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-t8dbv"
Nov 02 03:45:47.617772 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:47.617667    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e8b61fe-99dc-4705-ab37-cde3bfad0451-istio-data\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-t8dbv\" (UID: \"4e8b61fe-99dc-4705-ab37-cde3bfad0451\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-t8dbv"
Nov 02 03:45:47.617772 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:47.617734    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5b9s\" (UniqueName: \"kubernetes.io/projected/4e8b61fe-99dc-4705-ab37-cde3bfad0451-kube-api-access-k5b9s\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-t8dbv\" (UID: \"4e8b61fe-99dc-4705-ab37-cde3bfad0451\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-t8dbv"
Nov 02 03:45:47.617997 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:47.617780    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e8b61fe-99dc-4705-ab37-cde3bfad0451-credential-socket\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-t8dbv\" (UID: \"4e8b61fe-99dc-4705-ab37-cde3bfad0451\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-t8dbv"
Nov 02 03:45:47.617997 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:47.617830    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e8b61fe-99dc-4705-ab37-cde3bfad0451-workload-certs\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-t8dbv\" (UID: \"4e8b61fe-99dc-4705-ab37-cde3bfad0451\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-t8dbv"
Nov 02 03:45:47.618190 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:47.618127    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e8b61fe-99dc-4705-ab37-cde3bfad0451-istio-data\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-t8dbv\" (UID: \"4e8b61fe-99dc-4705-ab37-cde3bfad0451\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-t8dbv"
Nov 02 03:45:47.618322 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:47.618295    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e8b61fe-99dc-4705-ab37-cde3bfad0451-istiod-ca-cert\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-t8dbv\" (UID: \"4e8b61fe-99dc-4705-ab37-cde3bfad0451\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-t8dbv"
Nov 02 03:45:47.618395 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:47.618343    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e8b61fe-99dc-4705-ab37-cde3bfad0451-workload-socket\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-t8dbv\" (UID: \"4e8b61fe-99dc-4705-ab37-cde3bfad0451\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-t8dbv"
Nov 02 03:45:47.618460 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:47.618420    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e8b61fe-99dc-4705-ab37-cde3bfad0451-credential-socket\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-t8dbv\" (UID: \"4e8b61fe-99dc-4705-ab37-cde3bfad0451\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-t8dbv"
Nov 02 03:45:47.620133 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:47.620102    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e8b61fe-99dc-4705-ab37-cde3bfad0451-istio-envoy\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-t8dbv\" (UID: \"4e8b61fe-99dc-4705-ab37-cde3bfad0451\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-t8dbv"
Nov 02 03:45:47.620400 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:47.620382    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e8b61fe-99dc-4705-ab37-cde3bfad0451-istio-podinfo\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-t8dbv\" (UID: \"4e8b61fe-99dc-4705-ab37-cde3bfad0451\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-t8dbv"
Nov 02 03:45:47.675850 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:47.675812    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5b9s\" (UniqueName: \"kubernetes.io/projected/4e8b61fe-99dc-4705-ab37-cde3bfad0451-kube-api-access-k5b9s\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-t8dbv\" (UID: \"4e8b61fe-99dc-4705-ab37-cde3bfad0451\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-t8dbv"
Nov 02 03:45:47.675997 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:47.675913    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e8b61fe-99dc-4705-ab37-cde3bfad0451-istio-token\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-t8dbv\" (UID: \"4e8b61fe-99dc-4705-ab37-cde3bfad0451\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-t8dbv"
Nov 02 03:45:47.762845 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:47.762811    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-t8dbv"
Nov 02 03:45:48.204328 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:48.204294    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-t8dbv"]
Nov 02 03:45:48.205516 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:45:48.205486    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e8b61fe_99dc_4705_ab37_cde3bfad0451.slice/crio-da1334dee55b29a96f10536df49e28768532723642d95f7dce3a1accdd99ec94 WatchSource:0}: Error finding container da1334dee55b29a96f10536df49e28768532723642d95f7dce3a1accdd99ec94: Status 404 returned error can't find the container with id da1334dee55b29a96f10536df49e28768532723642d95f7dce3a1accdd99ec94
Nov 02 03:45:48.208239 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:48.208205    4496 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 03:45:48.208334 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:48.208286    4496 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 03:45:48.208334 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:48.208316    4496 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 03:45:48.928612 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:48.928575    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-t8dbv" event={"ID":"4e8b61fe-99dc-4705-ab37-cde3bfad0451","Type":"ContainerStarted","Data":"af4a995aadfaa406459d1502407866a0b7058ce358bdb7b56f0cfacc0aa1919c"}
Nov 02 03:45:48.929038 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:48.928635    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-t8dbv" event={"ID":"4e8b61fe-99dc-4705-ab37-cde3bfad0451","Type":"ContainerStarted","Data":"da1334dee55b29a96f10536df49e28768532723642d95f7dce3a1accdd99ec94"}
Nov 02 03:45:49.079885 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:49.079826    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-t8dbv" podStartSLOduration=2.079812604 podStartE2EDuration="2.079812604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:45:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:45:49.076969003 +0000 UTC m=+3489.655576383" watchObservedRunningTime="2026-11-02 03:45:49.079812604 +0000 UTC m=+3489.658419983"
Nov 02 03:45:49.300665 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:49.300626    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-aad578-c96769d99-2zkf4"]
Nov 02 03:45:49.304908 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:49.304888    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-aad578-c96769d99-2zkf4"
Nov 02 03:45:49.368039 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:49.359567    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-aad578-dockercfg-wnssw\""
Nov 02 03:45:49.368039 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:49.362762    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-aad578\""
Nov 02 03:45:49.372041 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:49.371362    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-aad578-c96769d99-2zkf4"]
Nov 02 03:45:49.385897 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:49.383671    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-294zk\" (UniqueName: \"kubernetes.io/projected/54092503-5ffa-4b0c-9e72-520195aa8966-kube-api-access-294zk\") pod \"payload-pre-processing-e2e-shared-a-w3-aad578-c96769d99-2zkf4\" (UID: \"54092503-5ffa-4b0c-9e72-520195aa8966\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-aad578-c96769d99-2zkf4"
Nov 02 03:45:49.385897 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:49.383812    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54092503-5ffa-4b0c-9e72-520195aa8966-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-aad578-c96769d99-2zkf4\" (UID: \"54092503-5ffa-4b0c-9e72-520195aa8966\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-aad578-c96769d99-2zkf4"
Nov 02 03:45:49.481703 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:49.481652    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-aad578-6d6755f65f-5ljdx"]
Nov 02 03:45:49.487050 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:49.485722    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-294zk\" (UniqueName: \"kubernetes.io/projected/54092503-5ffa-4b0c-9e72-520195aa8966-kube-api-access-294zk\") pod \"payload-pre-processing-e2e-shared-a-w3-aad578-c96769d99-2zkf4\" (UID: \"54092503-5ffa-4b0c-9e72-520195aa8966\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-aad578-c96769d99-2zkf4"
Nov 02 03:45:49.487050 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:49.485772    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54092503-5ffa-4b0c-9e72-520195aa8966-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-aad578-c96769d99-2zkf4\" (UID: \"54092503-5ffa-4b0c-9e72-520195aa8966\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-aad578-c96769d99-2zkf4"
Nov 02 03:45:49.487050 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:49.486564    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54092503-5ffa-4b0c-9e72-520195aa8966-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-aad578-c96769d99-2zkf4\" (UID: \"54092503-5ffa-4b0c-9e72-520195aa8966\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-aad578-c96769d99-2zkf4"
Nov 02 03:45:49.488543 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:49.488519    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-aad578-6d6755f65f-5ljdx"
Nov 02 03:45:49.522260 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:49.522228    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-aad578-6d6755f65f-5ljdx"]
Nov 02 03:45:49.574763 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:49.574673    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-294zk\" (UniqueName: \"kubernetes.io/projected/54092503-5ffa-4b0c-9e72-520195aa8966-kube-api-access-294zk\") pod \"payload-pre-processing-e2e-shared-a-w3-aad578-c96769d99-2zkf4\" (UID: \"54092503-5ffa-4b0c-9e72-520195aa8966\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-aad578-c96769d99-2zkf4"
Nov 02 03:45:49.588045 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:49.587470    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19637a1f-28d0-4615-abec-aa4ee3f20a4b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-aad578-6d6755f65f-5ljdx\" (UID: \"19637a1f-28d0-4615-abec-aa4ee3f20a4b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-aad578-6d6755f65f-5ljdx"
Nov 02 03:45:49.588045 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:49.587591    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86t5f\" (UniqueName: \"kubernetes.io/projected/19637a1f-28d0-4615-abec-aa4ee3f20a4b-kube-api-access-86t5f\") pod \"payload-processing-e2e-shared-a-w3-aad578-6d6755f65f-5ljdx\" (UID: \"19637a1f-28d0-4615-abec-aa4ee3f20a4b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-aad578-6d6755f65f-5ljdx"
Nov 02 03:45:49.622554 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:49.622524    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-aad578-c96769d99-2zkf4"
Nov 02 03:45:49.689596 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:49.689557    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19637a1f-28d0-4615-abec-aa4ee3f20a4b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-aad578-6d6755f65f-5ljdx\" (UID: \"19637a1f-28d0-4615-abec-aa4ee3f20a4b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-aad578-6d6755f65f-5ljdx"
Nov 02 03:45:49.689763 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:49.689621    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86t5f\" (UniqueName: \"kubernetes.io/projected/19637a1f-28d0-4615-abec-aa4ee3f20a4b-kube-api-access-86t5f\") pod \"payload-processing-e2e-shared-a-w3-aad578-6d6755f65f-5ljdx\" (UID: \"19637a1f-28d0-4615-abec-aa4ee3f20a4b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-aad578-6d6755f65f-5ljdx"
Nov 02 03:45:49.690745 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:49.690717    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19637a1f-28d0-4615-abec-aa4ee3f20a4b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-aad578-6d6755f65f-5ljdx\" (UID: \"19637a1f-28d0-4615-abec-aa4ee3f20a4b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-aad578-6d6755f65f-5ljdx"
Nov 02 03:45:49.771125 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:49.770628    4496 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-t8dbv"
Nov 02 03:45:49.771125 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:49.770790    4496 patch_prober.go:29] interesting pod/e2e-worker-w3-858236-openshift-default-74fcd64bf5-t8dbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" start-of-body=
Nov 02 03:45:49.771125 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:49.770831    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-t8dbv" podUID="4e8b61fe-99dc-4705-ab37-cde3bfad0451" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused"
Nov 02 03:45:49.773503 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:49.773472    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86t5f\" (UniqueName: \"kubernetes.io/projected/19637a1f-28d0-4615-abec-aa4ee3f20a4b-kube-api-access-86t5f\") pod \"payload-processing-e2e-shared-a-w3-aad578-6d6755f65f-5ljdx\" (UID: \"19637a1f-28d0-4615-abec-aa4ee3f20a4b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-aad578-6d6755f65f-5ljdx"
Nov 02 03:45:49.807381 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:49.807350    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-aad578-6d6755f65f-5ljdx"
Nov 02 03:45:49.873584 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:49.873499    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a7cf93-7b78677459-fvrpv"
Nov 02 03:45:49.873774 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:49.873590    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-a7cf93-859f5d856c-qgchk"
Nov 02 03:45:50.072622 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:45:50.072592    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54092503_5ffa_4b0c_9e72_520195aa8966.slice/crio-a21b49e9d7019139c4cb31774e5f6fe246ad69e98677ae048e0d1cb0562331b1 WatchSource:0}: Error finding container a21b49e9d7019139c4cb31774e5f6fe246ad69e98677ae048e0d1cb0562331b1: Status 404 returned error can't find the container with id a21b49e9d7019139c4cb31774e5f6fe246ad69e98677ae048e0d1cb0562331b1
Nov 02 03:45:50.074329 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:50.074299    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-aad578-c96769d99-2zkf4"]
Nov 02 03:45:50.225453 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:50.225424    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-aad578-6d6755f65f-5ljdx"]
Nov 02 03:45:50.226385 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:45:50.226335    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19637a1f_28d0_4615_abec_aa4ee3f20a4b.slice/crio-c49de4294529c19d0959c8a23810010d1dbbcbd9c1915917148d7e430b51a901 WatchSource:0}: Error finding container c49de4294529c19d0959c8a23810010d1dbbcbd9c1915917148d7e430b51a901: Status 404 returned error can't find the container with id c49de4294529c19d0959c8a23810010d1dbbcbd9c1915917148d7e430b51a901
Nov 02 03:45:50.764290 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:50.764253    4496 patch_prober.go:29] interesting pod/e2e-worker-w3-858236-openshift-default-74fcd64bf5-t8dbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" start-of-body=
Nov 02 03:45:50.764452 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:50.764306    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-t8dbv" podUID="4e8b61fe-99dc-4705-ab37-cde3bfad0451" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused"
Nov 02 03:45:50.968255 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:50.968213    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-aad578-6d6755f65f-5ljdx" event={"ID":"19637a1f-28d0-4615-abec-aa4ee3f20a4b","Type":"ContainerStarted","Data":"8cce88161dfcf9cf9bab09a3be36318db8741093357ab0a731e175956f5f39a1"}
Nov 02 03:45:50.968438 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:50.968280    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-aad578-6d6755f65f-5ljdx" event={"ID":"19637a1f-28d0-4615-abec-aa4ee3f20a4b","Type":"ContainerStarted","Data":"c49de4294529c19d0959c8a23810010d1dbbcbd9c1915917148d7e430b51a901"}
Nov 02 03:45:50.968438 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:50.968339    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-aad578-6d6755f65f-5ljdx"
Nov 02 03:45:50.969937 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:50.969907    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-aad578-c96769d99-2zkf4" event={"ID":"54092503-5ffa-4b0c-9e72-520195aa8966","Type":"ContainerStarted","Data":"1554d5bd975260e24cc742381dc525ea338f48388df3dc550110a015c1394ad7"}
Nov 02 03:45:50.970103 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:50.969955    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-aad578-c96769d99-2zkf4" event={"ID":"54092503-5ffa-4b0c-9e72-520195aa8966","Type":"ContainerStarted","Data":"a21b49e9d7019139c4cb31774e5f6fe246ad69e98677ae048e0d1cb0562331b1"}
Nov 02 03:45:50.970180 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:50.970104    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-aad578-c96769d99-2zkf4"
Nov 02 03:45:51.084501 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:51.084388    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-aad578-6d6755f65f-5ljdx" podStartSLOduration=2.08437372 podStartE2EDuration="2.08437372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:45:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:45:51.083457141 +0000 UTC m=+3491.662064520" watchObservedRunningTime="2026-11-02 03:45:51.08437372 +0000 UTC m=+3491.662981101"
Nov 02 03:45:51.225123 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:51.225065    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-aad578-c96769d99-2zkf4" podStartSLOduration=2.225049491 podStartE2EDuration="2.225049491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:45:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:45:51.223970074 +0000 UTC m=+3491.802577454" watchObservedRunningTime="2026-11-02 03:45:51.225049491 +0000 UTC m=+3491.803656870"
Nov 02 03:45:51.763670 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:51.763627    4496 patch_prober.go:29] interesting pod/e2e-worker-w3-858236-openshift-default-74fcd64bf5-t8dbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" start-of-body=
Nov 02 03:45:51.763887 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:51.763693    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-t8dbv" podUID="4e8b61fe-99dc-4705-ab37-cde3bfad0451" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused"
Nov 02 03:45:52.763205 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:52.763166    4496 patch_prober.go:29] interesting pod/e2e-worker-w3-858236-openshift-default-74fcd64bf5-t8dbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" start-of-body=
Nov 02 03:45:52.763703 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:52.763227    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-t8dbv" podUID="4e8b61fe-99dc-4705-ab37-cde3bfad0451" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused"
Nov 02 03:45:53.763585 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:53.763551    4496 patch_prober.go:29] interesting pod/e2e-worker-w3-858236-openshift-default-74fcd64bf5-t8dbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" start-of-body=
Nov 02 03:45:53.764104 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:53.763597    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-t8dbv" podUID="4e8b61fe-99dc-4705-ab37-cde3bfad0451" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused"
Nov 02 03:45:54.763680 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:54.763638    4496 patch_prober.go:29] interesting pod/e2e-worker-w3-858236-openshift-default-74fcd64bf5-t8dbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" start-of-body=
Nov 02 03:45:54.764292 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:54.763698    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-t8dbv" podUID="4e8b61fe-99dc-4705-ab37-cde3bfad0451" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused"
Nov 02 03:45:55.900678 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:55.900642    4496 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-t8dbv"
Nov 02 03:45:55.902136 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:55.902109    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-t8dbv"
Nov 02 03:45:55.902844 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:55.902825    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-t8dbv"
Nov 02 03:45:56.916696 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:56.916667    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7c1bee-694764dc5-zrthf"
Nov 02 03:45:56.917142 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:45:56.916741    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-7c1bee-84b77485bf-9bk69"
Nov 02 03:46:01.980289 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:46:01.980258    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-aad578-c96769d99-2zkf4"
Nov 02 03:46:01.980689 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:46:01.980328    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-aad578-6d6755f65f-5ljdx"
Nov 02 03:46:03.527107 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:46:03.527070    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 03:46:21.692238 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:46:21.692201    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ee0fed-58b9d75758-2smtj"]
Nov 02 03:46:21.696805 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:46:21.696782    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ee0fed-58b9d75758-2smtj"
Nov 02 03:46:21.753377 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:46:21.753346    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ee0fed-58b9d75758-2smtj"]
Nov 02 03:46:21.799465 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:46:21.799434    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-ee0fed-serving-cert\""
Nov 02 03:46:21.812568 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:46:21.812535    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cqmb\" (UniqueName: \"kubernetes.io/projected/9c4dbc8c-dcaf-4864-bcd7-0a8346d04d5b-kube-api-access-5cqmb\") pod \"maas-api-e2e-shared-a-w0-ee0fed-58b9d75758-2smtj\" (UID: \"9c4dbc8c-dcaf-4864-bcd7-0a8346d04d5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ee0fed-58b9d75758-2smtj"
Nov 02 03:46:21.812733 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:46:21.812587    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c4dbc8c-dcaf-4864-bcd7-0a8346d04d5b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-ee0fed-58b9d75758-2smtj\" (UID: \"9c4dbc8c-dcaf-4864-bcd7-0a8346d04d5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ee0fed-58b9d75758-2smtj"
Nov 02 03:46:21.812792 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:46:21.812733    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c4dbc8c-dcaf-4864-bcd7-0a8346d04d5b-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-ee0fed-58b9d75758-2smtj\" (UID: \"9c4dbc8c-dcaf-4864-bcd7-0a8346d04d5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ee0fed-58b9d75758-2smtj"
Nov 02 03:46:21.812852 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:46:21.812806    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c4dbc8c-dcaf-4864-bcd7-0a8346d04d5b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-ee0fed-58b9d75758-2smtj\" (UID: \"9c4dbc8c-dcaf-4864-bcd7-0a8346d04d5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ee0fed-58b9d75758-2smtj"
Nov 02 03:46:21.913720 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:46:21.913684    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c4dbc8c-dcaf-4864-bcd7-0a8346d04d5b-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-ee0fed-58b9d75758-2smtj\" (UID: \"9c4dbc8c-dcaf-4864-bcd7-0a8346d04d5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ee0fed-58b9d75758-2smtj"
Nov 02 03:46:21.913925 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:46:21.913773    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c4dbc8c-dcaf-4864-bcd7-0a8346d04d5b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-ee0fed-58b9d75758-2smtj\" (UID: \"9c4dbc8c-dcaf-4864-bcd7-0a8346d04d5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ee0fed-58b9d75758-2smtj"
Nov 02 03:46:21.913925 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:46:21.913902    4496 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ee0fed-serving-cert: secret "maas-api-e2e-shared-a-w0-ee0fed-serving-cert" not found
Nov 02 03:46:21.914096 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:46:21.913935    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cqmb\" (UniqueName: \"kubernetes.io/projected/9c4dbc8c-dcaf-4864-bcd7-0a8346d04d5b-kube-api-access-5cqmb\") pod \"maas-api-e2e-shared-a-w0-ee0fed-58b9d75758-2smtj\" (UID: \"9c4dbc8c-dcaf-4864-bcd7-0a8346d04d5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ee0fed-58b9d75758-2smtj"
Nov 02 03:46:21.914096 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:46:21.913975    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c4dbc8c-dcaf-4864-bcd7-0a8346d04d5b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-ee0fed-58b9d75758-2smtj\" (UID: \"9c4dbc8c-dcaf-4864-bcd7-0a8346d04d5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ee0fed-58b9d75758-2smtj"
Nov 02 03:46:21.914096 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:46:21.913990    4496 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c4dbc8c-dcaf-4864-bcd7-0a8346d04d5b-maas-api-tls podName:9c4dbc8c-dcaf-4864-bcd7-0a8346d04d5b nodeName:}" failed. No retries permitted until 2026-11-02 03:46:22.413971879 +0000 UTC m=+3522.992579237 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9c4dbc8c-dcaf-4864-bcd7-0a8346d04d5b-maas-api-tls") pod "maas-api-e2e-shared-a-w0-ee0fed-58b9d75758-2smtj" (UID: "9c4dbc8c-dcaf-4864-bcd7-0a8346d04d5b") : secret "maas-api-e2e-shared-a-w0-ee0fed-serving-cert" not found
Nov 02 03:46:21.914645 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:46:21.914622    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c4dbc8c-dcaf-4864-bcd7-0a8346d04d5b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-ee0fed-58b9d75758-2smtj\" (UID: \"9c4dbc8c-dcaf-4864-bcd7-0a8346d04d5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ee0fed-58b9d75758-2smtj"
Nov 02 03:46:21.916410 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:46:21.916387    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c4dbc8c-dcaf-4864-bcd7-0a8346d04d5b-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-ee0fed-58b9d75758-2smtj\" (UID: \"9c4dbc8c-dcaf-4864-bcd7-0a8346d04d5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ee0fed-58b9d75758-2smtj"
Nov 02 03:46:21.971988 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:46:21.971917    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cqmb\" (UniqueName: \"kubernetes.io/projected/9c4dbc8c-dcaf-4864-bcd7-0a8346d04d5b-kube-api-access-5cqmb\") pod \"maas-api-e2e-shared-a-w0-ee0fed-58b9d75758-2smtj\" (UID: \"9c4dbc8c-dcaf-4864-bcd7-0a8346d04d5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ee0fed-58b9d75758-2smtj"
Nov 02 03:46:22.421890 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:46:22.421855    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c4dbc8c-dcaf-4864-bcd7-0a8346d04d5b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-ee0fed-58b9d75758-2smtj\" (UID: \"9c4dbc8c-dcaf-4864-bcd7-0a8346d04d5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ee0fed-58b9d75758-2smtj"
Nov 02 03:46:22.424489 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:46:22.424460    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c4dbc8c-dcaf-4864-bcd7-0a8346d04d5b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-ee0fed-58b9d75758-2smtj\" (UID: \"9c4dbc8c-dcaf-4864-bcd7-0a8346d04d5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ee0fed-58b9d75758-2smtj"
Nov 02 03:46:22.611601 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:46:22.611566    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ee0fed-58b9d75758-2smtj"
Nov 02 03:46:22.992147 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:46:22.992116    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ee0fed-58b9d75758-2smtj"]
Nov 02 03:46:22.992596 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:46:22.992572    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c4dbc8c_dcaf_4864_bcd7_0a8346d04d5b.slice/crio-f5e3f360fa25855ede3c9ed9e03a78df24cacb88a36bf1682d33c5ae1d86730c WatchSource:0}: Error finding container f5e3f360fa25855ede3c9ed9e03a78df24cacb88a36bf1682d33c5ae1d86730c: Status 404 returned error can't find the container with id f5e3f360fa25855ede3c9ed9e03a78df24cacb88a36bf1682d33c5ae1d86730c
Nov 02 03:46:23.212846 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:46:23.212814    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ee0fed-58b9d75758-2smtj" event={"ID":"9c4dbc8c-dcaf-4864-bcd7-0a8346d04d5b","Type":"ContainerStarted","Data":"f5e3f360fa25855ede3c9ed9e03a78df24cacb88a36bf1682d33c5ae1d86730c"}
Nov 02 03:46:24.229372 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:46:24.229336    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ee0fed-58b9d75758-2smtj" event={"ID":"9c4dbc8c-dcaf-4864-bcd7-0a8346d04d5b","Type":"ContainerStarted","Data":"df8ee3a621836d426c45a7bce3b3e70c7b1ea4384a54b72b5dd1084b9a1a66ca"}
Nov 02 03:46:24.229829 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:46:24.229557    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ee0fed-58b9d75758-2smtj"
Nov 02 03:46:24.348477 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:46:24.348421    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ee0fed-58b9d75758-2smtj" podStartSLOduration=2.4653061640000002 podStartE2EDuration="3.34840827s" totalImagesPullingTime="883.102106ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:46:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:46:24.345459715 +0000 UTC m=+3524.924067095" watchObservedRunningTime="2026-11-02 03:46:24.34840827 +0000 UTC m=+3524.927015650"
Nov 02 03:46:27.042718 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:46:27.042679    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a7cf93-696b58646-gp5l7"]
Nov 02 03:46:27.048410 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:46:27.048384    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a7cf93-696b58646-gp5l7"
Nov 02 03:46:27.100927 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:46:27.100893    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a7cf93-696b58646-gp5l7"]
Nov 02 03:46:27.147859 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:46:27.147834    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-a7cf93-serving-cert\""
Nov 02 03:46:27.182468 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:46:27.182432    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44122d78-2a1a-43bb-a123-4494b445d939-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-a7cf93-696b58646-gp5l7\" (UID: \"44122d78-2a1a-43bb-a123-4494b445d939\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a7cf93-696b58646-gp5l7"
Nov 02 03:46:27.182740 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:46:27.182712    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wb5q8\" (UniqueName: \"kubernetes.io/projected/44122d78-2a1a-43bb-a123-4494b445d939-kube-api-access-wb5q8\") pod \"maas-api-e2e-shared-a-w4-a7cf93-696b58646-gp5l7\" (UID: \"44122d78-2a1a-43bb-a123-4494b445d939\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a7cf93-696b58646-gp5l7"
Nov 02 03:46:27.182794 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:46:27.182781    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44122d78-2a1a-43bb-a123-4494b445d939-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-a7cf93-696b58646-gp5l7\" (UID: \"44122d78-2a1a-43bb-a123-4494b445d939\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a7cf93-696b58646-gp5l7"
Nov 02 03:46:27.182831 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:46:27.182810    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44122d78-2a1a-43bb-a123-4494b445d939-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-a7cf93-696b58646-gp5l7\" (UID: \"44122d78-2a1a-43bb-a123-4494b445d939\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a7cf93-696b58646-gp5l7"
Nov 02 03:46:27.283982 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:46:27.283945    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wb5q8\" (UniqueName: \"kubernetes.io/projected/44122d78-2a1a-43bb-a123-4494b445d939-kube-api-access-wb5q8\") pod \"maas-api-e2e-shared-a-w4-a7cf93-696b58646-gp5l7\" (UID: \"44122d78-2a1a-43bb-a123-4494b445d939\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a7cf93-696b58646-gp5l7"
Nov 02 03:46:27.284190 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:46:27.284000    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44122d78-2a1a-43bb-a123-4494b445d939-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-a7cf93-696b58646-gp5l7\" (UID: \"44122d78-2a1a-43bb-a123-4494b445d939\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a7cf93-696b58646-gp5l7"
Nov 02 03:46:27.284190 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:46:27.284120    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44122d78-2a1a-43bb-a123-4494b445d939-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-a7cf93-696b58646-gp5l7\" (UID: \"44122d78-2a1a-43bb-a123-4494b445d939\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a7cf93-696b58646-gp5l7"
Nov 02 03:46:27.284190 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:46:27.284177    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44122d78-2a1a-43bb-a123-4494b445d939-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-a7cf93-696b58646-gp5l7\" (UID: \"44122d78-2a1a-43bb-a123-4494b445d939\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a7cf93-696b58646-gp5l7"
Nov 02 03:46:27.284378 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:46:27.284217    4496 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a7cf93-serving-cert: secret "maas-api-e2e-shared-a-w4-a7cf93-serving-cert" not found
Nov 02 03:46:27.284378 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:46:27.284297    4496 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44122d78-2a1a-43bb-a123-4494b445d939-maas-api-tls podName:44122d78-2a1a-43bb-a123-4494b445d939 nodeName:}" failed. No retries permitted until 2026-11-02 03:46:27.784276898 +0000 UTC m=+3528.362884276 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/44122d78-2a1a-43bb-a123-4494b445d939-maas-api-tls") pod "maas-api-e2e-shared-a-w4-a7cf93-696b58646-gp5l7" (UID: "44122d78-2a1a-43bb-a123-4494b445d939") : secret "maas-api-e2e-shared-a-w4-a7cf93-serving-cert" not found
Nov 02 03:46:27.284855 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:46:27.284807    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44122d78-2a1a-43bb-a123-4494b445d939-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-a7cf93-696b58646-gp5l7\" (UID: \"44122d78-2a1a-43bb-a123-4494b445d939\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a7cf93-696b58646-gp5l7"
Nov 02 03:46:27.286726 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:46:27.286705    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44122d78-2a1a-43bb-a123-4494b445d939-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-a7cf93-696b58646-gp5l7\" (UID: \"44122d78-2a1a-43bb-a123-4494b445d939\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a7cf93-696b58646-gp5l7"
Nov 02 03:46:27.361840 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:46:27.361746    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wb5q8\" (UniqueName: \"kubernetes.io/projected/44122d78-2a1a-43bb-a123-4494b445d939-kube-api-access-wb5q8\") pod \"maas-api-e2e-shared-a-w4-a7cf93-696b58646-gp5l7\" (UID: \"44122d78-2a1a-43bb-a123-4494b445d939\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a7cf93-696b58646-gp5l7"
Nov 02 03:46:27.790857 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:46:27.790825    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44122d78-2a1a-43bb-a123-4494b445d939-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-a7cf93-696b58646-gp5l7\" (UID: \"44122d78-2a1a-43bb-a123-4494b445d939\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a7cf93-696b58646-gp5l7"
Nov 02 03:46:27.793270 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:46:27.793243    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44122d78-2a1a-43bb-a123-4494b445d939-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-a7cf93-696b58646-gp5l7\" (UID: \"44122d78-2a1a-43bb-a123-4494b445d939\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a7cf93-696b58646-gp5l7"
Nov 02 03:46:27.965965 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:46:27.965929    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a7cf93-696b58646-gp5l7"
Nov 02 03:46:28.347198 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:46:28.347164    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a7cf93-696b58646-gp5l7"]
Nov 02 03:46:28.348620 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:46:28.348585    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44122d78_2a1a_43bb_a123_4494b445d939.slice/crio-f96a5c49665aca6fc9655259916f6b67c9212a9b18255f85342f03dec81d2e04 WatchSource:0}: Error finding container f96a5c49665aca6fc9655259916f6b67c9212a9b18255f85342f03dec81d2e04: Status 404 returned error can't find the container with id f96a5c49665aca6fc9655259916f6b67c9212a9b18255f85342f03dec81d2e04
Nov 02 03:46:29.271776 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:46:29.271743    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a7cf93-696b58646-gp5l7" event={"ID":"44122d78-2a1a-43bb-a123-4494b445d939","Type":"ContainerStarted","Data":"85e2a219bd0aa448adc75c57577cc3b0d58403a5accca00b8338a43cc004ae87"}
Nov 02 03:46:29.271964 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:46:29.271834    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a7cf93-696b58646-gp5l7" event={"ID":"44122d78-2a1a-43bb-a123-4494b445d939","Type":"ContainerStarted","Data":"f96a5c49665aca6fc9655259916f6b67c9212a9b18255f85342f03dec81d2e04"}
Nov 02 03:46:29.272009 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:46:29.271970    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a7cf93-696b58646-gp5l7"
Nov 02 03:46:29.425776 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:46:29.425675    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a7cf93-696b58646-gp5l7" podStartSLOduration=2.646777313 podStartE2EDuration="3.425661763s" totalImagesPullingTime="778.88445ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:46:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:46:29.423223657 +0000 UTC m=+3530.001831040" watchObservedRunningTime="2026-11-02 03:46:29.425661763 +0000 UTC m=+3530.004269160"
Nov 02 03:46:30.244572 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:46:30.244544    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ee0fed-58b9d75758-2smtj"
Nov 02 03:46:32.435193 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:46:32.435150    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7c1bee-67bb9f8cc7-lx755"]
Nov 02 03:46:32.439657 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:46:32.439635    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7c1bee-67bb9f8cc7-lx755"
Nov 02 03:46:32.495217 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:46:32.495188    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7c1bee-67bb9f8cc7-lx755"]
Nov 02 03:46:32.541486 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:46:32.541451    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-7c1bee-serving-cert\""
Nov 02 03:46:32.550984 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:46:32.550956    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5649b53a-018e-41be-8592-7d828b6ef19f-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-7c1bee-67bb9f8cc7-lx755\" (UID: \"5649b53a-018e-41be-8592-7d828b6ef19f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7c1bee-67bb9f8cc7-lx755"
Nov 02 03:46:32.551117 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:46:32.550989    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5649b53a-018e-41be-8592-7d828b6ef19f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-7c1bee-67bb9f8cc7-lx755\" (UID: \"5649b53a-018e-41be-8592-7d828b6ef19f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7c1bee-67bb9f8cc7-lx755"
Nov 02 03:46:32.551171 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:46:32.551124    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkbgh\" (UniqueName: \"kubernetes.io/projected/5649b53a-018e-41be-8592-7d828b6ef19f-kube-api-access-kkbgh\") pod \"maas-api-e2e-shared-a-w1-7c1bee-67bb9f8cc7-lx755\" (UID: \"5649b53a-018e-41be-8592-7d828b6ef19f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7c1bee-67bb9f8cc7-lx755"
Nov 02 03:46:32.551280 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:46:32.551260    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5649b53a-018e-41be-8592-7d828b6ef19f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-7c1bee-67bb9f8cc7-lx755\" (UID: \"5649b53a-018e-41be-8592-7d828b6ef19f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7c1bee-67bb9f8cc7-lx755"
Nov 02 03:46:32.652725 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:46:32.652306    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkbgh\" (UniqueName: \"kubernetes.io/projected/5649b53a-018e-41be-8592-7d828b6ef19f-kube-api-access-kkbgh\") pod \"maas-api-e2e-shared-a-w1-7c1bee-67bb9f8cc7-lx755\" (UID: \"5649b53a-018e-41be-8592-7d828b6ef19f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7c1bee-67bb9f8cc7-lx755"
Nov 02 03:46:32.652725 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:46:32.652487    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5649b53a-018e-41be-8592-7d828b6ef19f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-7c1bee-67bb9f8cc7-lx755\" (UID: \"5649b53a-018e-41be-8592-7d828b6ef19f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7c1bee-67bb9f8cc7-lx755"
Nov 02 03:46:32.652725 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:46:32.652622    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5649b53a-018e-41be-8592-7d828b6ef19f-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-7c1bee-67bb9f8cc7-lx755\" (UID: \"5649b53a-018e-41be-8592-7d828b6ef19f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7c1bee-67bb9f8cc7-lx755"
Nov 02 03:46:32.652725 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:46:32.652657    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5649b53a-018e-41be-8592-7d828b6ef19f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-7c1bee-67bb9f8cc7-lx755\" (UID: \"5649b53a-018e-41be-8592-7d828b6ef19f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7c1bee-67bb9f8cc7-lx755"
Nov 02 03:46:32.652725 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:46:32.652684    4496 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7c1bee-serving-cert: secret "maas-api-e2e-shared-a-w1-7c1bee-serving-cert" not found
Nov 02 03:46:32.653166 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:46:32.652774    4496 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5649b53a-018e-41be-8592-7d828b6ef19f-maas-api-tls podName:5649b53a-018e-41be-8592-7d828b6ef19f nodeName:}" failed. No retries permitted until 2026-11-02 03:46:33.152752164 +0000 UTC m=+3533.731359531 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5649b53a-018e-41be-8592-7d828b6ef19f-maas-api-tls") pod "maas-api-e2e-shared-a-w1-7c1bee-67bb9f8cc7-lx755" (UID: "5649b53a-018e-41be-8592-7d828b6ef19f") : secret "maas-api-e2e-shared-a-w1-7c1bee-serving-cert" not found
Nov 02 03:46:32.653913 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:46:32.653877    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5649b53a-018e-41be-8592-7d828b6ef19f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-7c1bee-67bb9f8cc7-lx755\" (UID: \"5649b53a-018e-41be-8592-7d828b6ef19f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7c1bee-67bb9f8cc7-lx755"
Nov 02 03:46:32.655712 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:46:32.655688    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5649b53a-018e-41be-8592-7d828b6ef19f-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-7c1bee-67bb9f8cc7-lx755\" (UID: \"5649b53a-018e-41be-8592-7d828b6ef19f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7c1bee-67bb9f8cc7-lx755"
Nov 02 03:46:32.709857 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:46:32.709794    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkbgh\" (UniqueName: \"kubernetes.io/projected/5649b53a-018e-41be-8592-7d828b6ef19f-kube-api-access-kkbgh\") pod \"maas-api-e2e-shared-a-w1-7c1bee-67bb9f8cc7-lx755\" (UID: \"5649b53a-018e-41be-8592-7d828b6ef19f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7c1bee-67bb9f8cc7-lx755"
Nov 02 03:46:33.159305 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:46:33.159263    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5649b53a-018e-41be-8592-7d828b6ef19f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-7c1bee-67bb9f8cc7-lx755\" (UID: \"5649b53a-018e-41be-8592-7d828b6ef19f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7c1bee-67bb9f8cc7-lx755"
Nov 02 03:46:33.162065 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:46:33.162036    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5649b53a-018e-41be-8592-7d828b6ef19f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-7c1bee-67bb9f8cc7-lx755\" (UID: \"5649b53a-018e-41be-8592-7d828b6ef19f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7c1bee-67bb9f8cc7-lx755"
Nov 02 03:46:33.353830 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:46:33.353796    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7c1bee-67bb9f8cc7-lx755"
Nov 02 03:46:33.528567 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:46:33.528528    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 03:46:33.779558 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:46:33.779528    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7c1bee-67bb9f8cc7-lx755"]
Nov 02 03:46:33.779970 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:46:33.779948    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5649b53a_018e_41be_8592_7d828b6ef19f.slice/crio-5c67376f8abc5814ffe3685d4d0608d15c5fcb1dd5133a57d8a288d00f571c50 WatchSource:0}: Error finding container 5c67376f8abc5814ffe3685d4d0608d15c5fcb1dd5133a57d8a288d00f571c50: Status 404 returned error can't find the container with id 5c67376f8abc5814ffe3685d4d0608d15c5fcb1dd5133a57d8a288d00f571c50
Nov 02 03:46:34.310992 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:46:34.310949    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7c1bee-67bb9f8cc7-lx755" event={"ID":"5649b53a-018e-41be-8592-7d828b6ef19f","Type":"ContainerStarted","Data":"5c67376f8abc5814ffe3685d4d0608d15c5fcb1dd5133a57d8a288d00f571c50"}
Nov 02 03:46:35.284567 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:46:35.284539    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a7cf93-696b58646-gp5l7"
Nov 02 03:46:35.322458 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:46:35.322429    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7c1bee-67bb9f8cc7-lx755" event={"ID":"5649b53a-018e-41be-8592-7d828b6ef19f","Type":"ContainerStarted","Data":"7403777d5d9b7994f9d441505e956f37eecb904494bd1c7a9e996cc1689ccc26"}
Nov 02 03:46:35.322613 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:46:35.322483    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7c1bee-67bb9f8cc7-lx755"
Nov 02 03:46:35.520006 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:46:35.519943    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7c1bee-67bb9f8cc7-lx755" podStartSLOduration=2.657137015 podStartE2EDuration="3.5199229s" totalImagesPullingTime="862.785885ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:46:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:46:35.518446889 +0000 UTC m=+3536.097054285" watchObservedRunningTime="2026-11-02 03:46:35.5199229 +0000 UTC m=+3536.098530280"
Nov 02 03:46:41.335095 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:46:41.335065    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7c1bee-67bb9f8cc7-lx755"
Nov 02 03:46:42.416874 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:46:42.416843    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-aad578-65f6d8585d-k7ds9"]
Nov 02 03:46:42.422811 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:46:42.422790    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-aad578-65f6d8585d-k7ds9"
Nov 02 03:46:42.478139 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:46:42.478106    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-aad578-65f6d8585d-k7ds9"]
Nov 02 03:46:42.523888 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:46:42.523859    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-aad578-serving-cert\""
Nov 02 03:46:42.579347 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:46:42.579318    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4fc73f5-ff8b-4099-a240-7dce74b1fd14-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-aad578-65f6d8585d-k7ds9\" (UID: \"f4fc73f5-ff8b-4099-a240-7dce74b1fd14\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-aad578-65f6d8585d-k7ds9"
Nov 02 03:46:42.579549 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:46:42.579356    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzzcm\" (UniqueName: \"kubernetes.io/projected/f4fc73f5-ff8b-4099-a240-7dce74b1fd14-kube-api-access-lzzcm\") pod \"maas-api-e2e-shared-a-w3-aad578-65f6d8585d-k7ds9\" (UID: \"f4fc73f5-ff8b-4099-a240-7dce74b1fd14\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-aad578-65f6d8585d-k7ds9"
Nov 02 03:46:42.579549 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:46:42.579503    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4fc73f5-ff8b-4099-a240-7dce74b1fd14-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-aad578-65f6d8585d-k7ds9\" (UID: \"f4fc73f5-ff8b-4099-a240-7dce74b1fd14\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-aad578-65f6d8585d-k7ds9"
Nov 02 03:46:42.579680 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:46:42.579558    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4fc73f5-ff8b-4099-a240-7dce74b1fd14-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-aad578-65f6d8585d-k7ds9\" (UID: \"f4fc73f5-ff8b-4099-a240-7dce74b1fd14\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-aad578-65f6d8585d-k7ds9"
Nov 02 03:46:42.681754 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:46:42.681160    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4fc73f5-ff8b-4099-a240-7dce74b1fd14-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-aad578-65f6d8585d-k7ds9\" (UID: \"f4fc73f5-ff8b-4099-a240-7dce74b1fd14\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-aad578-65f6d8585d-k7ds9"
Nov 02 03:46:42.681754 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:46:42.681208    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzzcm\" (UniqueName: \"kubernetes.io/projected/f4fc73f5-ff8b-4099-a240-7dce74b1fd14-kube-api-access-lzzcm\") pod \"maas-api-e2e-shared-a-w3-aad578-65f6d8585d-k7ds9\" (UID: \"f4fc73f5-ff8b-4099-a240-7dce74b1fd14\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-aad578-65f6d8585d-k7ds9"
Nov 02 03:46:42.681754 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:46:42.681289    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4fc73f5-ff8b-4099-a240-7dce74b1fd14-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-aad578-65f6d8585d-k7ds9\" (UID: \"f4fc73f5-ff8b-4099-a240-7dce74b1fd14\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-aad578-65f6d8585d-k7ds9"
Nov 02 03:46:42.681754 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:46:42.681370    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4fc73f5-ff8b-4099-a240-7dce74b1fd14-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-aad578-65f6d8585d-k7ds9\" (UID: \"f4fc73f5-ff8b-4099-a240-7dce74b1fd14\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-aad578-65f6d8585d-k7ds9"
Nov 02 03:46:42.681754 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:46:42.681488    4496 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-aad578-serving-cert: secret "maas-api-e2e-shared-a-w3-aad578-serving-cert" not found
Nov 02 03:46:42.681754 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:46:42.681561    4496 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4fc73f5-ff8b-4099-a240-7dce74b1fd14-maas-api-tls podName:f4fc73f5-ff8b-4099-a240-7dce74b1fd14 nodeName:}" failed. No retries permitted until 2026-11-02 03:46:43.181529995 +0000 UTC m=+3543.760137368 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f4fc73f5-ff8b-4099-a240-7dce74b1fd14-maas-api-tls") pod "maas-api-e2e-shared-a-w3-aad578-65f6d8585d-k7ds9" (UID: "f4fc73f5-ff8b-4099-a240-7dce74b1fd14") : secret "maas-api-e2e-shared-a-w3-aad578-serving-cert" not found
Nov 02 03:46:42.682229 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:46:42.681964    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4fc73f5-ff8b-4099-a240-7dce74b1fd14-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-aad578-65f6d8585d-k7ds9\" (UID: \"f4fc73f5-ff8b-4099-a240-7dce74b1fd14\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-aad578-65f6d8585d-k7ds9"
Nov 02 03:46:42.684345 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:46:42.684318    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4fc73f5-ff8b-4099-a240-7dce74b1fd14-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-aad578-65f6d8585d-k7ds9\" (UID: \"f4fc73f5-ff8b-4099-a240-7dce74b1fd14\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-aad578-65f6d8585d-k7ds9"
Nov 02 03:46:42.741817 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:46:42.741788    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzzcm\" (UniqueName: \"kubernetes.io/projected/f4fc73f5-ff8b-4099-a240-7dce74b1fd14-kube-api-access-lzzcm\") pod \"maas-api-e2e-shared-a-w3-aad578-65f6d8585d-k7ds9\" (UID: \"f4fc73f5-ff8b-4099-a240-7dce74b1fd14\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-aad578-65f6d8585d-k7ds9"
Nov 02 03:46:43.188429 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:46:43.188397    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4fc73f5-ff8b-4099-a240-7dce74b1fd14-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-aad578-65f6d8585d-k7ds9\" (UID: \"f4fc73f5-ff8b-4099-a240-7dce74b1fd14\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-aad578-65f6d8585d-k7ds9"
Nov 02 03:46:43.190869 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:46:43.190848    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4fc73f5-ff8b-4099-a240-7dce74b1fd14-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-aad578-65f6d8585d-k7ds9\" (UID: \"f4fc73f5-ff8b-4099-a240-7dce74b1fd14\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-aad578-65f6d8585d-k7ds9"
Nov 02 03:46:43.336873 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:46:43.336837    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-aad578-65f6d8585d-k7ds9"
Nov 02 03:46:43.722568 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:46:43.722541    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-aad578-65f6d8585d-k7ds9"]
Nov 02 03:46:43.723101 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:46:43.723073    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4fc73f5_ff8b_4099_a240_7dce74b1fd14.slice/crio-247345f42e91ed41de8d0adcbf20dd190586b8fe1311a9ba04444f7b989bc313 WatchSource:0}: Error finding container 247345f42e91ed41de8d0adcbf20dd190586b8fe1311a9ba04444f7b989bc313: Status 404 returned error can't find the container with id 247345f42e91ed41de8d0adcbf20dd190586b8fe1311a9ba04444f7b989bc313
Nov 02 03:46:44.403408 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:46:44.403372    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-aad578-65f6d8585d-k7ds9" event={"ID":"f4fc73f5-ff8b-4099-a240-7dce74b1fd14","Type":"ContainerStarted","Data":"247345f42e91ed41de8d0adcbf20dd190586b8fe1311a9ba04444f7b989bc313"}
Nov 02 03:46:45.422132 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:46:45.419416    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-aad578-65f6d8585d-k7ds9" event={"ID":"f4fc73f5-ff8b-4099-a240-7dce74b1fd14","Type":"ContainerStarted","Data":"efc9fc54d1491b2bcb1b9aa365d3277b6ee9cd150e68c3196fad1558cb90303f"}
Nov 02 03:46:45.422132 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:46:45.420406    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-aad578-65f6d8585d-k7ds9"
Nov 02 03:46:45.539996 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:46:45.539935    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-aad578-65f6d8585d-k7ds9" podStartSLOduration=2.7295813669999998 podStartE2EDuration="3.539915026s" totalImagesPullingTime="810.333659ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:46:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:46:45.53558572 +0000 UTC m=+3546.114193136" watchObservedRunningTime="2026-11-02 03:46:45.539915026 +0000 UTC m=+3546.118522407"
Nov 02 03:46:52.443617 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:46:52.443581    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-aad578-65f6d8585d-k7ds9"
Nov 02 03:47:03.527841 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:47:03.527804    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 03:47:26.729675 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:47:26.729648    4496 generic.go:356] "Generic (PLEG): container finished" podID="1e754d50-d985-4886-9184-102561eb17a8" containerID="2c8eac79e4c5e23da4285d294e4910ece44f2aca0053a9a741c669b4c5e04a6b" exitCode=0
Nov 02 03:47:26.730054 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:47:26.729718    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-aad578-openshift-default-688f6b55cc-qxqmg" event={"ID":"1e754d50-d985-4886-9184-102561eb17a8","Type":"ContainerDied","Data":"2c8eac79e4c5e23da4285d294e4910ece44f2aca0053a9a741c669b4c5e04a6b"}
Nov 02 03:47:26.730179 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:47:26.730159    4496 scope.go:122] "RemoveContainer" containerID="2c8eac79e4c5e23da4285d294e4910ece44f2aca0053a9a741c669b4c5e04a6b"
Nov 02 03:47:26.731699 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:47:26.731668    4496 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 03:47:26.731817 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:47:26.731753    4496 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 03:47:26.731817 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:47:26.731801    4496 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 03:47:27.760068 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:47:27.760005    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-aad578-openshift-default-688f6b55cc-qxqmg" event={"ID":"1e754d50-d985-4886-9184-102561eb17a8","Type":"ContainerStarted","Data":"6d1b07d6fcdd1c10233c620c5745a5c7a4b62b807ed9ffcc48a5c3bad31c2beb"}
Nov 02 03:47:27.764998 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:47:27.764971    4496 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-aad578-openshift-default-688f6b55cc-qxqmg"
Nov 02 03:47:27.765347 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:47:27.765256    4496 patch_prober.go:29] interesting pod/e2e-shared-a-w3-aad578-openshift-default-688f6b55cc-qxqmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused" start-of-body=
Nov 02 03:47:27.765522 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:47:27.765502    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-aad578-openshift-default-688f6b55cc-qxqmg" podUID="1e754d50-d985-4886-9184-102561eb17a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused"
Nov 02 03:47:28.765776 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:47:28.765621    4496 patch_prober.go:29] interesting pod/e2e-shared-a-w3-aad578-openshift-default-688f6b55cc-qxqmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused" start-of-body=
Nov 02 03:47:28.765776 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:47:28.765682    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-aad578-openshift-default-688f6b55cc-qxqmg" podUID="1e754d50-d985-4886-9184-102561eb17a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused"
Nov 02 03:47:29.765605 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:47:29.765565    4496 patch_prober.go:29] interesting pod/e2e-shared-a-w3-aad578-openshift-default-688f6b55cc-qxqmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused" start-of-body=
Nov 02 03:47:29.765940 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:47:29.765914    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-aad578-openshift-default-688f6b55cc-qxqmg" podUID="1e754d50-d985-4886-9184-102561eb17a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused"
Nov 02 03:47:30.765352 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:47:30.765320    4496 patch_prober.go:29] interesting pod/e2e-shared-a-w3-aad578-openshift-default-688f6b55cc-qxqmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused" start-of-body=
Nov 02 03:47:30.765601 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:47:30.765566    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-aad578-openshift-default-688f6b55cc-qxqmg" podUID="1e754d50-d985-4886-9184-102561eb17a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused"
Nov 02 03:47:31.765466 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:47:31.765427    4496 patch_prober.go:29] interesting pod/e2e-shared-a-w3-aad578-openshift-default-688f6b55cc-qxqmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused" start-of-body=
Nov 02 03:47:31.766108 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:47:31.766073    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-aad578-openshift-default-688f6b55cc-qxqmg" podUID="1e754d50-d985-4886-9184-102561eb17a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused"
Nov 02 03:47:32.765221 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:47:32.765146    4496 patch_prober.go:29] interesting pod/e2e-shared-a-w3-aad578-openshift-default-688f6b55cc-qxqmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused" start-of-body=
Nov 02 03:47:32.765221 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:47:32.765213    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-aad578-openshift-default-688f6b55cc-qxqmg" podUID="1e754d50-d985-4886-9184-102561eb17a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused"
Nov 02 03:47:33.529119 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:47:33.529088    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 03:47:33.770526 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:47:33.770494    4496 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-aad578-openshift-default-688f6b55cc-qxqmg"
Nov 02 03:47:33.771043 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:47:33.771024    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-aad578-openshift-default-688f6b55cc-qxqmg"
Nov 02 03:47:33.771694 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:47:33.771674    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-aad578-openshift-default-688f6b55cc-qxqmg"
Nov 02 03:47:40.326920 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:47:40.326893    4496 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hww8q_5c797b08-a314-44ed-8026-8a0ce1f6c36f/ovn-acl-logging/0.log"
Nov 02 03:47:40.335983 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:47:40.335959    4496 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hww8q_5c797b08-a314-44ed-8026-8a0ce1f6c36f/ovn-acl-logging/0.log"
Nov 02 03:48:03.528445 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:48:03.528404    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 03:48:22.684411 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:48:22.684337    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-1b1008-6b77584db9-pdxnh"]
Nov 02 03:48:22.684883 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:48:22.684548    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1b1008-6b77584db9-pdxnh" podUID="57c07a0d-0051-43d1-b102-f6f47f6c9151" containerName="maas-api" containerID="cri-o://81156118dafe93425d8c8967feaad8aa9642cb43085abbb53aa9a2d61e1062ca" gracePeriod=30
Nov 02 03:48:22.971571 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:48:22.971534    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1b1008-6b77584db9-pdxnh"
Nov 02 03:48:23.133121 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:48:23.133087    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57c07a0d-0051-43d1-b102-f6f47f6c9151-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57c07a0d-0051-43d1-b102-f6f47f6c9151-maas-api-tls\") pod \"57c07a0d-0051-43d1-b102-f6f47f6c9151\" (UID: \"57c07a0d-0051-43d1-b102-f6f47f6c9151\") "
Nov 02 03:48:23.133333 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:48:23.133145    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57c07a0d-0051-43d1-b102-f6f47f6c9151-kube-api-access-fshf5\" (UniqueName: \"kubernetes.io/projected/57c07a0d-0051-43d1-b102-f6f47f6c9151-kube-api-access-fshf5\") pod \"57c07a0d-0051-43d1-b102-f6f47f6c9151\" (UID: \"57c07a0d-0051-43d1-b102-f6f47f6c9151\") "
Nov 02 03:48:23.133333 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:48:23.133203    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57c07a0d-0051-43d1-b102-f6f47f6c9151-metrics-certs\" (UniqueName: \"kubernetes.io/secret/57c07a0d-0051-43d1-b102-f6f47f6c9151-metrics-certs\") pod \"57c07a0d-0051-43d1-b102-f6f47f6c9151\" (UID: \"57c07a0d-0051-43d1-b102-f6f47f6c9151\") "
Nov 02 03:48:23.133333 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:48:23.133325    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57c07a0d-0051-43d1-b102-f6f47f6c9151-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57c07a0d-0051-43d1-b102-f6f47f6c9151-openshift-service-ca\") pod \"57c07a0d-0051-43d1-b102-f6f47f6c9151\" (UID: \"57c07a0d-0051-43d1-b102-f6f47f6c9151\") "
Nov 02 03:48:23.134095 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:48:23.133983    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57c07a0d-0051-43d1-b102-f6f47f6c9151-openshift-service-ca" pod "57c07a0d-0051-43d1-b102-f6f47f6c9151" (UID: "57c07a0d-0051-43d1-b102-f6f47f6c9151"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:48:23.136287 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:48:23.136253    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57c07a0d-0051-43d1-b102-f6f47f6c9151-metrics-certs" pod "57c07a0d-0051-43d1-b102-f6f47f6c9151" (UID: "57c07a0d-0051-43d1-b102-f6f47f6c9151"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 03:48:23.136412 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:48:23.136388    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57c07a0d-0051-43d1-b102-f6f47f6c9151-kube-api-access-fshf5" pod "57c07a0d-0051-43d1-b102-f6f47f6c9151" (UID: "57c07a0d-0051-43d1-b102-f6f47f6c9151"). InnerVolumeSpecName "kube-api-access-fshf5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:48:23.136513 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:48:23.136495    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57c07a0d-0051-43d1-b102-f6f47f6c9151-maas-api-tls" pod "57c07a0d-0051-43d1-b102-f6f47f6c9151" (UID: "57c07a0d-0051-43d1-b102-f6f47f6c9151"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 03:48:23.194918 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:48:23.194838    4496 generic.go:356] "Generic (PLEG): container finished" podID="57c07a0d-0051-43d1-b102-f6f47f6c9151" containerID="81156118dafe93425d8c8967feaad8aa9642cb43085abbb53aa9a2d61e1062ca" exitCode=0
Nov 02 03:48:23.194918 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:48:23.194886    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1b1008-6b77584db9-pdxnh"
Nov 02 03:48:23.195151 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:48:23.194920    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1b1008-6b77584db9-pdxnh" event={"ID":"57c07a0d-0051-43d1-b102-f6f47f6c9151","Type":"ContainerDied","Data":"81156118dafe93425d8c8967feaad8aa9642cb43085abbb53aa9a2d61e1062ca"}
Nov 02 03:48:23.195151 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:48:23.195045    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1b1008-6b77584db9-pdxnh" event={"ID":"57c07a0d-0051-43d1-b102-f6f47f6c9151","Type":"ContainerDied","Data":"60ff7c6f72eacb0a8f453da6d2945fd9ddc1ff93371a7e271e07d2298bf1348d"}
Nov 02 03:48:23.195151 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:48:23.195075    4496 scope.go:122] "RemoveContainer" containerID="81156118dafe93425d8c8967feaad8aa9642cb43085abbb53aa9a2d61e1062ca"
Nov 02 03:48:23.207351 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:48:23.207331    4496 scope.go:122] "RemoveContainer" containerID="81156118dafe93425d8c8967feaad8aa9642cb43085abbb53aa9a2d61e1062ca"
Nov 02 03:48:23.207643 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:48:23.207624    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81156118dafe93425d8c8967feaad8aa9642cb43085abbb53aa9a2d61e1062ca\": container with ID starting with 81156118dafe93425d8c8967feaad8aa9642cb43085abbb53aa9a2d61e1062ca not found: ID does not exist" containerID="81156118dafe93425d8c8967feaad8aa9642cb43085abbb53aa9a2d61e1062ca"
Nov 02 03:48:23.207700 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:48:23.207653    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81156118dafe93425d8c8967feaad8aa9642cb43085abbb53aa9a2d61e1062ca"} err="failed to get container status \"81156118dafe93425d8c8967feaad8aa9642cb43085abbb53aa9a2d61e1062ca\": rpc error: code = NotFound desc = could not find container \"81156118dafe93425d8c8967feaad8aa9642cb43085abbb53aa9a2d61e1062ca\": container with ID starting with 81156118dafe93425d8c8967feaad8aa9642cb43085abbb53aa9a2d61e1062ca not found: ID does not exist"
Nov 02 03:48:23.235475 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:48:23.235446    4496 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57c07a0d-0051-43d1-b102-f6f47f6c9151-openshift-service-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:48:23.235475 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:48:23.235475    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57c07a0d-0051-43d1-b102-f6f47f6c9151-maas-api-tls\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:48:23.235657 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:48:23.235487    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fshf5\" (UniqueName: \"kubernetes.io/projected/57c07a0d-0051-43d1-b102-f6f47f6c9151-kube-api-access-fshf5\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:48:23.235657 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:48:23.235496    4496 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57c07a0d-0051-43d1-b102-f6f47f6c9151-metrics-certs\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:48:23.363003 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:48:23.362970    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-1b1008-6b77584db9-pdxnh"]
Nov 02 03:48:23.366361 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:48:23.366337    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-1b1008-6b77584db9-pdxnh"]
Nov 02 03:48:23.575224 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:48:23.575190    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57c07a0d-0051-43d1-b102-f6f47f6c9151" path="/var/lib/kubelet/pods/57c07a0d-0051-43d1-b102-f6f47f6c9151/volumes"
Nov 02 03:48:30.149643 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:48:30.149611    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-1b1008/praxis-route-1b1008-kserve-86f7f4ccbf-w2gnx"]
Nov 02 03:48:33.528478 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:48:33.528439    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 03:48:44.388654 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:48:44.388614    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-147104-658844d6c9-899t6"]
Nov 02 03:48:44.389378 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:48:44.389360    4496 cpu_manager.go:414] "containerMap: removing container" podUID="57c07a0d-0051-43d1-b102-f6f47f6c9151" containerName="maas-api"
Nov 02 03:48:44.389436 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:48:44.389380    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57c07a0d-0051-43d1-b102-f6f47f6c9151" containerName="maas-api"
Nov 02 03:48:44.389540 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:48:44.389529    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="57c07a0d-0051-43d1-b102-f6f47f6c9151" containerName="maas-api"
Nov 02 03:48:44.394688 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:48:44.394669    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-147104-658844d6c9-899t6"
Nov 02 03:48:44.448008 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:48:44.447971    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-147104-658844d6c9-899t6"]
Nov 02 03:48:44.492866 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:48:44.492834    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-147104-serving-cert\""
Nov 02 03:48:44.501625 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:48:44.501594    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrct2\" (UniqueName: \"kubernetes.io/projected/7f80f210-8056-4a78-a27f-5166ab50c9d0-kube-api-access-wrct2\") pod \"maas-api-e2e-shared-b-w0-147104-658844d6c9-899t6\" (UID: \"7f80f210-8056-4a78-a27f-5166ab50c9d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-147104-658844d6c9-899t6"
Nov 02 03:48:44.501708 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:48:44.501641    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f80f210-8056-4a78-a27f-5166ab50c9d0-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-147104-658844d6c9-899t6\" (UID: \"7f80f210-8056-4a78-a27f-5166ab50c9d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-147104-658844d6c9-899t6"
Nov 02 03:48:44.501775 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:48:44.501761    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f80f210-8056-4a78-a27f-5166ab50c9d0-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-147104-658844d6c9-899t6\" (UID: \"7f80f210-8056-4a78-a27f-5166ab50c9d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-147104-658844d6c9-899t6"
Nov 02 03:48:44.501812 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:48:44.501792    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f80f210-8056-4a78-a27f-5166ab50c9d0-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-147104-658844d6c9-899t6\" (UID: \"7f80f210-8056-4a78-a27f-5166ab50c9d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-147104-658844d6c9-899t6"
Nov 02 03:48:44.603008 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:48:44.602975    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrct2\" (UniqueName: \"kubernetes.io/projected/7f80f210-8056-4a78-a27f-5166ab50c9d0-kube-api-access-wrct2\") pod \"maas-api-e2e-shared-b-w0-147104-658844d6c9-899t6\" (UID: \"7f80f210-8056-4a78-a27f-5166ab50c9d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-147104-658844d6c9-899t6"
Nov 02 03:48:44.603210 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:48:44.603064    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f80f210-8056-4a78-a27f-5166ab50c9d0-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-147104-658844d6c9-899t6\" (UID: \"7f80f210-8056-4a78-a27f-5166ab50c9d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-147104-658844d6c9-899t6"
Nov 02 03:48:44.603210 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:48:44.603153    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f80f210-8056-4a78-a27f-5166ab50c9d0-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-147104-658844d6c9-899t6\" (UID: \"7f80f210-8056-4a78-a27f-5166ab50c9d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-147104-658844d6c9-899t6"
Nov 02 03:48:44.603210 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:48:44.603181    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f80f210-8056-4a78-a27f-5166ab50c9d0-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-147104-658844d6c9-899t6\" (UID: \"7f80f210-8056-4a78-a27f-5166ab50c9d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-147104-658844d6c9-899t6"
Nov 02 03:48:44.603401 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:48:44.603301    4496 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-147104-serving-cert: secret "maas-api-e2e-shared-b-w0-147104-serving-cert" not found
Nov 02 03:48:44.603401 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:48:44.603394    4496 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f80f210-8056-4a78-a27f-5166ab50c9d0-maas-api-tls podName:7f80f210-8056-4a78-a27f-5166ab50c9d0 nodeName:}" failed. No retries permitted until 2026-11-02 03:48:45.103371254 +0000 UTC m=+3665.681978614 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7f80f210-8056-4a78-a27f-5166ab50c9d0-maas-api-tls") pod "maas-api-e2e-shared-b-w0-147104-658844d6c9-899t6" (UID: "7f80f210-8056-4a78-a27f-5166ab50c9d0") : secret "maas-api-e2e-shared-b-w0-147104-serving-cert" not found
Nov 02 03:48:44.603813 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:48:44.603791    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f80f210-8056-4a78-a27f-5166ab50c9d0-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-147104-658844d6c9-899t6\" (UID: \"7f80f210-8056-4a78-a27f-5166ab50c9d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-147104-658844d6c9-899t6"
Nov 02 03:48:44.605578 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:48:44.605560    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f80f210-8056-4a78-a27f-5166ab50c9d0-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-147104-658844d6c9-899t6\" (UID: \"7f80f210-8056-4a78-a27f-5166ab50c9d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-147104-658844d6c9-899t6"
Nov 02 03:48:44.661793 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:48:44.661706    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrct2\" (UniqueName: \"kubernetes.io/projected/7f80f210-8056-4a78-a27f-5166ab50c9d0-kube-api-access-wrct2\") pod \"maas-api-e2e-shared-b-w0-147104-658844d6c9-899t6\" (UID: \"7f80f210-8056-4a78-a27f-5166ab50c9d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-147104-658844d6c9-899t6"
Nov 02 03:48:45.109467 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:48:45.109430    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f80f210-8056-4a78-a27f-5166ab50c9d0-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-147104-658844d6c9-899t6\" (UID: \"7f80f210-8056-4a78-a27f-5166ab50c9d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-147104-658844d6c9-899t6"
Nov 02 03:48:45.111781 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:48:45.111752    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f80f210-8056-4a78-a27f-5166ab50c9d0-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-147104-658844d6c9-899t6\" (UID: \"7f80f210-8056-4a78-a27f-5166ab50c9d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-147104-658844d6c9-899t6"
Nov 02 03:48:45.173601 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:48:45.173562    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-praxis-a-1b1008/praxis-route-1b1008-kserve-86f7f4ccbf-w2gnx" podUID="e9c80fc8-7563-4f48-9ba7-0bf52c371d25" containerName="main" containerID="cri-o://83d38862a811fef3724e2b67341c77fc4502c61e1b54efdc4f8026caa922969b" gracePeriod=45
Nov 02 03:48:45.310198 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:48:45.310168    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-147104-658844d6c9-899t6"
Nov 02 03:48:45.371628 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:48:45.371548    4496 generic.go:356] "Generic (PLEG): container finished" podID="e9c80fc8-7563-4f48-9ba7-0bf52c371d25" containerID="83d38862a811fef3724e2b67341c77fc4502c61e1b54efdc4f8026caa922969b" exitCode=0
Nov 02 03:48:45.371796 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:48:45.371660    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-1b1008/praxis-route-1b1008-kserve-86f7f4ccbf-w2gnx" event={"ID":"e9c80fc8-7563-4f48-9ba7-0bf52c371d25","Type":"ContainerDied","Data":"83d38862a811fef3724e2b67341c77fc4502c61e1b54efdc4f8026caa922969b"}
Nov 02 03:48:45.547247 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:48:45.547218    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-1b1008/praxis-route-1b1008-kserve-86f7f4ccbf-w2gnx"
Nov 02 03:48:45.716469 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:48:45.716376    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9c80fc8-7563-4f48-9ba7-0bf52c371d25-kube-api-access-8lzff\" (UniqueName: \"kubernetes.io/projected/e9c80fc8-7563-4f48-9ba7-0bf52c371d25-kube-api-access-8lzff\") pod \"e9c80fc8-7563-4f48-9ba7-0bf52c371d25\" (UID: \"e9c80fc8-7563-4f48-9ba7-0bf52c371d25\") "
Nov 02 03:48:45.716648 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:48:45.716479    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9c80fc8-7563-4f48-9ba7-0bf52c371d25-home\" (UniqueName: \"kubernetes.io/empty-dir/e9c80fc8-7563-4f48-9ba7-0bf52c371d25-home\") pod \"e9c80fc8-7563-4f48-9ba7-0bf52c371d25\" (UID: \"e9c80fc8-7563-4f48-9ba7-0bf52c371d25\") "
Nov 02 03:48:45.716648 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:48:45.716528    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9c80fc8-7563-4f48-9ba7-0bf52c371d25-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9c80fc8-7563-4f48-9ba7-0bf52c371d25-dshm\") pod \"e9c80fc8-7563-4f48-9ba7-0bf52c371d25\" (UID: \"e9c80fc8-7563-4f48-9ba7-0bf52c371d25\") "
Nov 02 03:48:45.716648 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:48:45.716597    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9c80fc8-7563-4f48-9ba7-0bf52c371d25-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9c80fc8-7563-4f48-9ba7-0bf52c371d25-model-cache\") pod \"e9c80fc8-7563-4f48-9ba7-0bf52c371d25\" (UID: \"e9c80fc8-7563-4f48-9ba7-0bf52c371d25\") "
Nov 02 03:48:45.716648 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:48:45.716641    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9c80fc8-7563-4f48-9ba7-0bf52c371d25-tls-certs\" (UniqueName: \"kubernetes.io/secret/e9c80fc8-7563-4f48-9ba7-0bf52c371d25-tls-certs\") pod \"e9c80fc8-7563-4f48-9ba7-0bf52c371d25\" (UID: \"e9c80fc8-7563-4f48-9ba7-0bf52c371d25\") "
Nov 02 03:48:45.716904 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:48:45.716684    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9c80fc8-7563-4f48-9ba7-0bf52c371d25-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9c80fc8-7563-4f48-9ba7-0bf52c371d25-tmp-dir\") pod \"e9c80fc8-7563-4f48-9ba7-0bf52c371d25\" (UID: \"e9c80fc8-7563-4f48-9ba7-0bf52c371d25\") "
Nov 02 03:48:45.716904 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:48:45.716768    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9c80fc8-7563-4f48-9ba7-0bf52c371d25-home" pod "e9c80fc8-7563-4f48-9ba7-0bf52c371d25" (UID: "e9c80fc8-7563-4f48-9ba7-0bf52c371d25"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:48:45.717037 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:48:45.716912    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9c80fc8-7563-4f48-9ba7-0bf52c371d25-model-cache" pod "e9c80fc8-7563-4f48-9ba7-0bf52c371d25" (UID: "e9c80fc8-7563-4f48-9ba7-0bf52c371d25"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:48:45.717429 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:48:45.717405    4496 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9c80fc8-7563-4f48-9ba7-0bf52c371d25-home\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:48:45.717528 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:48:45.717434    4496 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9c80fc8-7563-4f48-9ba7-0bf52c371d25-model-cache\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:48:45.718159 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:48:45.718128    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9c80fc8-7563-4f48-9ba7-0bf52c371d25-tmp-dir" pod "e9c80fc8-7563-4f48-9ba7-0bf52c371d25" (UID: "e9c80fc8-7563-4f48-9ba7-0bf52c371d25"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:48:45.718948 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:48:45.718918    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9c80fc8-7563-4f48-9ba7-0bf52c371d25-dshm" pod "e9c80fc8-7563-4f48-9ba7-0bf52c371d25" (UID: "e9c80fc8-7563-4f48-9ba7-0bf52c371d25"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:48:45.719072 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:48:45.719049    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9c80fc8-7563-4f48-9ba7-0bf52c371d25-kube-api-access-8lzff" pod "e9c80fc8-7563-4f48-9ba7-0bf52c371d25" (UID: "e9c80fc8-7563-4f48-9ba7-0bf52c371d25"). InnerVolumeSpecName "kube-api-access-8lzff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:48:45.719311 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:48:45.719289    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9c80fc8-7563-4f48-9ba7-0bf52c371d25-tls-certs" pod "e9c80fc8-7563-4f48-9ba7-0bf52c371d25" (UID: "e9c80fc8-7563-4f48-9ba7-0bf52c371d25"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 03:48:45.721065 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:48:45.721044    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-147104-658844d6c9-899t6"]
Nov 02 03:48:45.740243 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:48:45.740218    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f80f210_8056_4a78_a27f_5166ab50c9d0.slice/crio-9472cdf207c56d1e878d1b97f837b89bc13850035d42423cbb0b668ed039abbc WatchSource:0}: Error finding container 9472cdf207c56d1e878d1b97f837b89bc13850035d42423cbb0b668ed039abbc: Status 404 returned error can't find the container with id 9472cdf207c56d1e878d1b97f837b89bc13850035d42423cbb0b668ed039abbc
Nov 02 03:48:45.741738 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:48:45.741717    4496 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 03:48:45.819066 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:48:45.819008    4496 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9c80fc8-7563-4f48-9ba7-0bf52c371d25-dshm\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:48:45.819066 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:48:45.819063    4496 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9c80fc8-7563-4f48-9ba7-0bf52c371d25-tls-certs\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:48:45.819257 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:48:45.819075    4496 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9c80fc8-7563-4f48-9ba7-0bf52c371d25-tmp-dir\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:48:45.819257 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:48:45.819086    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lzff\" (UniqueName: \"kubernetes.io/projected/e9c80fc8-7563-4f48-9ba7-0bf52c371d25-kube-api-access-8lzff\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:48:46.380187 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:48:46.380157    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-1b1008/praxis-route-1b1008-kserve-86f7f4ccbf-w2gnx"
Nov 02 03:48:46.380404 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:48:46.380196    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-1b1008/praxis-route-1b1008-kserve-86f7f4ccbf-w2gnx" event={"ID":"e9c80fc8-7563-4f48-9ba7-0bf52c371d25","Type":"ContainerDied","Data":"24e4b2622cda6a8049a0b31158a5df3ce548b00524be9a6f83c750b0b5d23fb6"}
Nov 02 03:48:46.380404 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:48:46.380249    4496 scope.go:122] "RemoveContainer" containerID="83d38862a811fef3724e2b67341c77fc4502c61e1b54efdc4f8026caa922969b"
Nov 02 03:48:46.381764 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:48:46.381730    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-147104-658844d6c9-899t6" event={"ID":"7f80f210-8056-4a78-a27f-5166ab50c9d0","Type":"ContainerStarted","Data":"9472cdf207c56d1e878d1b97f837b89bc13850035d42423cbb0b668ed039abbc"}
Nov 02 03:48:46.547220 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:48:46.547184    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-1b1008/praxis-route-1b1008-kserve-86f7f4ccbf-w2gnx"]
Nov 02 03:48:46.563446 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:48:46.563421    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-praxis-a-1b1008/praxis-route-1b1008-kserve-86f7f4ccbf-w2gnx"]
Nov 02 03:48:47.394674 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:48:47.394637    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-147104-658844d6c9-899t6" event={"ID":"7f80f210-8056-4a78-a27f-5166ab50c9d0","Type":"ContainerStarted","Data":"1d478ed3c1c0cd05b0f9a93864fb833e8e26bb2d1fd8bc3e1672d638b752d65f"}
Nov 02 03:48:47.394876 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:48:47.394703    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-147104-658844d6c9-899t6"
Nov 02 03:48:47.511732 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:48:47.511682    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-147104-658844d6c9-899t6" podStartSLOduration=2.567543748 podStartE2EDuration="3.511667743s" totalImagesPullingTime="944.123995ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:48:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:48:47.509481955 +0000 UTC m=+3668.088089336" watchObservedRunningTime="2026-11-02 03:48:47.511667743 +0000 UTC m=+3668.090275142"
Nov 02 03:48:47.574887 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:48:47.574852    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9c80fc8-7563-4f48-9ba7-0bf52c371d25" path="/var/lib/kubelet/pods/e9c80fc8-7563-4f48-9ba7-0bf52c371d25/volumes"
Nov 02 03:48:53.407829 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:48:53.407800    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-147104-658844d6c9-899t6"
Nov 02 03:48:54.009661 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:48:54.009621    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-ac7ab4-547f75d6d4-dn58x"]
Nov 02 03:48:54.010348 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:48:54.010332    4496 cpu_manager.go:414] "containerMap: removing container" podUID="e9c80fc8-7563-4f48-9ba7-0bf52c371d25" containerName="main"
Nov 02 03:48:54.010348 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:48:54.010349    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9c80fc8-7563-4f48-9ba7-0bf52c371d25" containerName="main"
Nov 02 03:48:54.010470 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:48:54.010462    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9c80fc8-7563-4f48-9ba7-0bf52c371d25" containerName="main"
Nov 02 03:48:54.015487 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:48:54.015469    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-ac7ab4-547f75d6d4-dn58x"
Nov 02 03:48:54.068345 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:48:54.068317    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-ac7ab4-serving-cert\""
Nov 02 03:48:54.070452 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:48:54.070424    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-ac7ab4-547f75d6d4-dn58x"]
Nov 02 03:48:54.121635 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:48:54.121606    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzfsw\" (UniqueName: \"kubernetes.io/projected/4304f6cd-6a4b-40cf-985b-d42cb96d0ac9-kube-api-access-bzfsw\") pod \"maas-api-e2e-shared-b-w4-ac7ab4-547f75d6d4-dn58x\" (UID: \"4304f6cd-6a4b-40cf-985b-d42cb96d0ac9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-ac7ab4-547f75d6d4-dn58x"
Nov 02 03:48:54.121635 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:48:54.121639    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4304f6cd-6a4b-40cf-985b-d42cb96d0ac9-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-ac7ab4-547f75d6d4-dn58x\" (UID: \"4304f6cd-6a4b-40cf-985b-d42cb96d0ac9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-ac7ab4-547f75d6d4-dn58x"
Nov 02 03:48:54.121839 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:48:54.121673    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4304f6cd-6a4b-40cf-985b-d42cb96d0ac9-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-ac7ab4-547f75d6d4-dn58x\" (UID: \"4304f6cd-6a4b-40cf-985b-d42cb96d0ac9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-ac7ab4-547f75d6d4-dn58x"
Nov 02 03:48:54.121878 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:48:54.121850    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4304f6cd-6a4b-40cf-985b-d42cb96d0ac9-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-ac7ab4-547f75d6d4-dn58x\" (UID: \"4304f6cd-6a4b-40cf-985b-d42cb96d0ac9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-ac7ab4-547f75d6d4-dn58x"
Nov 02 03:48:54.223206 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:48:54.223173    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4304f6cd-6a4b-40cf-985b-d42cb96d0ac9-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-ac7ab4-547f75d6d4-dn58x\" (UID: \"4304f6cd-6a4b-40cf-985b-d42cb96d0ac9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-ac7ab4-547f75d6d4-dn58x"
Nov 02 03:48:54.223392 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:48:54.223284    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzfsw\" (UniqueName: \"kubernetes.io/projected/4304f6cd-6a4b-40cf-985b-d42cb96d0ac9-kube-api-access-bzfsw\") pod \"maas-api-e2e-shared-b-w4-ac7ab4-547f75d6d4-dn58x\" (UID: \"4304f6cd-6a4b-40cf-985b-d42cb96d0ac9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-ac7ab4-547f75d6d4-dn58x"
Nov 02 03:48:54.223392 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:48:54.223308    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4304f6cd-6a4b-40cf-985b-d42cb96d0ac9-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-ac7ab4-547f75d6d4-dn58x\" (UID: \"4304f6cd-6a4b-40cf-985b-d42cb96d0ac9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-ac7ab4-547f75d6d4-dn58x"
Nov 02 03:48:54.223392 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:48:54.223339    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4304f6cd-6a4b-40cf-985b-d42cb96d0ac9-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-ac7ab4-547f75d6d4-dn58x\" (UID: \"4304f6cd-6a4b-40cf-985b-d42cb96d0ac9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-ac7ab4-547f75d6d4-dn58x"
Nov 02 03:48:54.223569 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:48:54.223459    4496 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-ac7ab4-serving-cert: secret "maas-api-e2e-shared-b-w4-ac7ab4-serving-cert" not found
Nov 02 03:48:54.223569 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:48:54.223538    4496 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4304f6cd-6a4b-40cf-985b-d42cb96d0ac9-maas-api-tls podName:4304f6cd-6a4b-40cf-985b-d42cb96d0ac9 nodeName:}" failed. No retries permitted until 2026-11-02 03:48:54.72351669 +0000 UTC m=+3675.302124063 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4304f6cd-6a4b-40cf-985b-d42cb96d0ac9-maas-api-tls") pod "maas-api-e2e-shared-b-w4-ac7ab4-547f75d6d4-dn58x" (UID: "4304f6cd-6a4b-40cf-985b-d42cb96d0ac9") : secret "maas-api-e2e-shared-b-w4-ac7ab4-serving-cert" not found
Nov 02 03:48:54.224131 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:48:54.224109    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4304f6cd-6a4b-40cf-985b-d42cb96d0ac9-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-ac7ab4-547f75d6d4-dn58x\" (UID: \"4304f6cd-6a4b-40cf-985b-d42cb96d0ac9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-ac7ab4-547f75d6d4-dn58x"
Nov 02 03:48:54.225605 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:48:54.225582    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4304f6cd-6a4b-40cf-985b-d42cb96d0ac9-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-ac7ab4-547f75d6d4-dn58x\" (UID: \"4304f6cd-6a4b-40cf-985b-d42cb96d0ac9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-ac7ab4-547f75d6d4-dn58x"
Nov 02 03:48:54.282448 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:48:54.282358    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzfsw\" (UniqueName: \"kubernetes.io/projected/4304f6cd-6a4b-40cf-985b-d42cb96d0ac9-kube-api-access-bzfsw\") pod \"maas-api-e2e-shared-b-w4-ac7ab4-547f75d6d4-dn58x\" (UID: \"4304f6cd-6a4b-40cf-985b-d42cb96d0ac9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-ac7ab4-547f75d6d4-dn58x"
Nov 02 03:48:54.728561 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:48:54.728521    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4304f6cd-6a4b-40cf-985b-d42cb96d0ac9-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-ac7ab4-547f75d6d4-dn58x\" (UID: \"4304f6cd-6a4b-40cf-985b-d42cb96d0ac9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-ac7ab4-547f75d6d4-dn58x"
Nov 02 03:48:54.731217 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:48:54.731184    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4304f6cd-6a4b-40cf-985b-d42cb96d0ac9-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-ac7ab4-547f75d6d4-dn58x\" (UID: \"4304f6cd-6a4b-40cf-985b-d42cb96d0ac9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-ac7ab4-547f75d6d4-dn58x"
Nov 02 03:48:54.929913 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:48:54.929874    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-ac7ab4-547f75d6d4-dn58x"
Nov 02 03:48:55.317041 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:48:55.313891    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4304f6cd_6a4b_40cf_985b_d42cb96d0ac9.slice/crio-d11fa82fb3055f3e78ae2c43ce3f6b39f5aaa5268fedf761e6e96ff7552981b1 WatchSource:0}: Error finding container d11fa82fb3055f3e78ae2c43ce3f6b39f5aaa5268fedf761e6e96ff7552981b1: Status 404 returned error can't find the container with id d11fa82fb3055f3e78ae2c43ce3f6b39f5aaa5268fedf761e6e96ff7552981b1
Nov 02 03:48:55.317041 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:48:55.313928    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-ac7ab4-547f75d6d4-dn58x"]
Nov 02 03:48:55.468399 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:48:55.468363    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-ac7ab4-547f75d6d4-dn58x" event={"ID":"4304f6cd-6a4b-40cf-985b-d42cb96d0ac9","Type":"ContainerStarted","Data":"d11fa82fb3055f3e78ae2c43ce3f6b39f5aaa5268fedf761e6e96ff7552981b1"}
Nov 02 03:48:56.479565 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:48:56.479517    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-ac7ab4-547f75d6d4-dn58x" event={"ID":"4304f6cd-6a4b-40cf-985b-d42cb96d0ac9","Type":"ContainerStarted","Data":"e6501811c5d8fe951fe82478192c592c3eecd26c9b2fb941ec3f12b34a3a9b98"}
Nov 02 03:48:56.479997 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:48:56.479596    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-ac7ab4-547f75d6d4-dn58x"
Nov 02 03:48:56.597527 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:48:56.597470    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-ac7ab4-547f75d6d4-dn58x" podStartSLOduration=2.799695771 podStartE2EDuration="3.597455639s" totalImagesPullingTime="797.759868ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:48:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:48:56.596896108 +0000 UTC m=+3677.175503499" watchObservedRunningTime="2026-11-02 03:48:56.597455639 +0000 UTC m=+3677.176063019"
Nov 02 03:49:02.497597 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:49:02.497562    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-ac7ab4-547f75d6d4-dn58x"
Nov 02 03:49:03.529928 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:49:03.529887    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 03:49:08.386992 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:49:08.386954    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3b27a5-7d4d48687f-hq6zz"]
Nov 02 03:49:08.391566 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:49:08.391550    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3b27a5-7d4d48687f-hq6zz"
Nov 02 03:49:08.449115 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:49:08.449083    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3b27a5-7d4d48687f-hq6zz"]
Nov 02 03:49:08.493803 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:49:08.493772    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-3b27a5-serving-cert\""
Nov 02 03:49:08.510662 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:49:08.510633    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51249edc-f28f-4f57-94cd-eca441b6c5c6-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-3b27a5-7d4d48687f-hq6zz\" (UID: \"51249edc-f28f-4f57-94cd-eca441b6c5c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3b27a5-7d4d48687f-hq6zz"
Nov 02 03:49:08.510769 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:49:08.510668    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51249edc-f28f-4f57-94cd-eca441b6c5c6-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-3b27a5-7d4d48687f-hq6zz\" (UID: \"51249edc-f28f-4f57-94cd-eca441b6c5c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3b27a5-7d4d48687f-hq6zz"
Nov 02 03:49:08.510769 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:49:08.510692    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vx794\" (UniqueName: \"kubernetes.io/projected/51249edc-f28f-4f57-94cd-eca441b6c5c6-kube-api-access-vx794\") pod \"maas-api-e2e-shared-b-w1-3b27a5-7d4d48687f-hq6zz\" (UID: \"51249edc-f28f-4f57-94cd-eca441b6c5c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3b27a5-7d4d48687f-hq6zz"
Nov 02 03:49:08.510769 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:49:08.510718    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51249edc-f28f-4f57-94cd-eca441b6c5c6-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-3b27a5-7d4d48687f-hq6zz\" (UID: \"51249edc-f28f-4f57-94cd-eca441b6c5c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3b27a5-7d4d48687f-hq6zz"
Nov 02 03:49:08.612258 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:49:08.612218    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51249edc-f28f-4f57-94cd-eca441b6c5c6-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-3b27a5-7d4d48687f-hq6zz\" (UID: \"51249edc-f28f-4f57-94cd-eca441b6c5c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3b27a5-7d4d48687f-hq6zz"
Nov 02 03:49:08.612770 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:49:08.612739    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51249edc-f28f-4f57-94cd-eca441b6c5c6-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-3b27a5-7d4d48687f-hq6zz\" (UID: \"51249edc-f28f-4f57-94cd-eca441b6c5c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3b27a5-7d4d48687f-hq6zz"
Nov 02 03:49:08.612892 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:49:08.612781    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51249edc-f28f-4f57-94cd-eca441b6c5c6-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-3b27a5-7d4d48687f-hq6zz\" (UID: \"51249edc-f28f-4f57-94cd-eca441b6c5c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3b27a5-7d4d48687f-hq6zz"
Nov 02 03:49:08.612892 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:49:08.612822    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vx794\" (UniqueName: \"kubernetes.io/projected/51249edc-f28f-4f57-94cd-eca441b6c5c6-kube-api-access-vx794\") pod \"maas-api-e2e-shared-b-w1-3b27a5-7d4d48687f-hq6zz\" (UID: \"51249edc-f28f-4f57-94cd-eca441b6c5c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3b27a5-7d4d48687f-hq6zz"
Nov 02 03:49:08.613034 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:49:08.612930    4496 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3b27a5-serving-cert: secret "maas-api-e2e-shared-b-w1-3b27a5-serving-cert" not found
Nov 02 03:49:08.613103 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:49:08.613076    4496 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51249edc-f28f-4f57-94cd-eca441b6c5c6-maas-api-tls podName:51249edc-f28f-4f57-94cd-eca441b6c5c6 nodeName:}" failed. No retries permitted until 2026-11-02 03:49:09.113006991 +0000 UTC m=+3689.691614348 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/51249edc-f28f-4f57-94cd-eca441b6c5c6-maas-api-tls") pod "maas-api-e2e-shared-b-w1-3b27a5-7d4d48687f-hq6zz" (UID: "51249edc-f28f-4f57-94cd-eca441b6c5c6") : secret "maas-api-e2e-shared-b-w1-3b27a5-serving-cert" not found
Nov 02 03:49:08.613564 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:49:08.613531    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51249edc-f28f-4f57-94cd-eca441b6c5c6-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-3b27a5-7d4d48687f-hq6zz\" (UID: \"51249edc-f28f-4f57-94cd-eca441b6c5c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3b27a5-7d4d48687f-hq6zz"
Nov 02 03:49:08.614914 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:49:08.614890    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51249edc-f28f-4f57-94cd-eca441b6c5c6-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-3b27a5-7d4d48687f-hq6zz\" (UID: \"51249edc-f28f-4f57-94cd-eca441b6c5c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3b27a5-7d4d48687f-hq6zz"
Nov 02 03:49:08.671678 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:49:08.671602    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vx794\" (UniqueName: \"kubernetes.io/projected/51249edc-f28f-4f57-94cd-eca441b6c5c6-kube-api-access-vx794\") pod \"maas-api-e2e-shared-b-w1-3b27a5-7d4d48687f-hq6zz\" (UID: \"51249edc-f28f-4f57-94cd-eca441b6c5c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3b27a5-7d4d48687f-hq6zz"
Nov 02 03:49:09.120216 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:49:09.120179    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51249edc-f28f-4f57-94cd-eca441b6c5c6-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-3b27a5-7d4d48687f-hq6zz\" (UID: \"51249edc-f28f-4f57-94cd-eca441b6c5c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3b27a5-7d4d48687f-hq6zz"
Nov 02 03:49:09.122685 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:49:09.122653    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51249edc-f28f-4f57-94cd-eca441b6c5c6-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-3b27a5-7d4d48687f-hq6zz\" (UID: \"51249edc-f28f-4f57-94cd-eca441b6c5c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3b27a5-7d4d48687f-hq6zz"
Nov 02 03:49:09.305950 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:49:09.305918    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3b27a5-7d4d48687f-hq6zz"
Nov 02 03:49:09.692992 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:49:09.692959    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3b27a5-7d4d48687f-hq6zz"]
Nov 02 03:49:09.693834 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:49:09.693806    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51249edc_f28f_4f57_94cd_eca441b6c5c6.slice/crio-349eda8df30c5496d9f5cf96f58813e09f606016f70e9c3f5b45c23fdf4b0689 WatchSource:0}: Error finding container 349eda8df30c5496d9f5cf96f58813e09f606016f70e9c3f5b45c23fdf4b0689: Status 404 returned error can't find the container with id 349eda8df30c5496d9f5cf96f58813e09f606016f70e9c3f5b45c23fdf4b0689
Nov 02 03:49:10.585593 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:49:10.585553    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3b27a5-7d4d48687f-hq6zz" event={"ID":"51249edc-f28f-4f57-94cd-eca441b6c5c6","Type":"ContainerStarted","Data":"349eda8df30c5496d9f5cf96f58813e09f606016f70e9c3f5b45c23fdf4b0689"}
Nov 02 03:49:11.594107 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:49:11.594066    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3b27a5-7d4d48687f-hq6zz" event={"ID":"51249edc-f28f-4f57-94cd-eca441b6c5c6","Type":"ContainerStarted","Data":"fb7547cd4d3400c22dc944ea4d18afb65d09d8dfce34e3fc4e43b13716192d92"}
Nov 02 03:49:11.594502 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:49:11.594134    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3b27a5-7d4d48687f-hq6zz"
Nov 02 03:49:11.712360 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:49:11.712305    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3b27a5-7d4d48687f-hq6zz" podStartSLOduration=2.888599999 podStartE2EDuration="3.712290677s" totalImagesPullingTime="823.690678ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:49:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:49:11.709855475 +0000 UTC m=+3692.288462856" watchObservedRunningTime="2026-11-02 03:49:11.712290677 +0000 UTC m=+3692.290898105"
Nov 02 03:49:17.607489 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:49:17.607459    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3b27a5-7d4d48687f-hq6zz"
Nov 02 03:49:23.396879 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:49:23.396844    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3d14ee-6b76657b9-rs68l"]
Nov 02 03:49:23.401430 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:49:23.401400    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3d14ee-6b76657b9-rs68l"
Nov 02 03:49:23.460843 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:49:23.460792    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3d14ee-6b76657b9-rs68l"]
Nov 02 03:49:23.478100 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:49:23.478068    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-3d14ee-serving-cert\""
Nov 02 03:49:23.503039 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:49:23.499402    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0591e186-a26e-46a9-b1a6-482335af9889-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-3d14ee-6b76657b9-rs68l\" (UID: \"0591e186-a26e-46a9-b1a6-482335af9889\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3d14ee-6b76657b9-rs68l"
Nov 02 03:49:23.503039 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:49:23.499611    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0591e186-a26e-46a9-b1a6-482335af9889-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-3d14ee-6b76657b9-rs68l\" (UID: \"0591e186-a26e-46a9-b1a6-482335af9889\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3d14ee-6b76657b9-rs68l"
Nov 02 03:49:23.503039 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:49:23.499662    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zrb8\" (UniqueName: \"kubernetes.io/projected/0591e186-a26e-46a9-b1a6-482335af9889-kube-api-access-4zrb8\") pod \"maas-api-e2e-shared-b-w3-3d14ee-6b76657b9-rs68l\" (UID: \"0591e186-a26e-46a9-b1a6-482335af9889\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3d14ee-6b76657b9-rs68l"
Nov 02 03:49:23.503039 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:49:23.499694    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0591e186-a26e-46a9-b1a6-482335af9889-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-3d14ee-6b76657b9-rs68l\" (UID: \"0591e186-a26e-46a9-b1a6-482335af9889\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3d14ee-6b76657b9-rs68l"
Nov 02 03:49:23.603040 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:49:23.601229    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zrb8\" (UniqueName: \"kubernetes.io/projected/0591e186-a26e-46a9-b1a6-482335af9889-kube-api-access-4zrb8\") pod \"maas-api-e2e-shared-b-w3-3d14ee-6b76657b9-rs68l\" (UID: \"0591e186-a26e-46a9-b1a6-482335af9889\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3d14ee-6b76657b9-rs68l"
Nov 02 03:49:23.603040 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:49:23.601289    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0591e186-a26e-46a9-b1a6-482335af9889-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-3d14ee-6b76657b9-rs68l\" (UID: \"0591e186-a26e-46a9-b1a6-482335af9889\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3d14ee-6b76657b9-rs68l"
Nov 02 03:49:23.603040 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:49:23.601473    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0591e186-a26e-46a9-b1a6-482335af9889-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-3d14ee-6b76657b9-rs68l\" (UID: \"0591e186-a26e-46a9-b1a6-482335af9889\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3d14ee-6b76657b9-rs68l"
Nov 02 03:49:23.603040 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:49:23.601837    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0591e186-a26e-46a9-b1a6-482335af9889-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-3d14ee-6b76657b9-rs68l\" (UID: \"0591e186-a26e-46a9-b1a6-482335af9889\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3d14ee-6b76657b9-rs68l"
Nov 02 03:49:23.603040 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:49:23.602049    4496 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3d14ee-serving-cert: secret "maas-api-e2e-shared-b-w3-3d14ee-serving-cert" not found
Nov 02 03:49:23.603040 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:49:23.602124    4496 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0591e186-a26e-46a9-b1a6-482335af9889-maas-api-tls podName:0591e186-a26e-46a9-b1a6-482335af9889 nodeName:}" failed. No retries permitted until 2026-11-02 03:49:24.102103572 +0000 UTC m=+3704.680710948 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0591e186-a26e-46a9-b1a6-482335af9889-maas-api-tls") pod "maas-api-e2e-shared-b-w3-3d14ee-6b76657b9-rs68l" (UID: "0591e186-a26e-46a9-b1a6-482335af9889") : secret "maas-api-e2e-shared-b-w3-3d14ee-serving-cert" not found
Nov 02 03:49:23.607131 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:49:23.607093    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0591e186-a26e-46a9-b1a6-482335af9889-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-3d14ee-6b76657b9-rs68l\" (UID: \"0591e186-a26e-46a9-b1a6-482335af9889\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3d14ee-6b76657b9-rs68l"
Nov 02 03:49:23.609657 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:49:23.609634    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0591e186-a26e-46a9-b1a6-482335af9889-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-3d14ee-6b76657b9-rs68l\" (UID: \"0591e186-a26e-46a9-b1a6-482335af9889\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3d14ee-6b76657b9-rs68l"
Nov 02 03:49:23.663312 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:49:23.663236    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zrb8\" (UniqueName: \"kubernetes.io/projected/0591e186-a26e-46a9-b1a6-482335af9889-kube-api-access-4zrb8\") pod \"maas-api-e2e-shared-b-w3-3d14ee-6b76657b9-rs68l\" (UID: \"0591e186-a26e-46a9-b1a6-482335af9889\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3d14ee-6b76657b9-rs68l"
Nov 02 03:49:24.108217 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:49:24.108178    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0591e186-a26e-46a9-b1a6-482335af9889-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-3d14ee-6b76657b9-rs68l\" (UID: \"0591e186-a26e-46a9-b1a6-482335af9889\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3d14ee-6b76657b9-rs68l"
Nov 02 03:49:24.110827 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:49:24.110789    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0591e186-a26e-46a9-b1a6-482335af9889-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-3d14ee-6b76657b9-rs68l\" (UID: \"0591e186-a26e-46a9-b1a6-482335af9889\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3d14ee-6b76657b9-rs68l"
Nov 02 03:49:24.319290 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:49:24.319256    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3d14ee-6b76657b9-rs68l"
Nov 02 03:49:24.715435 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:49:24.715403    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0591e186_a26e_46a9_b1a6_482335af9889.slice/crio-3993f8f5f241a9e48a1d11c2b13856c0b2251af2e89f6bacfe916fc2ede43e81 WatchSource:0}: Error finding container 3993f8f5f241a9e48a1d11c2b13856c0b2251af2e89f6bacfe916fc2ede43e81: Status 404 returned error can't find the container with id 3993f8f5f241a9e48a1d11c2b13856c0b2251af2e89f6bacfe916fc2ede43e81
Nov 02 03:49:24.716098 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:49:24.716071    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3d14ee-6b76657b9-rs68l"]
Nov 02 03:49:25.704647 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:49:25.704605    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3d14ee-6b76657b9-rs68l" event={"ID":"0591e186-a26e-46a9-b1a6-482335af9889","Type":"ContainerStarted","Data":"eed8eb0799e9200e460377b31b8de68ff1854cc5effd9adaec99197e99043473"}
Nov 02 03:49:25.704834 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:49:25.704664    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3d14ee-6b76657b9-rs68l" event={"ID":"0591e186-a26e-46a9-b1a6-482335af9889","Type":"ContainerStarted","Data":"3993f8f5f241a9e48a1d11c2b13856c0b2251af2e89f6bacfe916fc2ede43e81"}
Nov 02 03:49:25.704834 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:49:25.704705    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3d14ee-6b76657b9-rs68l"
Nov 02 03:49:25.822945 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:49:25.822843    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3d14ee-6b76657b9-rs68l" podStartSLOduration=2.043624295 podStartE2EDuration="2.822830231s" totalImagesPullingTime="779.205936ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:49:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:49:25.821754263 +0000 UTC m=+3706.400361644" watchObservedRunningTime="2026-11-02 03:49:25.822830231 +0000 UTC m=+3706.401437610"
Nov 02 03:49:31.718295 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:49:31.718267    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3d14ee-6b76657b9-rs68l"
Nov 02 03:49:33.528028 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:49:33.527983    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 03:49:47.140163 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:49:47.140124    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-f68c7e-5c59f9756f-mqvxw"]
Nov 02 03:49:47.140688 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:49:47.140349    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f68c7e-5c59f9756f-mqvxw" podUID="0c9230a0-ebae-42a0-b100-5c48210e8cd3" containerName="maas-api" containerID="cri-o://5d82d880615764d6700babd69cc368557cef6915a6f13450abf9dabd9577485a" gracePeriod=30
Nov 02 03:49:47.440002 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:49:47.439971    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f68c7e-5c59f9756f-mqvxw"
Nov 02 03:49:47.509269 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:49:47.509233    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c9230a0-ebae-42a0-b100-5c48210e8cd3-kube-api-access-5v6qx\" (UniqueName: \"kubernetes.io/projected/0c9230a0-ebae-42a0-b100-5c48210e8cd3-kube-api-access-5v6qx\") pod \"0c9230a0-ebae-42a0-b100-5c48210e8cd3\" (UID: \"0c9230a0-ebae-42a0-b100-5c48210e8cd3\") "
Nov 02 03:49:47.509467 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:49:47.509323    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c9230a0-ebae-42a0-b100-5c48210e8cd3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c9230a0-ebae-42a0-b100-5c48210e8cd3-openshift-service-ca\") pod \"0c9230a0-ebae-42a0-b100-5c48210e8cd3\" (UID: \"0c9230a0-ebae-42a0-b100-5c48210e8cd3\") "
Nov 02 03:49:47.509467 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:49:47.509451    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c9230a0-ebae-42a0-b100-5c48210e8cd3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c9230a0-ebae-42a0-b100-5c48210e8cd3-metrics-certs\") pod \"0c9230a0-ebae-42a0-b100-5c48210e8cd3\" (UID: \"0c9230a0-ebae-42a0-b100-5c48210e8cd3\") "
Nov 02 03:49:47.509618 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:49:47.509494    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c9230a0-ebae-42a0-b100-5c48210e8cd3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c9230a0-ebae-42a0-b100-5c48210e8cd3-maas-api-tls\") pod \"0c9230a0-ebae-42a0-b100-5c48210e8cd3\" (UID: \"0c9230a0-ebae-42a0-b100-5c48210e8cd3\") "
Nov 02 03:49:47.509782 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:49:47.509756    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c9230a0-ebae-42a0-b100-5c48210e8cd3-openshift-service-ca" pod "0c9230a0-ebae-42a0-b100-5c48210e8cd3" (UID: "0c9230a0-ebae-42a0-b100-5c48210e8cd3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:49:47.510082 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:49:47.510061    4496 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c9230a0-ebae-42a0-b100-5c48210e8cd3-openshift-service-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:49:47.511546 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:49:47.511515    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c9230a0-ebae-42a0-b100-5c48210e8cd3-kube-api-access-5v6qx" pod "0c9230a0-ebae-42a0-b100-5c48210e8cd3" (UID: "0c9230a0-ebae-42a0-b100-5c48210e8cd3"). InnerVolumeSpecName "kube-api-access-5v6qx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:49:47.511666 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:49:47.511554    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c9230a0-ebae-42a0-b100-5c48210e8cd3-metrics-certs" pod "0c9230a0-ebae-42a0-b100-5c48210e8cd3" (UID: "0c9230a0-ebae-42a0-b100-5c48210e8cd3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 03:49:47.511666 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:49:47.511617    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c9230a0-ebae-42a0-b100-5c48210e8cd3-maas-api-tls" pod "0c9230a0-ebae-42a0-b100-5c48210e8cd3" (UID: "0c9230a0-ebae-42a0-b100-5c48210e8cd3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 03:49:47.611728 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:49:47.611700    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5v6qx\" (UniqueName: \"kubernetes.io/projected/0c9230a0-ebae-42a0-b100-5c48210e8cd3-kube-api-access-5v6qx\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:49:47.611728 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:49:47.611726    4496 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c9230a0-ebae-42a0-b100-5c48210e8cd3-metrics-certs\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:49:47.611728 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:49:47.611736    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c9230a0-ebae-42a0-b100-5c48210e8cd3-maas-api-tls\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:49:47.880790 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:49:47.880749    4496 generic.go:356] "Generic (PLEG): container finished" podID="0c9230a0-ebae-42a0-b100-5c48210e8cd3" containerID="5d82d880615764d6700babd69cc368557cef6915a6f13450abf9dabd9577485a" exitCode=0
Nov 02 03:49:47.881002 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:49:47.880799    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f68c7e-5c59f9756f-mqvxw"
Nov 02 03:49:47.881002 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:49:47.880832    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f68c7e-5c59f9756f-mqvxw" event={"ID":"0c9230a0-ebae-42a0-b100-5c48210e8cd3","Type":"ContainerDied","Data":"5d82d880615764d6700babd69cc368557cef6915a6f13450abf9dabd9577485a"}
Nov 02 03:49:47.881002 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:49:47.880903    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f68c7e-5c59f9756f-mqvxw" event={"ID":"0c9230a0-ebae-42a0-b100-5c48210e8cd3","Type":"ContainerDied","Data":"587f36934ec481a486468ea131d5c9dcf9bba6c05310d22a0ec67c6c588b13c6"}
Nov 02 03:49:47.881002 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:49:47.880933    4496 scope.go:122] "RemoveContainer" containerID="5d82d880615764d6700babd69cc368557cef6915a6f13450abf9dabd9577485a"
Nov 02 03:49:47.894626 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:49:47.894605    4496 scope.go:122] "RemoveContainer" containerID="5d82d880615764d6700babd69cc368557cef6915a6f13450abf9dabd9577485a"
Nov 02 03:49:47.894920 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:49:47.894897    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d82d880615764d6700babd69cc368557cef6915a6f13450abf9dabd9577485a\": container with ID starting with 5d82d880615764d6700babd69cc368557cef6915a6f13450abf9dabd9577485a not found: ID does not exist" containerID="5d82d880615764d6700babd69cc368557cef6915a6f13450abf9dabd9577485a"
Nov 02 03:49:47.895004 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:49:47.894931    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d82d880615764d6700babd69cc368557cef6915a6f13450abf9dabd9577485a"} err="failed to get container status \"5d82d880615764d6700babd69cc368557cef6915a6f13450abf9dabd9577485a\": rpc error: code = NotFound desc = could not find container \"5d82d880615764d6700babd69cc368557cef6915a6f13450abf9dabd9577485a\": container with ID starting with 5d82d880615764d6700babd69cc368557cef6915a6f13450abf9dabd9577485a not found: ID does not exist"
Nov 02 03:49:48.051892 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:49:48.051860    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-f68c7e-5c59f9756f-mqvxw"]
Nov 02 03:49:48.058920 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:49:48.058891    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-f68c7e-5c59f9756f-mqvxw"]
Nov 02 03:49:49.576447 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:49:49.576410    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c9230a0-ebae-42a0-b100-5c48210e8cd3" path="/var/lib/kubelet/pods/0c9230a0-ebae-42a0-b100-5c48210e8cd3/volumes"
Nov 02 03:49:55.843606 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:49:55.843528    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-f68c7e-openshift-default-db7d68949-c7q6l"]
Nov 02 03:49:55.844103 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:49:55.843747    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-f68c7e-openshift-default-db7d68949-c7q6l" podUID="568d206f-e9d9-4f0a-b641-d7b2d90c8ee7" containerName="istio-proxy" containerID="cri-o://265541d06fa08c6ef738cae800ba6dff8c6d791c70bca23166a4ce599b068b92" gracePeriod=30
Nov 02 03:50:00.708543 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:50:00.708501    4496 patch_prober.go:29] interesting pod/e2e-praxis-b-f68c7e-openshift-default-db7d68949-c7q6l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 03:50:00.708929 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:50:00.708577    4496 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-f68c7e-openshift-default-db7d68949-c7q6l" podUID="568d206f-e9d9-4f0a-b641-d7b2d90c8ee7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 03:50:00.999725 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:50:00.999692    4496 generic.go:356] "Generic (PLEG): container finished" podID="568d206f-e9d9-4f0a-b641-d7b2d90c8ee7" containerID="265541d06fa08c6ef738cae800ba6dff8c6d791c70bca23166a4ce599b068b92" exitCode=0
Nov 02 03:50:00.999901 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:50:00.999773    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-f68c7e-openshift-default-db7d68949-c7q6l" event={"ID":"568d206f-e9d9-4f0a-b641-d7b2d90c8ee7","Type":"ContainerDied","Data":"265541d06fa08c6ef738cae800ba6dff8c6d791c70bca23166a4ce599b068b92"}
Nov 02 03:50:01.115374 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:50:01.115349    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-f68c7e-openshift-default-db7d68949-c7q6l"
Nov 02 03:50:01.279708 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:50:01.279674    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/568d206f-e9d9-4f0a-b641-d7b2d90c8ee7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/568d206f-e9d9-4f0a-b641-d7b2d90c8ee7-istio-data\") pod \"568d206f-e9d9-4f0a-b641-d7b2d90c8ee7\" (UID: \"568d206f-e9d9-4f0a-b641-d7b2d90c8ee7\") "
Nov 02 03:50:01.279919 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:50:01.279743    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/568d206f-e9d9-4f0a-b641-d7b2d90c8ee7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/568d206f-e9d9-4f0a-b641-d7b2d90c8ee7-workload-socket\") pod \"568d206f-e9d9-4f0a-b641-d7b2d90c8ee7\" (UID: \"568d206f-e9d9-4f0a-b641-d7b2d90c8ee7\") "
Nov 02 03:50:01.279919 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:50:01.279786    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/568d206f-e9d9-4f0a-b641-d7b2d90c8ee7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/568d206f-e9d9-4f0a-b641-d7b2d90c8ee7-istio-podinfo\") pod \"568d206f-e9d9-4f0a-b641-d7b2d90c8ee7\" (UID: \"568d206f-e9d9-4f0a-b641-d7b2d90c8ee7\") "
Nov 02 03:50:01.279919 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:50:01.279858    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/568d206f-e9d9-4f0a-b641-d7b2d90c8ee7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/568d206f-e9d9-4f0a-b641-d7b2d90c8ee7-istiod-ca-cert\") pod \"568d206f-e9d9-4f0a-b641-d7b2d90c8ee7\" (UID: \"568d206f-e9d9-4f0a-b641-d7b2d90c8ee7\") "
Nov 02 03:50:01.279919 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:50:01.279919    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/568d206f-e9d9-4f0a-b641-d7b2d90c8ee7-kube-api-access-nrgq8\" (UniqueName: \"kubernetes.io/projected/568d206f-e9d9-4f0a-b641-d7b2d90c8ee7-kube-api-access-nrgq8\") pod \"568d206f-e9d9-4f0a-b641-d7b2d90c8ee7\" (UID: \"568d206f-e9d9-4f0a-b641-d7b2d90c8ee7\") "
Nov 02 03:50:01.280216 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:50:01.279981    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/568d206f-e9d9-4f0a-b641-d7b2d90c8ee7-istio-token\" (UniqueName: \"kubernetes.io/projected/568d206f-e9d9-4f0a-b641-d7b2d90c8ee7-istio-token\") pod \"568d206f-e9d9-4f0a-b641-d7b2d90c8ee7\" (UID: \"568d206f-e9d9-4f0a-b641-d7b2d90c8ee7\") "
Nov 02 03:50:01.280216 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:50:01.280026    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/568d206f-e9d9-4f0a-b641-d7b2d90c8ee7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/568d206f-e9d9-4f0a-b641-d7b2d90c8ee7-workload-certs\") pod \"568d206f-e9d9-4f0a-b641-d7b2d90c8ee7\" (UID: \"568d206f-e9d9-4f0a-b641-d7b2d90c8ee7\") "
Nov 02 03:50:01.280216 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:50:01.280091    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/568d206f-e9d9-4f0a-b641-d7b2d90c8ee7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/568d206f-e9d9-4f0a-b641-d7b2d90c8ee7-istio-envoy\") pod \"568d206f-e9d9-4f0a-b641-d7b2d90c8ee7\" (UID: \"568d206f-e9d9-4f0a-b641-d7b2d90c8ee7\") "
Nov 02 03:50:01.280216 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:50:01.280123    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/568d206f-e9d9-4f0a-b641-d7b2d90c8ee7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/568d206f-e9d9-4f0a-b641-d7b2d90c8ee7-credential-socket\") pod \"568d206f-e9d9-4f0a-b641-d7b2d90c8ee7\" (UID: \"568d206f-e9d9-4f0a-b641-d7b2d90c8ee7\") "
Nov 02 03:50:01.280440 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:50:01.280295    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/568d206f-e9d9-4f0a-b641-d7b2d90c8ee7-istio-data" pod "568d206f-e9d9-4f0a-b641-d7b2d90c8ee7" (UID: "568d206f-e9d9-4f0a-b641-d7b2d90c8ee7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:50:01.280642 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:50:01.280591    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/568d206f-e9d9-4f0a-b641-d7b2d90c8ee7-workload-certs" pod "568d206f-e9d9-4f0a-b641-d7b2d90c8ee7" (UID: "568d206f-e9d9-4f0a-b641-d7b2d90c8ee7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:50:01.280642 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:50:01.280600    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/568d206f-e9d9-4f0a-b641-d7b2d90c8ee7-istiod-ca-cert" pod "568d206f-e9d9-4f0a-b641-d7b2d90c8ee7" (UID: "568d206f-e9d9-4f0a-b641-d7b2d90c8ee7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:50:01.280946 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:50:01.280921    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/568d206f-e9d9-4f0a-b641-d7b2d90c8ee7-credential-socket" pod "568d206f-e9d9-4f0a-b641-d7b2d90c8ee7" (UID: "568d206f-e9d9-4f0a-b641-d7b2d90c8ee7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:50:01.281775 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:50:01.281091    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/568d206f-e9d9-4f0a-b641-d7b2d90c8ee7-workload-socket" pod "568d206f-e9d9-4f0a-b641-d7b2d90c8ee7" (UID: "568d206f-e9d9-4f0a-b641-d7b2d90c8ee7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:50:01.281895 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:50:01.281760    4496 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/568d206f-e9d9-4f0a-b641-d7b2d90c8ee7-workload-certs\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:50:01.282000 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:50:01.281983    4496 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/568d206f-e9d9-4f0a-b641-d7b2d90c8ee7-credential-socket\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:50:01.282157 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:50:01.282127    4496 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/568d206f-e9d9-4f0a-b641-d7b2d90c8ee7-istio-data\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:50:01.282337 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:50:01.282317    4496 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/568d206f-e9d9-4f0a-b641-d7b2d90c8ee7-istiod-ca-cert\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:50:01.283489 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:50:01.283459    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/568d206f-e9d9-4f0a-b641-d7b2d90c8ee7-istio-token" pod "568d206f-e9d9-4f0a-b641-d7b2d90c8ee7" (UID: "568d206f-e9d9-4f0a-b641-d7b2d90c8ee7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:50:01.283700 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:50:01.283675    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/568d206f-e9d9-4f0a-b641-d7b2d90c8ee7-istio-podinfo" pod "568d206f-e9d9-4f0a-b641-d7b2d90c8ee7" (UID: "568d206f-e9d9-4f0a-b641-d7b2d90c8ee7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 03:50:01.283973 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:50:01.283948    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/568d206f-e9d9-4f0a-b641-d7b2d90c8ee7-kube-api-access-nrgq8" pod "568d206f-e9d9-4f0a-b641-d7b2d90c8ee7" (UID: "568d206f-e9d9-4f0a-b641-d7b2d90c8ee7"). InnerVolumeSpecName "kube-api-access-nrgq8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:50:01.284610 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:50:01.284590    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/568d206f-e9d9-4f0a-b641-d7b2d90c8ee7-istio-envoy" pod "568d206f-e9d9-4f0a-b641-d7b2d90c8ee7" (UID: "568d206f-e9d9-4f0a-b641-d7b2d90c8ee7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:50:01.383044 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:50:01.382998    4496 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/568d206f-e9d9-4f0a-b641-d7b2d90c8ee7-istio-envoy\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:50:01.383044 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:50:01.383037    4496 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/568d206f-e9d9-4f0a-b641-d7b2d90c8ee7-workload-socket\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:50:01.383044 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:50:01.383049    4496 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/568d206f-e9d9-4f0a-b641-d7b2d90c8ee7-istio-podinfo\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:50:01.383310 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:50:01.383059    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrgq8\" (UniqueName: \"kubernetes.io/projected/568d206f-e9d9-4f0a-b641-d7b2d90c8ee7-kube-api-access-nrgq8\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:50:01.383310 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:50:01.383069    4496 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/568d206f-e9d9-4f0a-b641-d7b2d90c8ee7-istio-token\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:50:02.009369 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:50:02.009337    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-f68c7e-openshift-default-db7d68949-c7q6l"
Nov 02 03:50:02.009828 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:50:02.009364    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-f68c7e-openshift-default-db7d68949-c7q6l" event={"ID":"568d206f-e9d9-4f0a-b641-d7b2d90c8ee7","Type":"ContainerDied","Data":"f97c73ff9a8dc0c9f28a842279d7831d101bb9ea9b44f91649321b5c0e391f62"}
Nov 02 03:50:02.009828 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:50:02.009430    4496 scope.go:122] "RemoveContainer" containerID="265541d06fa08c6ef738cae800ba6dff8c6d791c70bca23166a4ce599b068b92"
Nov 02 03:50:02.178446 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:50:02.178412    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-f68c7e-openshift-default-db7d68949-c7q6l"]
Nov 02 03:50:02.228933 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:50:02.228897    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-f68c7e-openshift-default-db7d68949-c7q6l"]
Nov 02 03:50:03.526831 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:50:03.526798    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 03:50:03.574550 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:50:03.574519    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="568d206f-e9d9-4f0a-b641-d7b2d90c8ee7" path="/var/lib/kubelet/pods/568d206f-e9d9-4f0a-b641-d7b2d90c8ee7/volumes"
Nov 02 03:50:19.529079 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:50:19.529041    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-ee0fed/auth-test-model-ee0fed-kserve-94748b6fc-4shsp"]
Nov 02 03:50:19.529773 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:50:19.529757    4496 cpu_manager.go:414] "containerMap: removing container" podUID="0c9230a0-ebae-42a0-b100-5c48210e8cd3" containerName="maas-api"
Nov 02 03:50:19.529773 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:50:19.529773    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c9230a0-ebae-42a0-b100-5c48210e8cd3" containerName="maas-api"
Nov 02 03:50:19.529854 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:50:19.529786    4496 cpu_manager.go:414] "containerMap: removing container" podUID="568d206f-e9d9-4f0a-b641-d7b2d90c8ee7" containerName="istio-proxy"
Nov 02 03:50:19.529854 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:50:19.529797    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="568d206f-e9d9-4f0a-b641-d7b2d90c8ee7" containerName="istio-proxy"
Nov 02 03:50:19.529939 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:50:19.529928    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c9230a0-ebae-42a0-b100-5c48210e8cd3" containerName="maas-api"
Nov 02 03:50:19.529976 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:50:19.529941    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="568d206f-e9d9-4f0a-b641-d7b2d90c8ee7" containerName="istio-proxy"
Nov 02 03:50:19.533544 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:50:19.533524    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-ee0fed/auth-test-model-ee0fed-kserve-94748b6fc-4shsp"
Nov 02 03:50:19.589168 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:50:19.589130    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-ee0fed/auth-test-model-ee0fed-kserve-94748b6fc-4shsp"]
Nov 02 03:50:19.605144 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:50:19.605115    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46ffad11-c873-4046-ad9a-cc120fb70101-tmp-dir\") pod \"auth-test-model-ee0fed-kserve-94748b6fc-4shsp\" (UID: \"46ffad11-c873-4046-ad9a-cc120fb70101\") " pod="ai-tenant-e2e-shared-a-w0-ee0fed/auth-test-model-ee0fed-kserve-94748b6fc-4shsp"
Nov 02 03:50:19.605322 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:50:19.605162    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46ffad11-c873-4046-ad9a-cc120fb70101-tls-certs\") pod \"auth-test-model-ee0fed-kserve-94748b6fc-4shsp\" (UID: \"46ffad11-c873-4046-ad9a-cc120fb70101\") " pod="ai-tenant-e2e-shared-a-w0-ee0fed/auth-test-model-ee0fed-kserve-94748b6fc-4shsp"
Nov 02 03:50:19.605322 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:50:19.605183    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46ffad11-c873-4046-ad9a-cc120fb70101-home\") pod \"auth-test-model-ee0fed-kserve-94748b6fc-4shsp\" (UID: \"46ffad11-c873-4046-ad9a-cc120fb70101\") " pod="ai-tenant-e2e-shared-a-w0-ee0fed/auth-test-model-ee0fed-kserve-94748b6fc-4shsp"
Nov 02 03:50:19.605435 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:50:19.605410    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46ffad11-c873-4046-ad9a-cc120fb70101-model-cache\") pod \"auth-test-model-ee0fed-kserve-94748b6fc-4shsp\" (UID: \"46ffad11-c873-4046-ad9a-cc120fb70101\") " pod="ai-tenant-e2e-shared-a-w0-ee0fed/auth-test-model-ee0fed-kserve-94748b6fc-4shsp"
Nov 02 03:50:19.605572 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:50:19.605544    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46ffad11-c873-4046-ad9a-cc120fb70101-dshm\") pod \"auth-test-model-ee0fed-kserve-94748b6fc-4shsp\" (UID: \"46ffad11-c873-4046-ad9a-cc120fb70101\") " pod="ai-tenant-e2e-shared-a-w0-ee0fed/auth-test-model-ee0fed-kserve-94748b6fc-4shsp"
Nov 02 03:50:19.605654 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:50:19.605628    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nrqx\" (UniqueName: \"kubernetes.io/projected/46ffad11-c873-4046-ad9a-cc120fb70101-kube-api-access-5nrqx\") pod \"auth-test-model-ee0fed-kserve-94748b6fc-4shsp\" (UID: \"46ffad11-c873-4046-ad9a-cc120fb70101\") " pod="ai-tenant-e2e-shared-a-w0-ee0fed/auth-test-model-ee0fed-kserve-94748b6fc-4shsp"
Nov 02 03:50:19.635412 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:50:19.635386    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-ee0fed\"/\"kube-root-ca.crt\""
Nov 02 03:50:19.635412 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:50:19.635398    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-ee0fed\"/\"auth-test-model-ee0fed-kserve-self-signed-certs\""
Nov 02 03:50:19.635596 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:50:19.635388    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-ee0fed\"/\"openshift-service-ca.crt\""
Nov 02 03:50:19.637424 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:50:19.637409    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-ee0fed\"/\"default-dockercfg-8jrb8\""
Nov 02 03:50:19.706805 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:50:19.706769    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46ffad11-c873-4046-ad9a-cc120fb70101-tmp-dir\") pod \"auth-test-model-ee0fed-kserve-94748b6fc-4shsp\" (UID: \"46ffad11-c873-4046-ad9a-cc120fb70101\") " pod="ai-tenant-e2e-shared-a-w0-ee0fed/auth-test-model-ee0fed-kserve-94748b6fc-4shsp"
Nov 02 03:50:19.707053 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:50:19.706822    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46ffad11-c873-4046-ad9a-cc120fb70101-tls-certs\") pod \"auth-test-model-ee0fed-kserve-94748b6fc-4shsp\" (UID: \"46ffad11-c873-4046-ad9a-cc120fb70101\") " pod="ai-tenant-e2e-shared-a-w0-ee0fed/auth-test-model-ee0fed-kserve-94748b6fc-4shsp"
Nov 02 03:50:19.707053 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:50:19.706851    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46ffad11-c873-4046-ad9a-cc120fb70101-home\") pod \"auth-test-model-ee0fed-kserve-94748b6fc-4shsp\" (UID: \"46ffad11-c873-4046-ad9a-cc120fb70101\") " pod="ai-tenant-e2e-shared-a-w0-ee0fed/auth-test-model-ee0fed-kserve-94748b6fc-4shsp"
Nov 02 03:50:19.707053 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:50:19.706952    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46ffad11-c873-4046-ad9a-cc120fb70101-model-cache\") pod \"auth-test-model-ee0fed-kserve-94748b6fc-4shsp\" (UID: \"46ffad11-c873-4046-ad9a-cc120fb70101\") " pod="ai-tenant-e2e-shared-a-w0-ee0fed/auth-test-model-ee0fed-kserve-94748b6fc-4shsp"
Nov 02 03:50:19.707231 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:50:19.707153    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46ffad11-c873-4046-ad9a-cc120fb70101-dshm\") pod \"auth-test-model-ee0fed-kserve-94748b6fc-4shsp\" (UID: \"46ffad11-c873-4046-ad9a-cc120fb70101\") " pod="ai-tenant-e2e-shared-a-w0-ee0fed/auth-test-model-ee0fed-kserve-94748b6fc-4shsp"
Nov 02 03:50:19.707231 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:50:19.707201    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nrqx\" (UniqueName: \"kubernetes.io/projected/46ffad11-c873-4046-ad9a-cc120fb70101-kube-api-access-5nrqx\") pod \"auth-test-model-ee0fed-kserve-94748b6fc-4shsp\" (UID: \"46ffad11-c873-4046-ad9a-cc120fb70101\") " pod="ai-tenant-e2e-shared-a-w0-ee0fed/auth-test-model-ee0fed-kserve-94748b6fc-4shsp"
Nov 02 03:50:19.707351 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:50:19.707281    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46ffad11-c873-4046-ad9a-cc120fb70101-home\") pod \"auth-test-model-ee0fed-kserve-94748b6fc-4shsp\" (UID: \"46ffad11-c873-4046-ad9a-cc120fb70101\") " pod="ai-tenant-e2e-shared-a-w0-ee0fed/auth-test-model-ee0fed-kserve-94748b6fc-4shsp"
Nov 02 03:50:19.707548 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:50:19.707526    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46ffad11-c873-4046-ad9a-cc120fb70101-model-cache\") pod \"auth-test-model-ee0fed-kserve-94748b6fc-4shsp\" (UID: \"46ffad11-c873-4046-ad9a-cc120fb70101\") " pod="ai-tenant-e2e-shared-a-w0-ee0fed/auth-test-model-ee0fed-kserve-94748b6fc-4shsp"
Nov 02 03:50:19.707627 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:50:19.707566    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46ffad11-c873-4046-ad9a-cc120fb70101-tmp-dir\") pod \"auth-test-model-ee0fed-kserve-94748b6fc-4shsp\" (UID: \"46ffad11-c873-4046-ad9a-cc120fb70101\") " pod="ai-tenant-e2e-shared-a-w0-ee0fed/auth-test-model-ee0fed-kserve-94748b6fc-4shsp"
Nov 02 03:50:19.709450 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:50:19.709428    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46ffad11-c873-4046-ad9a-cc120fb70101-dshm\") pod \"auth-test-model-ee0fed-kserve-94748b6fc-4shsp\" (UID: \"46ffad11-c873-4046-ad9a-cc120fb70101\") " pod="ai-tenant-e2e-shared-a-w0-ee0fed/auth-test-model-ee0fed-kserve-94748b6fc-4shsp"
Nov 02 03:50:19.709517 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:50:19.709466    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46ffad11-c873-4046-ad9a-cc120fb70101-tls-certs\") pod \"auth-test-model-ee0fed-kserve-94748b6fc-4shsp\" (UID: \"46ffad11-c873-4046-ad9a-cc120fb70101\") " pod="ai-tenant-e2e-shared-a-w0-ee0fed/auth-test-model-ee0fed-kserve-94748b6fc-4shsp"
Nov 02 03:50:19.764803 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:50:19.764762    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nrqx\" (UniqueName: \"kubernetes.io/projected/46ffad11-c873-4046-ad9a-cc120fb70101-kube-api-access-5nrqx\") pod \"auth-test-model-ee0fed-kserve-94748b6fc-4shsp\" (UID: \"46ffad11-c873-4046-ad9a-cc120fb70101\") " pod="ai-tenant-e2e-shared-a-w0-ee0fed/auth-test-model-ee0fed-kserve-94748b6fc-4shsp"
Nov 02 03:50:19.848044 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:50:19.847940    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-ee0fed/auth-test-model-ee0fed-kserve-94748b6fc-4shsp"
Nov 02 03:50:20.236827 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:50:20.236795    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-ee0fed/auth-test-model-ee0fed-kserve-94748b6fc-4shsp"]
Nov 02 03:50:20.237421 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:50:20.237392    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod46ffad11_c873_4046_ad9a_cc120fb70101.slice/crio-f74440819013842e170fefa5159b6d8779d3ca5290821481585ba44735a38234 WatchSource:0}: Error finding container f74440819013842e170fefa5159b6d8779d3ca5290821481585ba44735a38234: Status 404 returned error can't find the container with id f74440819013842e170fefa5159b6d8779d3ca5290821481585ba44735a38234
Nov 02 03:50:21.161416 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:50:21.161375    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-ee0fed/auth-test-model-ee0fed-kserve-94748b6fc-4shsp" event={"ID":"46ffad11-c873-4046-ad9a-cc120fb70101","Type":"ContainerStarted","Data":"2b65a2c3e112f9ca83470c1ff93c7ba12ba2a3ec18cb5d03a40268a5a8e34ca4"}
Nov 02 03:50:21.161924 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:50:21.161429    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-ee0fed/auth-test-model-ee0fed-kserve-94748b6fc-4shsp" event={"ID":"46ffad11-c873-4046-ad9a-cc120fb70101","Type":"ContainerStarted","Data":"f74440819013842e170fefa5159b6d8779d3ca5290821481585ba44735a38234"}
Nov 02 03:50:21.278220 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:50:21.278170    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-ee0fed/auth-test-model-ee0fed-kserve-94748b6fc-4shsp" podStartSLOduration=2.27815504 podStartE2EDuration="2.27815504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:50:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:50:21.277010708 +0000 UTC m=+3761.855618089" watchObservedRunningTime="2026-11-02 03:50:21.27815504 +0000 UTC m=+3761.856762422"
Nov 02 03:50:21.849071 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:50:21.849008    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-ee0fed/auth-test-model-ee0fed-kserve-94748b6fc-4shsp"
Nov 02 03:50:29.848431 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:50:29.848394    4496 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-ee0fed/auth-test-model-ee0fed-kserve-94748b6fc-4shsp"
Nov 02 03:50:29.861539 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:50:29.861509    4496 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-ee0fed/auth-test-model-ee0fed-kserve-94748b6fc-4shsp"
Nov 02 03:50:30.236820 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:50:30.236788    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-ee0fed/auth-test-model-ee0fed-kserve-94748b6fc-4shsp"
Nov 02 03:50:33.528419 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:50:33.528380    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 03:51:03.528561 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:51:03.528516    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 03:51:23.628963 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:51:23.628886    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-86f4c4-654dfc988c-7cgf9"]
Nov 02 03:51:23.633157 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:51:23.633138    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-86f4c4-654dfc988c-7cgf9"
Nov 02 03:51:23.737806 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:51:23.737771    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-86f4c4-serving-cert\""
Nov 02 03:51:23.741668 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:51:23.741638    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-86f4c4-654dfc988c-7cgf9"]
Nov 02 03:51:23.781863 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:51:23.781823    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca4d18b6-ca1b-472d-a6ff-bfdca90eea1e-openshift-service-ca\") pod \"maas-api-e2e-explicit-86f4c4-654dfc988c-7cgf9\" (UID: \"ca4d18b6-ca1b-472d-a6ff-bfdca90eea1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-86f4c4-654dfc988c-7cgf9"
Nov 02 03:51:23.782084 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:51:23.782059    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5g6x\" (UniqueName: \"kubernetes.io/projected/ca4d18b6-ca1b-472d-a6ff-bfdca90eea1e-kube-api-access-t5g6x\") pod \"maas-api-e2e-explicit-86f4c4-654dfc988c-7cgf9\" (UID: \"ca4d18b6-ca1b-472d-a6ff-bfdca90eea1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-86f4c4-654dfc988c-7cgf9"
Nov 02 03:51:23.782184 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:51:23.782125    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca4d18b6-ca1b-472d-a6ff-bfdca90eea1e-metrics-certs\") pod \"maas-api-e2e-explicit-86f4c4-654dfc988c-7cgf9\" (UID: \"ca4d18b6-ca1b-472d-a6ff-bfdca90eea1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-86f4c4-654dfc988c-7cgf9"
Nov 02 03:51:23.782321 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:51:23.782267    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca4d18b6-ca1b-472d-a6ff-bfdca90eea1e-maas-api-tls\") pod \"maas-api-e2e-explicit-86f4c4-654dfc988c-7cgf9\" (UID: \"ca4d18b6-ca1b-472d-a6ff-bfdca90eea1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-86f4c4-654dfc988c-7cgf9"
Nov 02 03:51:23.884140 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:51:23.884042    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca4d18b6-ca1b-472d-a6ff-bfdca90eea1e-openshift-service-ca\") pod \"maas-api-e2e-explicit-86f4c4-654dfc988c-7cgf9\" (UID: \"ca4d18b6-ca1b-472d-a6ff-bfdca90eea1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-86f4c4-654dfc988c-7cgf9"
Nov 02 03:51:23.884323 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:51:23.884183    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5g6x\" (UniqueName: \"kubernetes.io/projected/ca4d18b6-ca1b-472d-a6ff-bfdca90eea1e-kube-api-access-t5g6x\") pod \"maas-api-e2e-explicit-86f4c4-654dfc988c-7cgf9\" (UID: \"ca4d18b6-ca1b-472d-a6ff-bfdca90eea1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-86f4c4-654dfc988c-7cgf9"
Nov 02 03:51:23.884323 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:51:23.884230    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca4d18b6-ca1b-472d-a6ff-bfdca90eea1e-metrics-certs\") pod \"maas-api-e2e-explicit-86f4c4-654dfc988c-7cgf9\" (UID: \"ca4d18b6-ca1b-472d-a6ff-bfdca90eea1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-86f4c4-654dfc988c-7cgf9"
Nov 02 03:51:23.884450 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:51:23.884334    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca4d18b6-ca1b-472d-a6ff-bfdca90eea1e-maas-api-tls\") pod \"maas-api-e2e-explicit-86f4c4-654dfc988c-7cgf9\" (UID: \"ca4d18b6-ca1b-472d-a6ff-bfdca90eea1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-86f4c4-654dfc988c-7cgf9"
Nov 02 03:51:23.884922 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:51:23.884894    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca4d18b6-ca1b-472d-a6ff-bfdca90eea1e-openshift-service-ca\") pod \"maas-api-e2e-explicit-86f4c4-654dfc988c-7cgf9\" (UID: \"ca4d18b6-ca1b-472d-a6ff-bfdca90eea1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-86f4c4-654dfc988c-7cgf9"
Nov 02 03:51:23.887162 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:51:23.887136    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca4d18b6-ca1b-472d-a6ff-bfdca90eea1e-metrics-certs\") pod \"maas-api-e2e-explicit-86f4c4-654dfc988c-7cgf9\" (UID: \"ca4d18b6-ca1b-472d-a6ff-bfdca90eea1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-86f4c4-654dfc988c-7cgf9"
Nov 02 03:51:23.887332 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:51:23.887312    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca4d18b6-ca1b-472d-a6ff-bfdca90eea1e-maas-api-tls\") pod \"maas-api-e2e-explicit-86f4c4-654dfc988c-7cgf9\" (UID: \"ca4d18b6-ca1b-472d-a6ff-bfdca90eea1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-86f4c4-654dfc988c-7cgf9"
Nov 02 03:51:23.942139 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:51:23.942105    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5g6x\" (UniqueName: \"kubernetes.io/projected/ca4d18b6-ca1b-472d-a6ff-bfdca90eea1e-kube-api-access-t5g6x\") pod \"maas-api-e2e-explicit-86f4c4-654dfc988c-7cgf9\" (UID: \"ca4d18b6-ca1b-472d-a6ff-bfdca90eea1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-86f4c4-654dfc988c-7cgf9"
Nov 02 03:51:23.947940 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:51:23.947917    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-86f4c4-654dfc988c-7cgf9"
Nov 02 03:51:24.334977 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:51:24.334943    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca4d18b6_ca1b_472d_a6ff_bfdca90eea1e.slice/crio-ad07617bb1c6e6245e239389560b7fe373ceb6b9ee13070c42678980009463d5 WatchSource:0}: Error finding container ad07617bb1c6e6245e239389560b7fe373ceb6b9ee13070c42678980009463d5: Status 404 returned error can't find the container with id ad07617bb1c6e6245e239389560b7fe373ceb6b9ee13070c42678980009463d5
Nov 02 03:51:24.337495 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:51:24.337459    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-86f4c4-654dfc988c-7cgf9"]
Nov 02 03:51:24.628422 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:51:24.628340    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-86f4c4-654dfc988c-7cgf9" event={"ID":"ca4d18b6-ca1b-472d-a6ff-bfdca90eea1e","Type":"ContainerStarted","Data":"ad07617bb1c6e6245e239389560b7fe373ceb6b9ee13070c42678980009463d5"}
Nov 02 03:51:25.636361 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:51:25.636317    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-86f4c4-654dfc988c-7cgf9" event={"ID":"ca4d18b6-ca1b-472d-a6ff-bfdca90eea1e","Type":"ContainerStarted","Data":"4a7887a3826c4b36973a8d6dd57eaa443b63f79a8366263a21b4fc3e983ce0b4"}
Nov 02 03:51:25.636765 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:51:25.636410    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-86f4c4-654dfc988c-7cgf9"
Nov 02 03:51:25.776126 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:51:25.776076    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-86f4c4-654dfc988c-7cgf9" podStartSLOduration=1.922840887 podStartE2EDuration="2.776061882s" totalImagesPullingTime="853.220995ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:51:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:51:25.774917625 +0000 UTC m=+3826.353525004" watchObservedRunningTime="2026-11-02 03:51:25.776061882 +0000 UTC m=+3826.354669261"
Nov 02 03:51:31.649728 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:51:31.649698    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-86f4c4-654dfc988c-7cgf9"
Nov 02 03:51:33.527365 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:51:33.527327    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 03:51:35.052970 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:51:35.052918    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-t8dbv"]
Nov 02 03:51:35.053605 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:51:35.053159    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-t8dbv" podUID="4e8b61fe-99dc-4705-ab37-cde3bfad0451" containerName="istio-proxy" containerID="cri-o://af4a995aadfaa406459d1502407866a0b7058ce358bdb7b56f0cfacc0aa1919c" gracePeriod=30
Nov 02 03:51:40.336434 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:51:40.336369    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-t8dbv"
Nov 02 03:51:40.493339 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:51:40.493300    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e8b61fe-99dc-4705-ab37-cde3bfad0451-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e8b61fe-99dc-4705-ab37-cde3bfad0451-workload-certs\") pod \"4e8b61fe-99dc-4705-ab37-cde3bfad0451\" (UID: \"4e8b61fe-99dc-4705-ab37-cde3bfad0451\") "
Nov 02 03:51:40.493537 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:51:40.493388    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e8b61fe-99dc-4705-ab37-cde3bfad0451-istio-token\" (UniqueName: \"kubernetes.io/projected/4e8b61fe-99dc-4705-ab37-cde3bfad0451-istio-token\") pod \"4e8b61fe-99dc-4705-ab37-cde3bfad0451\" (UID: \"4e8b61fe-99dc-4705-ab37-cde3bfad0451\") "
Nov 02 03:51:40.493537 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:51:40.493419    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e8b61fe-99dc-4705-ab37-cde3bfad0451-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e8b61fe-99dc-4705-ab37-cde3bfad0451-istiod-ca-cert\") pod \"4e8b61fe-99dc-4705-ab37-cde3bfad0451\" (UID: \"4e8b61fe-99dc-4705-ab37-cde3bfad0451\") "
Nov 02 03:51:40.493537 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:51:40.493445    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e8b61fe-99dc-4705-ab37-cde3bfad0451-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e8b61fe-99dc-4705-ab37-cde3bfad0451-workload-socket\") pod \"4e8b61fe-99dc-4705-ab37-cde3bfad0451\" (UID: \"4e8b61fe-99dc-4705-ab37-cde3bfad0451\") "
Nov 02 03:51:40.493537 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:51:40.493478    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e8b61fe-99dc-4705-ab37-cde3bfad0451-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e8b61fe-99dc-4705-ab37-cde3bfad0451-istio-envoy\") pod \"4e8b61fe-99dc-4705-ab37-cde3bfad0451\" (UID: \"4e8b61fe-99dc-4705-ab37-cde3bfad0451\") "
Nov 02 03:51:40.493537 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:51:40.493534    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e8b61fe-99dc-4705-ab37-cde3bfad0451-kube-api-access-k5b9s\" (UniqueName: \"kubernetes.io/projected/4e8b61fe-99dc-4705-ab37-cde3bfad0451-kube-api-access-k5b9s\") pod \"4e8b61fe-99dc-4705-ab37-cde3bfad0451\" (UID: \"4e8b61fe-99dc-4705-ab37-cde3bfad0451\") "
Nov 02 03:51:40.493932 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:51:40.493589    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4e8b61fe-99dc-4705-ab37-cde3bfad0451-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e8b61fe-99dc-4705-ab37-cde3bfad0451-istio-podinfo\") pod \"4e8b61fe-99dc-4705-ab37-cde3bfad0451\" (UID: \"4e8b61fe-99dc-4705-ab37-cde3bfad0451\") "
Nov 02 03:51:40.493932 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:51:40.493594    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e8b61fe-99dc-4705-ab37-cde3bfad0451-workload-certs" pod "4e8b61fe-99dc-4705-ab37-cde3bfad0451" (UID: "4e8b61fe-99dc-4705-ab37-cde3bfad0451"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:51:40.493932 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:51:40.493636    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e8b61fe-99dc-4705-ab37-cde3bfad0451-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e8b61fe-99dc-4705-ab37-cde3bfad0451-credential-socket\") pod \"4e8b61fe-99dc-4705-ab37-cde3bfad0451\" (UID: \"4e8b61fe-99dc-4705-ab37-cde3bfad0451\") "
Nov 02 03:51:40.493932 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:51:40.493678    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e8b61fe-99dc-4705-ab37-cde3bfad0451-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e8b61fe-99dc-4705-ab37-cde3bfad0451-istio-data\") pod \"4e8b61fe-99dc-4705-ab37-cde3bfad0451\" (UID: \"4e8b61fe-99dc-4705-ab37-cde3bfad0451\") "
Nov 02 03:51:40.493932 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:51:40.493722    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e8b61fe-99dc-4705-ab37-cde3bfad0451-workload-socket" pod "4e8b61fe-99dc-4705-ab37-cde3bfad0451" (UID: "4e8b61fe-99dc-4705-ab37-cde3bfad0451"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:51:40.494248 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:51:40.494073    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e8b61fe-99dc-4705-ab37-cde3bfad0451-credential-socket" pod "4e8b61fe-99dc-4705-ab37-cde3bfad0451" (UID: "4e8b61fe-99dc-4705-ab37-cde3bfad0451"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:51:40.494248 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:51:40.494108    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e8b61fe-99dc-4705-ab37-cde3bfad0451-istiod-ca-cert" pod "4e8b61fe-99dc-4705-ab37-cde3bfad0451" (UID: "4e8b61fe-99dc-4705-ab37-cde3bfad0451"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:51:40.494367 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:51:40.494267    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e8b61fe-99dc-4705-ab37-cde3bfad0451-istio-data" pod "4e8b61fe-99dc-4705-ab37-cde3bfad0451" (UID: "4e8b61fe-99dc-4705-ab37-cde3bfad0451"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:51:40.494464 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:51:40.494447    4496 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e8b61fe-99dc-4705-ab37-cde3bfad0451-workload-certs\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:51:40.494523 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:51:40.494471    4496 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e8b61fe-99dc-4705-ab37-cde3bfad0451-istiod-ca-cert\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:51:40.494523 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:51:40.494489    4496 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e8b61fe-99dc-4705-ab37-cde3bfad0451-workload-socket\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:51:40.494523 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:51:40.494502    4496 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e8b61fe-99dc-4705-ab37-cde3bfad0451-credential-socket\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:51:40.494523 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:51:40.494516    4496 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e8b61fe-99dc-4705-ab37-cde3bfad0451-istio-data\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:51:40.496350 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:51:40.496320    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e8b61fe-99dc-4705-ab37-cde3bfad0451-kube-api-access-k5b9s" pod "4e8b61fe-99dc-4705-ab37-cde3bfad0451" (UID: "4e8b61fe-99dc-4705-ab37-cde3bfad0451"). InnerVolumeSpecName "kube-api-access-k5b9s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:51:40.496480 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:51:40.496328    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4e8b61fe-99dc-4705-ab37-cde3bfad0451-istio-podinfo" pod "4e8b61fe-99dc-4705-ab37-cde3bfad0451" (UID: "4e8b61fe-99dc-4705-ab37-cde3bfad0451"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 03:51:40.496480 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:51:40.496334    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e8b61fe-99dc-4705-ab37-cde3bfad0451-istio-token" pod "4e8b61fe-99dc-4705-ab37-cde3bfad0451" (UID: "4e8b61fe-99dc-4705-ab37-cde3bfad0451"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:51:40.496480 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:51:40.496414    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e8b61fe-99dc-4705-ab37-cde3bfad0451-istio-envoy" pod "4e8b61fe-99dc-4705-ab37-cde3bfad0451" (UID: "4e8b61fe-99dc-4705-ab37-cde3bfad0451"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:51:40.596333 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:51:40.596255    4496 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e8b61fe-99dc-4705-ab37-cde3bfad0451-istio-envoy\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:51:40.596333 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:51:40.596286    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5b9s\" (UniqueName: \"kubernetes.io/projected/4e8b61fe-99dc-4705-ab37-cde3bfad0451-kube-api-access-k5b9s\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:51:40.596333 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:51:40.596296    4496 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e8b61fe-99dc-4705-ab37-cde3bfad0451-istio-podinfo\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:51:40.596333 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:51:40.596304    4496 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e8b61fe-99dc-4705-ab37-cde3bfad0451-istio-token\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:51:40.753389 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:51:40.753351    4496 generic.go:356] "Generic (PLEG): container finished" podID="4e8b61fe-99dc-4705-ab37-cde3bfad0451" containerID="af4a995aadfaa406459d1502407866a0b7058ce358bdb7b56f0cfacc0aa1919c" exitCode=0
Nov 02 03:51:40.753591 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:51:40.753411    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-t8dbv"
Nov 02 03:51:40.753591 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:51:40.753443    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-t8dbv" event={"ID":"4e8b61fe-99dc-4705-ab37-cde3bfad0451","Type":"ContainerDied","Data":"af4a995aadfaa406459d1502407866a0b7058ce358bdb7b56f0cfacc0aa1919c"}
Nov 02 03:51:40.753591 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:51:40.753500    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-t8dbv" event={"ID":"4e8b61fe-99dc-4705-ab37-cde3bfad0451","Type":"ContainerDied","Data":"da1334dee55b29a96f10536df49e28768532723642d95f7dce3a1accdd99ec94"}
Nov 02 03:51:40.753591 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:51:40.753526    4496 scope.go:122] "RemoveContainer" containerID="af4a995aadfaa406459d1502407866a0b7058ce358bdb7b56f0cfacc0aa1919c"
Nov 02 03:51:40.765509 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:51:40.765491    4496 scope.go:122] "RemoveContainer" containerID="af4a995aadfaa406459d1502407866a0b7058ce358bdb7b56f0cfacc0aa1919c"
Nov 02 03:51:40.765832 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:51:40.765798    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af4a995aadfaa406459d1502407866a0b7058ce358bdb7b56f0cfacc0aa1919c\": container with ID starting with af4a995aadfaa406459d1502407866a0b7058ce358bdb7b56f0cfacc0aa1919c not found: ID does not exist" containerID="af4a995aadfaa406459d1502407866a0b7058ce358bdb7b56f0cfacc0aa1919c"
Nov 02 03:51:40.765912 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:51:40.765842    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af4a995aadfaa406459d1502407866a0b7058ce358bdb7b56f0cfacc0aa1919c"} err="failed to get container status \"af4a995aadfaa406459d1502407866a0b7058ce358bdb7b56f0cfacc0aa1919c\": rpc error: code = NotFound desc = could not find container \"af4a995aadfaa406459d1502407866a0b7058ce358bdb7b56f0cfacc0aa1919c\": container with ID starting with af4a995aadfaa406459d1502407866a0b7058ce358bdb7b56f0cfacc0aa1919c not found: ID does not exist"
Nov 02 03:51:40.921737 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:51:40.921667    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-t8dbv"]
Nov 02 03:51:40.925116 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:51:40.925092    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-t8dbv"]
Nov 02 03:51:41.577263 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:51:41.577232    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e8b61fe-99dc-4705-ab37-cde3bfad0451" path="/var/lib/kubelet/pods/4e8b61fe-99dc-4705-ab37-cde3bfad0451/volumes"
Nov 02 03:52:03.527113 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:52:03.527075    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 03:52:15.629330 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:52:15.629294    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-147104/auth-test-model-147104-kserve-74f4568577-85cnx"]
Nov 02 03:52:15.629999 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:52:15.629983    4496 cpu_manager.go:414] "containerMap: removing container" podUID="4e8b61fe-99dc-4705-ab37-cde3bfad0451" containerName="istio-proxy"
Nov 02 03:52:15.630067 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:52:15.629999    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e8b61fe-99dc-4705-ab37-cde3bfad0451" containerName="istio-proxy"
Nov 02 03:52:15.630123 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:52:15.630113    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e8b61fe-99dc-4705-ab37-cde3bfad0451" containerName="istio-proxy"
Nov 02 03:52:15.633507 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:52:15.633488    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-147104/auth-test-model-147104-kserve-74f4568577-85cnx"
Nov 02 03:52:15.690495 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:52:15.690458    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-147104/auth-test-model-147104-kserve-74f4568577-85cnx"]
Nov 02 03:52:15.735992 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:52:15.735947    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-147104\"/\"openshift-service-ca.crt\""
Nov 02 03:52:15.735992 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:52:15.735960    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-147104\"/\"kube-root-ca.crt\""
Nov 02 03:52:15.737168 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:52:15.737148    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-147104\"/\"default-dockercfg-5zgmb\""
Nov 02 03:52:15.737253 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:52:15.737179    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-147104\"/\"auth-test-model-147104-kserve-self-signed-certs\""
Nov 02 03:52:15.821988 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:52:15.821951    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d10c8e8-9717-4cdc-a7df-229ab28ae02a-model-cache\") pod \"auth-test-model-147104-kserve-74f4568577-85cnx\" (UID: \"4d10c8e8-9717-4cdc-a7df-229ab28ae02a\") " pod="ai-tenant-e2e-shared-b-w0-147104/auth-test-model-147104-kserve-74f4568577-85cnx"
Nov 02 03:52:15.822218 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:52:15.822058    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d10c8e8-9717-4cdc-a7df-229ab28ae02a-tls-certs\") pod \"auth-test-model-147104-kserve-74f4568577-85cnx\" (UID: \"4d10c8e8-9717-4cdc-a7df-229ab28ae02a\") " pod="ai-tenant-e2e-shared-b-w0-147104/auth-test-model-147104-kserve-74f4568577-85cnx"
Nov 02 03:52:15.822218 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:52:15.822138    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d10c8e8-9717-4cdc-a7df-229ab28ae02a-home\") pod \"auth-test-model-147104-kserve-74f4568577-85cnx\" (UID: \"4d10c8e8-9717-4cdc-a7df-229ab28ae02a\") " pod="ai-tenant-e2e-shared-b-w0-147104/auth-test-model-147104-kserve-74f4568577-85cnx"
Nov 02 03:52:15.822218 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:52:15.822184    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d10c8e8-9717-4cdc-a7df-229ab28ae02a-tmp-dir\") pod \"auth-test-model-147104-kserve-74f4568577-85cnx\" (UID: \"4d10c8e8-9717-4cdc-a7df-229ab28ae02a\") " pod="ai-tenant-e2e-shared-b-w0-147104/auth-test-model-147104-kserve-74f4568577-85cnx"
Nov 02 03:52:15.822377 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:52:15.822239    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4zb2\" (UniqueName: \"kubernetes.io/projected/4d10c8e8-9717-4cdc-a7df-229ab28ae02a-kube-api-access-c4zb2\") pod \"auth-test-model-147104-kserve-74f4568577-85cnx\" (UID: \"4d10c8e8-9717-4cdc-a7df-229ab28ae02a\") " pod="ai-tenant-e2e-shared-b-w0-147104/auth-test-model-147104-kserve-74f4568577-85cnx"
Nov 02 03:52:15.822377 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:52:15.822339    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d10c8e8-9717-4cdc-a7df-229ab28ae02a-dshm\") pod \"auth-test-model-147104-kserve-74f4568577-85cnx\" (UID: \"4d10c8e8-9717-4cdc-a7df-229ab28ae02a\") " pod="ai-tenant-e2e-shared-b-w0-147104/auth-test-model-147104-kserve-74f4568577-85cnx"
Nov 02 03:52:15.923348 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:52:15.923265    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d10c8e8-9717-4cdc-a7df-229ab28ae02a-model-cache\") pod \"auth-test-model-147104-kserve-74f4568577-85cnx\" (UID: \"4d10c8e8-9717-4cdc-a7df-229ab28ae02a\") " pod="ai-tenant-e2e-shared-b-w0-147104/auth-test-model-147104-kserve-74f4568577-85cnx"
Nov 02 03:52:15.923348 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:52:15.923308    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d10c8e8-9717-4cdc-a7df-229ab28ae02a-tls-certs\") pod \"auth-test-model-147104-kserve-74f4568577-85cnx\" (UID: \"4d10c8e8-9717-4cdc-a7df-229ab28ae02a\") " pod="ai-tenant-e2e-shared-b-w0-147104/auth-test-model-147104-kserve-74f4568577-85cnx"
Nov 02 03:52:15.923348 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:52:15.923333    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d10c8e8-9717-4cdc-a7df-229ab28ae02a-home\") pod \"auth-test-model-147104-kserve-74f4568577-85cnx\" (UID: \"4d10c8e8-9717-4cdc-a7df-229ab28ae02a\") " pod="ai-tenant-e2e-shared-b-w0-147104/auth-test-model-147104-kserve-74f4568577-85cnx"
Nov 02 03:52:15.923622 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:52:15.923360    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d10c8e8-9717-4cdc-a7df-229ab28ae02a-tmp-dir\") pod \"auth-test-model-147104-kserve-74f4568577-85cnx\" (UID: \"4d10c8e8-9717-4cdc-a7df-229ab28ae02a\") " pod="ai-tenant-e2e-shared-b-w0-147104/auth-test-model-147104-kserve-74f4568577-85cnx"
Nov 02 03:52:15.923622 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:52:15.923401    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4zb2\" (UniqueName: \"kubernetes.io/projected/4d10c8e8-9717-4cdc-a7df-229ab28ae02a-kube-api-access-c4zb2\") pod \"auth-test-model-147104-kserve-74f4568577-85cnx\" (UID: \"4d10c8e8-9717-4cdc-a7df-229ab28ae02a\") " pod="ai-tenant-e2e-shared-b-w0-147104/auth-test-model-147104-kserve-74f4568577-85cnx"
Nov 02 03:52:15.923622 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:52:15.923450    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d10c8e8-9717-4cdc-a7df-229ab28ae02a-dshm\") pod \"auth-test-model-147104-kserve-74f4568577-85cnx\" (UID: \"4d10c8e8-9717-4cdc-a7df-229ab28ae02a\") " pod="ai-tenant-e2e-shared-b-w0-147104/auth-test-model-147104-kserve-74f4568577-85cnx"
Nov 02 03:52:15.923796 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:52:15.923723    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d10c8e8-9717-4cdc-a7df-229ab28ae02a-model-cache\") pod \"auth-test-model-147104-kserve-74f4568577-85cnx\" (UID: \"4d10c8e8-9717-4cdc-a7df-229ab28ae02a\") " pod="ai-tenant-e2e-shared-b-w0-147104/auth-test-model-147104-kserve-74f4568577-85cnx"
Nov 02 03:52:15.923796 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:52:15.923740    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d10c8e8-9717-4cdc-a7df-229ab28ae02a-home\") pod \"auth-test-model-147104-kserve-74f4568577-85cnx\" (UID: \"4d10c8e8-9717-4cdc-a7df-229ab28ae02a\") " pod="ai-tenant-e2e-shared-b-w0-147104/auth-test-model-147104-kserve-74f4568577-85cnx"
Nov 02 03:52:15.923892 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:52:15.923809    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d10c8e8-9717-4cdc-a7df-229ab28ae02a-tmp-dir\") pod \"auth-test-model-147104-kserve-74f4568577-85cnx\" (UID: \"4d10c8e8-9717-4cdc-a7df-229ab28ae02a\") " pod="ai-tenant-e2e-shared-b-w0-147104/auth-test-model-147104-kserve-74f4568577-85cnx"
Nov 02 03:52:15.925615 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:52:15.925591    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d10c8e8-9717-4cdc-a7df-229ab28ae02a-dshm\") pod \"auth-test-model-147104-kserve-74f4568577-85cnx\" (UID: \"4d10c8e8-9717-4cdc-a7df-229ab28ae02a\") " pod="ai-tenant-e2e-shared-b-w0-147104/auth-test-model-147104-kserve-74f4568577-85cnx"
Nov 02 03:52:15.926165 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:52:15.926147    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d10c8e8-9717-4cdc-a7df-229ab28ae02a-tls-certs\") pod \"auth-test-model-147104-kserve-74f4568577-85cnx\" (UID: \"4d10c8e8-9717-4cdc-a7df-229ab28ae02a\") " pod="ai-tenant-e2e-shared-b-w0-147104/auth-test-model-147104-kserve-74f4568577-85cnx"
Nov 02 03:52:15.981169 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:52:15.981140    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4zb2\" (UniqueName: \"kubernetes.io/projected/4d10c8e8-9717-4cdc-a7df-229ab28ae02a-kube-api-access-c4zb2\") pod \"auth-test-model-147104-kserve-74f4568577-85cnx\" (UID: \"4d10c8e8-9717-4cdc-a7df-229ab28ae02a\") " pod="ai-tenant-e2e-shared-b-w0-147104/auth-test-model-147104-kserve-74f4568577-85cnx"
Nov 02 03:52:16.246453 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:52:16.246419    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-147104/auth-test-model-147104-kserve-74f4568577-85cnx"
Nov 02 03:52:16.631365 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:52:16.631331    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-147104/auth-test-model-147104-kserve-74f4568577-85cnx"]
Nov 02 03:52:16.631710 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:52:16.631538    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4d10c8e8_9717_4cdc_a7df_229ab28ae02a.slice/crio-e56c6a7ffb19baee64cca515cda8508f6d76a402737be96e5c8370c5ff724d3d WatchSource:0}: Error finding container e56c6a7ffb19baee64cca515cda8508f6d76a402737be96e5c8370c5ff724d3d: Status 404 returned error can't find the container with id e56c6a7ffb19baee64cca515cda8508f6d76a402737be96e5c8370c5ff724d3d
Nov 02 03:52:17.021344 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:52:17.021308    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-147104/auth-test-model-147104-kserve-74f4568577-85cnx" event={"ID":"4d10c8e8-9717-4cdc-a7df-229ab28ae02a","Type":"ContainerStarted","Data":"33a2a5e1488ba81c27ea28fa1b801d59f7f619bc85969005bbc4bfd2b8dc501e"}
Nov 02 03:52:17.021535 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:52:17.021356    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-147104/auth-test-model-147104-kserve-74f4568577-85cnx" event={"ID":"4d10c8e8-9717-4cdc-a7df-229ab28ae02a","Type":"ContainerStarted","Data":"e56c6a7ffb19baee64cca515cda8508f6d76a402737be96e5c8370c5ff724d3d"}
Nov 02 03:52:17.141807 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:52:17.141750    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-147104/auth-test-model-147104-kserve-74f4568577-85cnx" podStartSLOduration=2.141735564 podStartE2EDuration="2.141735564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:52:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:52:17.139431087 +0000 UTC m=+3877.718038468" watchObservedRunningTime="2026-11-02 03:52:17.141735564 +0000 UTC m=+3877.720342963"
Nov 02 03:52:17.247004 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:52:17.246968    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-147104/auth-test-model-147104-kserve-74f4568577-85cnx"
Nov 02 03:52:26.246607 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:52:26.246570    4496 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-147104/auth-test-model-147104-kserve-74f4568577-85cnx"
Nov 02 03:52:26.259802 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:52:26.259776    4496 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-147104/auth-test-model-147104-kserve-74f4568577-85cnx"
Nov 02 03:52:27.109205 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:52:27.109175    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-147104/auth-test-model-147104-kserve-74f4568577-85cnx"
Nov 02 03:52:33.527488 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:52:33.527448    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 03:52:40.383327 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:52:40.383299    4496 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hww8q_5c797b08-a314-44ed-8026-8a0ce1f6c36f/ovn-acl-logging/0.log"
Nov 02 03:52:40.394058 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:52:40.394036    4496 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hww8q_5c797b08-a314-44ed-8026-8a0ce1f6c36f/ovn-acl-logging/0.log"
Nov 02 03:53:03.528449 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:53:03.528410    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 03:53:20.889960 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:53:20.889925    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xzskx"]
Nov 02 03:53:20.890454 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:53:20.890166    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xzskx" podUID="19b510d7-4f25-4f4a-8c61-ce8a6013229a" containerName="istio-proxy" containerID="cri-o://949d5401aba010bb77183a09e430b5ffe788b614f8dc1b2e9086b598df6ed780" gracePeriod=30
Nov 02 03:53:25.305841 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:53:25.305805    4496 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xzskx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 03:53:25.306347 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:53:25.305873    4496 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xzskx" podUID="19b510d7-4f25-4f4a-8c61-ce8a6013229a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 03:53:26.156341 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:53:26.156279    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xzskx"
Nov 02 03:53:26.288998 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:53:26.288957    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/19b510d7-4f25-4f4a-8c61-ce8a6013229a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19b510d7-4f25-4f4a-8c61-ce8a6013229a-istio-podinfo\") pod \"19b510d7-4f25-4f4a-8c61-ce8a6013229a\" (UID: \"19b510d7-4f25-4f4a-8c61-ce8a6013229a\") "
Nov 02 03:53:26.288998 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:53:26.289026    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19b510d7-4f25-4f4a-8c61-ce8a6013229a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19b510d7-4f25-4f4a-8c61-ce8a6013229a-credential-socket\") pod \"19b510d7-4f25-4f4a-8c61-ce8a6013229a\" (UID: \"19b510d7-4f25-4f4a-8c61-ce8a6013229a\") "
Nov 02 03:53:26.289283 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:53:26.289070    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19b510d7-4f25-4f4a-8c61-ce8a6013229a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19b510d7-4f25-4f4a-8c61-ce8a6013229a-istio-envoy\") pod \"19b510d7-4f25-4f4a-8c61-ce8a6013229a\" (UID: \"19b510d7-4f25-4f4a-8c61-ce8a6013229a\") "
Nov 02 03:53:26.289283 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:53:26.289197    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19b510d7-4f25-4f4a-8c61-ce8a6013229a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19b510d7-4f25-4f4a-8c61-ce8a6013229a-istio-data\") pod \"19b510d7-4f25-4f4a-8c61-ce8a6013229a\" (UID: \"19b510d7-4f25-4f4a-8c61-ce8a6013229a\") "
Nov 02 03:53:26.289283 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:53:26.289260    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19b510d7-4f25-4f4a-8c61-ce8a6013229a-istio-token\" (UniqueName: \"kubernetes.io/projected/19b510d7-4f25-4f4a-8c61-ce8a6013229a-istio-token\") pod \"19b510d7-4f25-4f4a-8c61-ce8a6013229a\" (UID: \"19b510d7-4f25-4f4a-8c61-ce8a6013229a\") "
Nov 02 03:53:26.289454 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:53:26.289301    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19b510d7-4f25-4f4a-8c61-ce8a6013229a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19b510d7-4f25-4f4a-8c61-ce8a6013229a-workload-socket\") pod \"19b510d7-4f25-4f4a-8c61-ce8a6013229a\" (UID: \"19b510d7-4f25-4f4a-8c61-ce8a6013229a\") "
Nov 02 03:53:26.289454 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:53:26.289359    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19b510d7-4f25-4f4a-8c61-ce8a6013229a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19b510d7-4f25-4f4a-8c61-ce8a6013229a-workload-certs\") pod \"19b510d7-4f25-4f4a-8c61-ce8a6013229a\" (UID: \"19b510d7-4f25-4f4a-8c61-ce8a6013229a\") "
Nov 02 03:53:26.289454 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:53:26.289397    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19b510d7-4f25-4f4a-8c61-ce8a6013229a-kube-api-access-lqrl5\" (UniqueName: \"kubernetes.io/projected/19b510d7-4f25-4f4a-8c61-ce8a6013229a-kube-api-access-lqrl5\") pod \"19b510d7-4f25-4f4a-8c61-ce8a6013229a\" (UID: \"19b510d7-4f25-4f4a-8c61-ce8a6013229a\") "
Nov 02 03:53:26.289454 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:53:26.289396    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19b510d7-4f25-4f4a-8c61-ce8a6013229a-credential-socket" pod "19b510d7-4f25-4f4a-8c61-ce8a6013229a" (UID: "19b510d7-4f25-4f4a-8c61-ce8a6013229a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:53:26.289752 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:53:26.289493    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19b510d7-4f25-4f4a-8c61-ce8a6013229a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19b510d7-4f25-4f4a-8c61-ce8a6013229a-istiod-ca-cert\") pod \"19b510d7-4f25-4f4a-8c61-ce8a6013229a\" (UID: \"19b510d7-4f25-4f4a-8c61-ce8a6013229a\") "
Nov 02 03:53:26.289752 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:53:26.289493    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19b510d7-4f25-4f4a-8c61-ce8a6013229a-istio-data" pod "19b510d7-4f25-4f4a-8c61-ce8a6013229a" (UID: "19b510d7-4f25-4f4a-8c61-ce8a6013229a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:53:26.289752 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:53:26.289640    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19b510d7-4f25-4f4a-8c61-ce8a6013229a-workload-certs" pod "19b510d7-4f25-4f4a-8c61-ce8a6013229a" (UID: "19b510d7-4f25-4f4a-8c61-ce8a6013229a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:53:26.289929 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:53:26.289775    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19b510d7-4f25-4f4a-8c61-ce8a6013229a-workload-socket" pod "19b510d7-4f25-4f4a-8c61-ce8a6013229a" (UID: "19b510d7-4f25-4f4a-8c61-ce8a6013229a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:53:26.289929 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:53:26.289864    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19b510d7-4f25-4f4a-8c61-ce8a6013229a-istiod-ca-cert" pod "19b510d7-4f25-4f4a-8c61-ce8a6013229a" (UID: "19b510d7-4f25-4f4a-8c61-ce8a6013229a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:53:26.290153 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:53:26.290132    4496 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19b510d7-4f25-4f4a-8c61-ce8a6013229a-istio-data\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:53:26.290231 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:53:26.290158    4496 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19b510d7-4f25-4f4a-8c61-ce8a6013229a-workload-socket\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:53:26.290231 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:53:26.290173    4496 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19b510d7-4f25-4f4a-8c61-ce8a6013229a-workload-certs\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:53:26.290231 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:53:26.290188    4496 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19b510d7-4f25-4f4a-8c61-ce8a6013229a-istiod-ca-cert\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:53:26.290231 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:53:26.290200    4496 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19b510d7-4f25-4f4a-8c61-ce8a6013229a-credential-socket\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:53:26.291586 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:53:26.291560    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/19b510d7-4f25-4f4a-8c61-ce8a6013229a-istio-podinfo" pod "19b510d7-4f25-4f4a-8c61-ce8a6013229a" (UID: "19b510d7-4f25-4f4a-8c61-ce8a6013229a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 03:53:26.291755 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:53:26.291680    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19b510d7-4f25-4f4a-8c61-ce8a6013229a-istio-envoy" pod "19b510d7-4f25-4f4a-8c61-ce8a6013229a" (UID: "19b510d7-4f25-4f4a-8c61-ce8a6013229a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:53:26.291856 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:53:26.291834    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19b510d7-4f25-4f4a-8c61-ce8a6013229a-kube-api-access-lqrl5" pod "19b510d7-4f25-4f4a-8c61-ce8a6013229a" (UID: "19b510d7-4f25-4f4a-8c61-ce8a6013229a"). InnerVolumeSpecName "kube-api-access-lqrl5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:53:26.292251 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:53:26.292234    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19b510d7-4f25-4f4a-8c61-ce8a6013229a-istio-token" pod "19b510d7-4f25-4f4a-8c61-ce8a6013229a" (UID: "19b510d7-4f25-4f4a-8c61-ce8a6013229a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:53:26.390954 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:53:26.390921    4496 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19b510d7-4f25-4f4a-8c61-ce8a6013229a-istio-podinfo\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:53:26.390954 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:53:26.390952    4496 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19b510d7-4f25-4f4a-8c61-ce8a6013229a-istio-envoy\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:53:26.391366 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:53:26.390962    4496 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19b510d7-4f25-4f4a-8c61-ce8a6013229a-istio-token\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:53:26.391366 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:53:26.390972    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqrl5\" (UniqueName: \"kubernetes.io/projected/19b510d7-4f25-4f4a-8c61-ce8a6013229a-kube-api-access-lqrl5\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:53:26.577816 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:53:26.577780    4496 generic.go:356] "Generic (PLEG): container finished" podID="19b510d7-4f25-4f4a-8c61-ce8a6013229a" containerID="949d5401aba010bb77183a09e430b5ffe788b614f8dc1b2e9086b598df6ed780" exitCode=0
Nov 02 03:53:26.578030 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:53:26.577843    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xzskx"
Nov 02 03:53:26.578030 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:53:26.577858    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xzskx" event={"ID":"19b510d7-4f25-4f4a-8c61-ce8a6013229a","Type":"ContainerDied","Data":"949d5401aba010bb77183a09e430b5ffe788b614f8dc1b2e9086b598df6ed780"}
Nov 02 03:53:26.578030 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:53:26.577924    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xzskx" event={"ID":"19b510d7-4f25-4f4a-8c61-ce8a6013229a","Type":"ContainerDied","Data":"ca115b2434ad9b7f2e66e3a2faba29c70b695fc3d0324530103ee62b3a9fc44c"}
Nov 02 03:53:26.578030 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:53:26.577953    4496 scope.go:122] "RemoveContainer" containerID="949d5401aba010bb77183a09e430b5ffe788b614f8dc1b2e9086b598df6ed780"
Nov 02 03:53:26.590145 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:53:26.590124    4496 scope.go:122] "RemoveContainer" containerID="949d5401aba010bb77183a09e430b5ffe788b614f8dc1b2e9086b598df6ed780"
Nov 02 03:53:26.590434 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:53:26.590413    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"949d5401aba010bb77183a09e430b5ffe788b614f8dc1b2e9086b598df6ed780\": container with ID starting with 949d5401aba010bb77183a09e430b5ffe788b614f8dc1b2e9086b598df6ed780 not found: ID does not exist" containerID="949d5401aba010bb77183a09e430b5ffe788b614f8dc1b2e9086b598df6ed780"
Nov 02 03:53:26.590487 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:53:26.590441    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"949d5401aba010bb77183a09e430b5ffe788b614f8dc1b2e9086b598df6ed780"} err="failed to get container status \"949d5401aba010bb77183a09e430b5ffe788b614f8dc1b2e9086b598df6ed780\": rpc error: code = NotFound desc = could not find container \"949d5401aba010bb77183a09e430b5ffe788b614f8dc1b2e9086b598df6ed780\": container with ID starting with 949d5401aba010bb77183a09e430b5ffe788b614f8dc1b2e9086b598df6ed780 not found: ID does not exist"
Nov 02 03:53:26.745341 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:53:26.745309    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xzskx"]
Nov 02 03:53:26.795873 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:53:26.795840    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xzskx"]
Nov 02 03:53:27.574914 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:53:27.574879    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19b510d7-4f25-4f4a-8c61-ce8a6013229a" path="/var/lib/kubelet/pods/19b510d7-4f25-4f4a-8c61-ce8a6013229a/volumes"
Nov 02 03:53:30.229415 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:53:30.229379    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-3b27a5/auth-test-model-3b27a5-kserve-74c5647cbf-zfzcs"]
Nov 02 03:53:30.230483 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:53:30.230459    4496 cpu_manager.go:414] "containerMap: removing container" podUID="19b510d7-4f25-4f4a-8c61-ce8a6013229a" containerName="istio-proxy"
Nov 02 03:53:30.230558 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:53:30.230486    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19b510d7-4f25-4f4a-8c61-ce8a6013229a" containerName="istio-proxy"
Nov 02 03:53:30.230706 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:53:30.230693    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="19b510d7-4f25-4f4a-8c61-ce8a6013229a" containerName="istio-proxy"
Nov 02 03:53:30.236282 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:53:30.236263    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-3b27a5/auth-test-model-3b27a5-kserve-74c5647cbf-zfzcs"
Nov 02 03:53:30.289924 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:53:30.289716    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-3b27a5/auth-test-model-3b27a5-kserve-74c5647cbf-zfzcs"]
Nov 02 03:53:30.334702 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:53:30.334674    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-3b27a5\"/\"auth-test-model-3b27a5-kserve-self-signed-certs\""
Nov 02 03:53:30.334702 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:53:30.334708    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-3b27a5\"/\"default-dockercfg-nkckn\""
Nov 02 03:53:30.339435 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:53:30.339411    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ad2c2f7-2198-4ec7-bd1c-bcf5ab3cc455-home\") pod \"auth-test-model-3b27a5-kserve-74c5647cbf-zfzcs\" (UID: \"9ad2c2f7-2198-4ec7-bd1c-bcf5ab3cc455\") " pod="ai-tenant-e2e-shared-b-w1-3b27a5/auth-test-model-3b27a5-kserve-74c5647cbf-zfzcs"
Nov 02 03:53:30.339500 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:53:30.339451    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9xzd\" (UniqueName: \"kubernetes.io/projected/9ad2c2f7-2198-4ec7-bd1c-bcf5ab3cc455-kube-api-access-f9xzd\") pod \"auth-test-model-3b27a5-kserve-74c5647cbf-zfzcs\" (UID: \"9ad2c2f7-2198-4ec7-bd1c-bcf5ab3cc455\") " pod="ai-tenant-e2e-shared-b-w1-3b27a5/auth-test-model-3b27a5-kserve-74c5647cbf-zfzcs"
Nov 02 03:53:30.339558 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:53:30.339538    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ad2c2f7-2198-4ec7-bd1c-bcf5ab3cc455-tls-certs\") pod \"auth-test-model-3b27a5-kserve-74c5647cbf-zfzcs\" (UID: \"9ad2c2f7-2198-4ec7-bd1c-bcf5ab3cc455\") " pod="ai-tenant-e2e-shared-b-w1-3b27a5/auth-test-model-3b27a5-kserve-74c5647cbf-zfzcs"
Nov 02 03:53:30.339600 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:53:30.339575    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ad2c2f7-2198-4ec7-bd1c-bcf5ab3cc455-dshm\") pod \"auth-test-model-3b27a5-kserve-74c5647cbf-zfzcs\" (UID: \"9ad2c2f7-2198-4ec7-bd1c-bcf5ab3cc455\") " pod="ai-tenant-e2e-shared-b-w1-3b27a5/auth-test-model-3b27a5-kserve-74c5647cbf-zfzcs"
Nov 02 03:53:30.339639 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:53:30.339617    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ad2c2f7-2198-4ec7-bd1c-bcf5ab3cc455-model-cache\") pod \"auth-test-model-3b27a5-kserve-74c5647cbf-zfzcs\" (UID: \"9ad2c2f7-2198-4ec7-bd1c-bcf5ab3cc455\") " pod="ai-tenant-e2e-shared-b-w1-3b27a5/auth-test-model-3b27a5-kserve-74c5647cbf-zfzcs"
Nov 02 03:53:30.339682 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:53:30.339668    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ad2c2f7-2198-4ec7-bd1c-bcf5ab3cc455-tmp-dir\") pod \"auth-test-model-3b27a5-kserve-74c5647cbf-zfzcs\" (UID: \"9ad2c2f7-2198-4ec7-bd1c-bcf5ab3cc455\") " pod="ai-tenant-e2e-shared-b-w1-3b27a5/auth-test-model-3b27a5-kserve-74c5647cbf-zfzcs"
Nov 02 03:53:30.339844 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:53:30.339832    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-3b27a5\"/\"openshift-service-ca.crt\""
Nov 02 03:53:30.339883 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:53:30.339859    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-3b27a5\"/\"kube-root-ca.crt\""
Nov 02 03:53:30.441848 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:53:30.441818    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ad2c2f7-2198-4ec7-bd1c-bcf5ab3cc455-home\") pod \"auth-test-model-3b27a5-kserve-74c5647cbf-zfzcs\" (UID: \"9ad2c2f7-2198-4ec7-bd1c-bcf5ab3cc455\") " pod="ai-tenant-e2e-shared-b-w1-3b27a5/auth-test-model-3b27a5-kserve-74c5647cbf-zfzcs"
Nov 02 03:53:30.442035 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:53:30.441857    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9xzd\" (UniqueName: \"kubernetes.io/projected/9ad2c2f7-2198-4ec7-bd1c-bcf5ab3cc455-kube-api-access-f9xzd\") pod \"auth-test-model-3b27a5-kserve-74c5647cbf-zfzcs\" (UID: \"9ad2c2f7-2198-4ec7-bd1c-bcf5ab3cc455\") " pod="ai-tenant-e2e-shared-b-w1-3b27a5/auth-test-model-3b27a5-kserve-74c5647cbf-zfzcs"
Nov 02 03:53:30.442035 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:53:30.441913    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ad2c2f7-2198-4ec7-bd1c-bcf5ab3cc455-tls-certs\") pod \"auth-test-model-3b27a5-kserve-74c5647cbf-zfzcs\" (UID: \"9ad2c2f7-2198-4ec7-bd1c-bcf5ab3cc455\") " pod="ai-tenant-e2e-shared-b-w1-3b27a5/auth-test-model-3b27a5-kserve-74c5647cbf-zfzcs"
Nov 02 03:53:30.442035 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:53:30.441947    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ad2c2f7-2198-4ec7-bd1c-bcf5ab3cc455-dshm\") pod \"auth-test-model-3b27a5-kserve-74c5647cbf-zfzcs\" (UID: \"9ad2c2f7-2198-4ec7-bd1c-bcf5ab3cc455\") " pod="ai-tenant-e2e-shared-b-w1-3b27a5/auth-test-model-3b27a5-kserve-74c5647cbf-zfzcs"
Nov 02 03:53:30.442035 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:53:30.441988    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ad2c2f7-2198-4ec7-bd1c-bcf5ab3cc455-model-cache\") pod \"auth-test-model-3b27a5-kserve-74c5647cbf-zfzcs\" (UID: \"9ad2c2f7-2198-4ec7-bd1c-bcf5ab3cc455\") " pod="ai-tenant-e2e-shared-b-w1-3b27a5/auth-test-model-3b27a5-kserve-74c5647cbf-zfzcs"
Nov 02 03:53:30.442263 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:53:30.442054    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ad2c2f7-2198-4ec7-bd1c-bcf5ab3cc455-tmp-dir\") pod \"auth-test-model-3b27a5-kserve-74c5647cbf-zfzcs\" (UID: \"9ad2c2f7-2198-4ec7-bd1c-bcf5ab3cc455\") " pod="ai-tenant-e2e-shared-b-w1-3b27a5/auth-test-model-3b27a5-kserve-74c5647cbf-zfzcs"
Nov 02 03:53:30.442464 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:53:30.442441    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ad2c2f7-2198-4ec7-bd1c-bcf5ab3cc455-home\") pod \"auth-test-model-3b27a5-kserve-74c5647cbf-zfzcs\" (UID: \"9ad2c2f7-2198-4ec7-bd1c-bcf5ab3cc455\") " pod="ai-tenant-e2e-shared-b-w1-3b27a5/auth-test-model-3b27a5-kserve-74c5647cbf-zfzcs"
Nov 02 03:53:30.442464 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:53:30.442457    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ad2c2f7-2198-4ec7-bd1c-bcf5ab3cc455-model-cache\") pod \"auth-test-model-3b27a5-kserve-74c5647cbf-zfzcs\" (UID: \"9ad2c2f7-2198-4ec7-bd1c-bcf5ab3cc455\") " pod="ai-tenant-e2e-shared-b-w1-3b27a5/auth-test-model-3b27a5-kserve-74c5647cbf-zfzcs"
Nov 02 03:53:30.442588 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:53:30.442516    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ad2c2f7-2198-4ec7-bd1c-bcf5ab3cc455-tmp-dir\") pod \"auth-test-model-3b27a5-kserve-74c5647cbf-zfzcs\" (UID: \"9ad2c2f7-2198-4ec7-bd1c-bcf5ab3cc455\") " pod="ai-tenant-e2e-shared-b-w1-3b27a5/auth-test-model-3b27a5-kserve-74c5647cbf-zfzcs"
Nov 02 03:53:30.444244 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:53:30.444218    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ad2c2f7-2198-4ec7-bd1c-bcf5ab3cc455-dshm\") pod \"auth-test-model-3b27a5-kserve-74c5647cbf-zfzcs\" (UID: \"9ad2c2f7-2198-4ec7-bd1c-bcf5ab3cc455\") " pod="ai-tenant-e2e-shared-b-w1-3b27a5/auth-test-model-3b27a5-kserve-74c5647cbf-zfzcs"
Nov 02 03:53:30.444539 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:53:30.444521    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ad2c2f7-2198-4ec7-bd1c-bcf5ab3cc455-tls-certs\") pod \"auth-test-model-3b27a5-kserve-74c5647cbf-zfzcs\" (UID: \"9ad2c2f7-2198-4ec7-bd1c-bcf5ab3cc455\") " pod="ai-tenant-e2e-shared-b-w1-3b27a5/auth-test-model-3b27a5-kserve-74c5647cbf-zfzcs"
Nov 02 03:53:30.499463 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:53:30.499401    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9xzd\" (UniqueName: \"kubernetes.io/projected/9ad2c2f7-2198-4ec7-bd1c-bcf5ab3cc455-kube-api-access-f9xzd\") pod \"auth-test-model-3b27a5-kserve-74c5647cbf-zfzcs\" (UID: \"9ad2c2f7-2198-4ec7-bd1c-bcf5ab3cc455\") " pod="ai-tenant-e2e-shared-b-w1-3b27a5/auth-test-model-3b27a5-kserve-74c5647cbf-zfzcs"
Nov 02 03:53:30.549715 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:53:30.549690    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-3b27a5/auth-test-model-3b27a5-kserve-74c5647cbf-zfzcs"
Nov 02 03:53:30.932035 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:53:30.931993    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9ad2c2f7_2198_4ec7_bd1c_bcf5ab3cc455.slice/crio-02d9e9983315d4328dc371ad29c7d62487f0521acbc954673720689365fa0587 WatchSource:0}: Error finding container 02d9e9983315d4328dc371ad29c7d62487f0521acbc954673720689365fa0587: Status 404 returned error can't find the container with id 02d9e9983315d4328dc371ad29c7d62487f0521acbc954673720689365fa0587
Nov 02 03:53:30.932559 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:53:30.932455    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-3b27a5/auth-test-model-3b27a5-kserve-74c5647cbf-zfzcs"]
Nov 02 03:53:31.623392 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:53:31.623354    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-3b27a5/auth-test-model-3b27a5-kserve-74c5647cbf-zfzcs" event={"ID":"9ad2c2f7-2198-4ec7-bd1c-bcf5ab3cc455","Type":"ContainerStarted","Data":"28f8457572157facb0dcf23d145a9256e11ba8f6d63a0b3c3722bac6245ef7d4"}
Nov 02 03:53:31.623392 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:53:31.623398    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-3b27a5/auth-test-model-3b27a5-kserve-74c5647cbf-zfzcs" event={"ID":"9ad2c2f7-2198-4ec7-bd1c-bcf5ab3cc455","Type":"ContainerStarted","Data":"02d9e9983315d4328dc371ad29c7d62487f0521acbc954673720689365fa0587"}
Nov 02 03:53:31.739746 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:53:31.739694    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-3b27a5/auth-test-model-3b27a5-kserve-74c5647cbf-zfzcs" podStartSLOduration=1.739682002 podStartE2EDuration="1.739682002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:53:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:53:31.737620154 +0000 UTC m=+3952.316227535" watchObservedRunningTime="2026-11-02 03:53:31.739682002 +0000 UTC m=+3952.318289419"
Nov 02 03:53:32.550145 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:53:32.550111    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-3b27a5/auth-test-model-3b27a5-kserve-74c5647cbf-zfzcs"
Nov 02 03:53:33.529385 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:53:33.529344    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 03:53:40.549910 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:53:40.549873    4496 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-3b27a5/auth-test-model-3b27a5-kserve-74c5647cbf-zfzcs"
Nov 02 03:53:40.562738 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:53:40.562711    4496 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-3b27a5/auth-test-model-3b27a5-kserve-74c5647cbf-zfzcs"
Nov 02 03:53:40.703315 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:53:40.703283    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-3b27a5/auth-test-model-3b27a5-kserve-74c5647cbf-zfzcs"
Nov 02 03:54:03.527144 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:54:03.527108    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 03:54:33.527865 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:54:33.527787    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 03:55:03.527385 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:55:03.527352    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 03:55:27.456515 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:55:27.456482    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-86f4c4-654dfc988c-7cgf9"]
Nov 02 03:55:27.456992 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:55:27.456665    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-86f4c4-654dfc988c-7cgf9" podUID="ca4d18b6-ca1b-472d-a6ff-bfdca90eea1e" containerName="maas-api" containerID="cri-o://4a7887a3826c4b36973a8d6dd57eaa443b63f79a8366263a21b4fc3e983ce0b4" gracePeriod=30
Nov 02 03:55:27.736995 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:55:27.736964    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-86f4c4-654dfc988c-7cgf9"
Nov 02 03:55:27.865975 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:55:27.865935    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca4d18b6-ca1b-472d-a6ff-bfdca90eea1e-openshift-service-ca" pod "ca4d18b6-ca1b-472d-a6ff-bfdca90eea1e" (UID: "ca4d18b6-ca1b-472d-a6ff-bfdca90eea1e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 03:55:27.866229 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:55:27.865997    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca4d18b6-ca1b-472d-a6ff-bfdca90eea1e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca4d18b6-ca1b-472d-a6ff-bfdca90eea1e-openshift-service-ca\") pod \"ca4d18b6-ca1b-472d-a6ff-bfdca90eea1e\" (UID: \"ca4d18b6-ca1b-472d-a6ff-bfdca90eea1e\") "
Nov 02 03:55:27.866229 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:55:27.866109    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca4d18b6-ca1b-472d-a6ff-bfdca90eea1e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca4d18b6-ca1b-472d-a6ff-bfdca90eea1e-maas-api-tls\") pod \"ca4d18b6-ca1b-472d-a6ff-bfdca90eea1e\" (UID: \"ca4d18b6-ca1b-472d-a6ff-bfdca90eea1e\") "
Nov 02 03:55:27.866371 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:55:27.866264    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca4d18b6-ca1b-472d-a6ff-bfdca90eea1e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca4d18b6-ca1b-472d-a6ff-bfdca90eea1e-metrics-certs\") pod \"ca4d18b6-ca1b-472d-a6ff-bfdca90eea1e\" (UID: \"ca4d18b6-ca1b-472d-a6ff-bfdca90eea1e\") "
Nov 02 03:55:27.866435 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:55:27.866395    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca4d18b6-ca1b-472d-a6ff-bfdca90eea1e-kube-api-access-t5g6x\" (UniqueName: \"kubernetes.io/projected/ca4d18b6-ca1b-472d-a6ff-bfdca90eea1e-kube-api-access-t5g6x\") pod \"ca4d18b6-ca1b-472d-a6ff-bfdca90eea1e\" (UID: \"ca4d18b6-ca1b-472d-a6ff-bfdca90eea1e\") "
Nov 02 03:55:27.866980 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:55:27.866951    4496 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca4d18b6-ca1b-472d-a6ff-bfdca90eea1e-openshift-service-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:55:27.869973 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:55:27.869943    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca4d18b6-ca1b-472d-a6ff-bfdca90eea1e-kube-api-access-t5g6x" pod "ca4d18b6-ca1b-472d-a6ff-bfdca90eea1e" (UID: "ca4d18b6-ca1b-472d-a6ff-bfdca90eea1e"). InnerVolumeSpecName "kube-api-access-t5g6x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:55:27.870117 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:55:27.870072    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca4d18b6-ca1b-472d-a6ff-bfdca90eea1e-maas-api-tls" pod "ca4d18b6-ca1b-472d-a6ff-bfdca90eea1e" (UID: "ca4d18b6-ca1b-472d-a6ff-bfdca90eea1e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 03:55:27.870191 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:55:27.870154    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca4d18b6-ca1b-472d-a6ff-bfdca90eea1e-metrics-certs" pod "ca4d18b6-ca1b-472d-a6ff-bfdca90eea1e" (UID: "ca4d18b6-ca1b-472d-a6ff-bfdca90eea1e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 03:55:27.967824 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:55:27.967740    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5g6x\" (UniqueName: \"kubernetes.io/projected/ca4d18b6-ca1b-472d-a6ff-bfdca90eea1e-kube-api-access-t5g6x\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:55:27.967824 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:55:27.967774    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca4d18b6-ca1b-472d-a6ff-bfdca90eea1e-maas-api-tls\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:55:27.967824 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:55:27.967790    4496 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca4d18b6-ca1b-472d-a6ff-bfdca90eea1e-metrics-certs\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:55:28.531996 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:55:28.531964    4496 generic.go:356] "Generic (PLEG): container finished" podID="ca4d18b6-ca1b-472d-a6ff-bfdca90eea1e" containerID="4a7887a3826c4b36973a8d6dd57eaa443b63f79a8366263a21b4fc3e983ce0b4" exitCode=0
Nov 02 03:55:28.532437 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:55:28.532001    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-86f4c4-654dfc988c-7cgf9" event={"ID":"ca4d18b6-ca1b-472d-a6ff-bfdca90eea1e","Type":"ContainerDied","Data":"4a7887a3826c4b36973a8d6dd57eaa443b63f79a8366263a21b4fc3e983ce0b4"}
Nov 02 03:55:28.532437 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:55:28.532078    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-86f4c4-654dfc988c-7cgf9" event={"ID":"ca4d18b6-ca1b-472d-a6ff-bfdca90eea1e","Type":"ContainerDied","Data":"ad07617bb1c6e6245e239389560b7fe373ceb6b9ee13070c42678980009463d5"}
Nov 02 03:55:28.532437 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:55:28.532029    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-86f4c4-654dfc988c-7cgf9"
Nov 02 03:55:28.532437 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:55:28.532111    4496 scope.go:122] "RemoveContainer" containerID="4a7887a3826c4b36973a8d6dd57eaa443b63f79a8366263a21b4fc3e983ce0b4"
Nov 02 03:55:28.544417 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:55:28.544398    4496 scope.go:122] "RemoveContainer" containerID="4a7887a3826c4b36973a8d6dd57eaa443b63f79a8366263a21b4fc3e983ce0b4"
Nov 02 03:55:28.544741 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:55:28.544692    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a7887a3826c4b36973a8d6dd57eaa443b63f79a8366263a21b4fc3e983ce0b4\": container with ID starting with 4a7887a3826c4b36973a8d6dd57eaa443b63f79a8366263a21b4fc3e983ce0b4 not found: ID does not exist" containerID="4a7887a3826c4b36973a8d6dd57eaa443b63f79a8366263a21b4fc3e983ce0b4"
Nov 02 03:55:28.544814 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:55:28.544751    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a7887a3826c4b36973a8d6dd57eaa443b63f79a8366263a21b4fc3e983ce0b4"} err="failed to get container status \"4a7887a3826c4b36973a8d6dd57eaa443b63f79a8366263a21b4fc3e983ce0b4\": rpc error: code = NotFound desc = could not find container \"4a7887a3826c4b36973a8d6dd57eaa443b63f79a8366263a21b4fc3e983ce0b4\": container with ID starting with 4a7887a3826c4b36973a8d6dd57eaa443b63f79a8366263a21b4fc3e983ce0b4 not found: ID does not exist"
Nov 02 03:55:28.704908 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:55:28.704871    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-86f4c4-654dfc988c-7cgf9"]
Nov 02 03:55:28.709449 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:55:28.709419    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-86f4c4-654dfc988c-7cgf9"]
Nov 02 03:55:29.575404 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:55:29.575370    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca4d18b6-ca1b-472d-a6ff-bfdca90eea1e" path="/var/lib/kubelet/pods/ca4d18b6-ca1b-472d-a6ff-bfdca90eea1e/volumes"
Nov 02 03:55:33.530115 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:55:33.530078    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 03:56:03.529165 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:56:03.529125    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 03:56:33.528739 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:56:33.528692    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 03:56:58.929662 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:56:58.929567    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-ee0fed/auth-test-model-ee0fed-kserve-94748b6fc-4shsp"]
Nov 02 03:57:03.528993 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:57:03.528954    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 03:57:13.950753 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:57:13.950709    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-ee0fed/auth-test-model-ee0fed-kserve-94748b6fc-4shsp" podUID="46ffad11-c873-4046-ad9a-cc120fb70101" containerName="main" containerID="cri-o://2b65a2c3e112f9ca83470c1ff93c7ba12ba2a3ec18cb5d03a40268a5a8e34ca4" gracePeriod=45
Nov 02 03:57:14.214642 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:57:14.214584    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-ee0fed/auth-test-model-ee0fed-kserve-94748b6fc-4shsp"
Nov 02 03:57:14.272251 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:57:14.272221    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46ffad11-c873-4046-ad9a-cc120fb70101-dshm\" (UniqueName: \"kubernetes.io/empty-dir/46ffad11-c873-4046-ad9a-cc120fb70101-dshm\") pod \"46ffad11-c873-4046-ad9a-cc120fb70101\" (UID: \"46ffad11-c873-4046-ad9a-cc120fb70101\") "
Nov 02 03:57:14.272442 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:57:14.272314    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46ffad11-c873-4046-ad9a-cc120fb70101-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46ffad11-c873-4046-ad9a-cc120fb70101-model-cache\") pod \"46ffad11-c873-4046-ad9a-cc120fb70101\" (UID: \"46ffad11-c873-4046-ad9a-cc120fb70101\") "
Nov 02 03:57:14.272442 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:57:14.272365    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46ffad11-c873-4046-ad9a-cc120fb70101-tls-certs\" (UniqueName: \"kubernetes.io/secret/46ffad11-c873-4046-ad9a-cc120fb70101-tls-certs\") pod \"46ffad11-c873-4046-ad9a-cc120fb70101\" (UID: \"46ffad11-c873-4046-ad9a-cc120fb70101\") "
Nov 02 03:57:14.272442 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:57:14.272397    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46ffad11-c873-4046-ad9a-cc120fb70101-home\" (UniqueName: \"kubernetes.io/empty-dir/46ffad11-c873-4046-ad9a-cc120fb70101-home\") pod \"46ffad11-c873-4046-ad9a-cc120fb70101\" (UID: \"46ffad11-c873-4046-ad9a-cc120fb70101\") "
Nov 02 03:57:14.272623 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:57:14.272488    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46ffad11-c873-4046-ad9a-cc120fb70101-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46ffad11-c873-4046-ad9a-cc120fb70101-tmp-dir\") pod \"46ffad11-c873-4046-ad9a-cc120fb70101\" (UID: \"46ffad11-c873-4046-ad9a-cc120fb70101\") "
Nov 02 03:57:14.272623 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:57:14.272535    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46ffad11-c873-4046-ad9a-cc120fb70101-kube-api-access-5nrqx\" (UniqueName: \"kubernetes.io/projected/46ffad11-c873-4046-ad9a-cc120fb70101-kube-api-access-5nrqx\") pod \"46ffad11-c873-4046-ad9a-cc120fb70101\" (UID: \"46ffad11-c873-4046-ad9a-cc120fb70101\") "
Nov 02 03:57:14.272623 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:57:14.272601    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46ffad11-c873-4046-ad9a-cc120fb70101-model-cache" pod "46ffad11-c873-4046-ad9a-cc120fb70101" (UID: "46ffad11-c873-4046-ad9a-cc120fb70101"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:57:14.272784 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:57:14.272751    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46ffad11-c873-4046-ad9a-cc120fb70101-tmp-dir" pod "46ffad11-c873-4046-ad9a-cc120fb70101" (UID: "46ffad11-c873-4046-ad9a-cc120fb70101"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:57:14.272784 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:57:14.272767    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46ffad11-c873-4046-ad9a-cc120fb70101-home" pod "46ffad11-c873-4046-ad9a-cc120fb70101" (UID: "46ffad11-c873-4046-ad9a-cc120fb70101"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:57:14.273200 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:57:14.273172    4496 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46ffad11-c873-4046-ad9a-cc120fb70101-model-cache\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:57:14.273200 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:57:14.273202    4496 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46ffad11-c873-4046-ad9a-cc120fb70101-home\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:57:14.273396 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:57:14.273217    4496 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46ffad11-c873-4046-ad9a-cc120fb70101-tmp-dir\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:57:14.274829 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:57:14.274798    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46ffad11-c873-4046-ad9a-cc120fb70101-dshm" pod "46ffad11-c873-4046-ad9a-cc120fb70101" (UID: "46ffad11-c873-4046-ad9a-cc120fb70101"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:57:14.275115 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:57:14.275094    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46ffad11-c873-4046-ad9a-cc120fb70101-kube-api-access-5nrqx" pod "46ffad11-c873-4046-ad9a-cc120fb70101" (UID: "46ffad11-c873-4046-ad9a-cc120fb70101"). InnerVolumeSpecName "kube-api-access-5nrqx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:57:14.275188 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:57:14.275170    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46ffad11-c873-4046-ad9a-cc120fb70101-tls-certs" pod "46ffad11-c873-4046-ad9a-cc120fb70101" (UID: "46ffad11-c873-4046-ad9a-cc120fb70101"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 03:57:14.337750 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:57:14.337690    4496 generic.go:356] "Generic (PLEG): container finished" podID="46ffad11-c873-4046-ad9a-cc120fb70101" containerID="2b65a2c3e112f9ca83470c1ff93c7ba12ba2a3ec18cb5d03a40268a5a8e34ca4" exitCode=0
Nov 02 03:57:14.337889 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:57:14.337853    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-ee0fed/auth-test-model-ee0fed-kserve-94748b6fc-4shsp" event={"ID":"46ffad11-c873-4046-ad9a-cc120fb70101","Type":"ContainerDied","Data":"2b65a2c3e112f9ca83470c1ff93c7ba12ba2a3ec18cb5d03a40268a5a8e34ca4"}
Nov 02 03:57:14.337945 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:57:14.337904    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-ee0fed/auth-test-model-ee0fed-kserve-94748b6fc-4shsp" event={"ID":"46ffad11-c873-4046-ad9a-cc120fb70101","Type":"ContainerDied","Data":"f74440819013842e170fefa5159b6d8779d3ca5290821481585ba44735a38234"}
Nov 02 03:57:14.337945 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:57:14.337928    4496 scope.go:122] "RemoveContainer" containerID="2b65a2c3e112f9ca83470c1ff93c7ba12ba2a3ec18cb5d03a40268a5a8e34ca4"
Nov 02 03:57:14.338170 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:57:14.338150    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-ee0fed/auth-test-model-ee0fed-kserve-94748b6fc-4shsp"
Nov 02 03:57:14.363818 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:57:14.363790    4496 scope.go:122] "RemoveContainer" containerID="2b65a2c3e112f9ca83470c1ff93c7ba12ba2a3ec18cb5d03a40268a5a8e34ca4"
Nov 02 03:57:14.365414 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:57:14.365381    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b65a2c3e112f9ca83470c1ff93c7ba12ba2a3ec18cb5d03a40268a5a8e34ca4\": container with ID starting with 2b65a2c3e112f9ca83470c1ff93c7ba12ba2a3ec18cb5d03a40268a5a8e34ca4 not found: ID does not exist" containerID="2b65a2c3e112f9ca83470c1ff93c7ba12ba2a3ec18cb5d03a40268a5a8e34ca4"
Nov 02 03:57:14.365607 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:57:14.365576    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b65a2c3e112f9ca83470c1ff93c7ba12ba2a3ec18cb5d03a40268a5a8e34ca4"} err="failed to get container status \"2b65a2c3e112f9ca83470c1ff93c7ba12ba2a3ec18cb5d03a40268a5a8e34ca4\": rpc error: code = NotFound desc = could not find container \"2b65a2c3e112f9ca83470c1ff93c7ba12ba2a3ec18cb5d03a40268a5a8e34ca4\": container with ID starting with 2b65a2c3e112f9ca83470c1ff93c7ba12ba2a3ec18cb5d03a40268a5a8e34ca4 not found: ID does not exist"
Nov 02 03:57:14.374182 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:57:14.374156    4496 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46ffad11-c873-4046-ad9a-cc120fb70101-dshm\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:57:14.374309 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:57:14.374196    4496 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46ffad11-c873-4046-ad9a-cc120fb70101-tls-certs\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:57:14.374309 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:57:14.374206    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nrqx\" (UniqueName: \"kubernetes.io/projected/46ffad11-c873-4046-ad9a-cc120fb70101-kube-api-access-5nrqx\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:57:14.508802 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:57:14.508771    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-ee0fed/auth-test-model-ee0fed-kserve-94748b6fc-4shsp"]
Nov 02 03:57:14.512004 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:57:14.511979    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-ee0fed/auth-test-model-ee0fed-kserve-94748b6fc-4shsp"]
Nov 02 03:57:15.576178 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:57:15.576140    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46ffad11-c873-4046-ad9a-cc120fb70101" path="/var/lib/kubelet/pods/46ffad11-c873-4046-ad9a-cc120fb70101/volumes"
Nov 02 03:57:33.530235 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:57:33.530197    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 03:57:40.436895 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:57:40.436865    4496 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hww8q_5c797b08-a314-44ed-8026-8a0ce1f6c36f/ovn-acl-logging/0.log"
Nov 02 03:57:40.450757 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:57:40.450735    4496 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hww8q_5c797b08-a314-44ed-8026-8a0ce1f6c36f/ovn-acl-logging/0.log"
Nov 02 03:57:41.729203 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:57:41.729167    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-147104/auth-test-model-147104-kserve-74f4568577-85cnx"]
Nov 02 03:57:50.533992 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:57:50.533956    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11262-299ddb"]
Nov 02 03:57:50.534791 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:57:50.534772    4496 cpu_manager.go:414] "containerMap: removing container" podUID="46ffad11-c873-4046-ad9a-cc120fb70101" containerName="main"
Nov 02 03:57:50.534851 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:57:50.534794    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46ffad11-c873-4046-ad9a-cc120fb70101" containerName="main"
Nov 02 03:57:50.534851 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:57:50.534814    4496 cpu_manager.go:414] "containerMap: removing container" podUID="ca4d18b6-ca1b-472d-a6ff-bfdca90eea1e" containerName="maas-api"
Nov 02 03:57:50.534851 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:57:50.534819    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca4d18b6-ca1b-472d-a6ff-bfdca90eea1e" containerName="maas-api"
Nov 02 03:57:50.534953 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:57:50.534933    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="46ffad11-c873-4046-ad9a-cc120fb70101" containerName="main"
Nov 02 03:57:50.534953 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:57:50.534943    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca4d18b6-ca1b-472d-a6ff-bfdca90eea1e" containerName="maas-api"
Nov 02 03:57:50.538533 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:57:50.538512    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11262-299ddb"
Nov 02 03:57:50.593812 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:57:50.593775    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11262-299ddb"]
Nov 02 03:57:50.640357 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:57:50.640333    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-t4r8t\""
Nov 02 03:57:50.643562 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:57:50.643535    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcw9p\" (UniqueName: \"kubernetes.io/projected/7f8d3509-ea3b-40a3-b828-f4a966032c37-kube-api-access-dcw9p\") pod \"test-curl-11262-299ddb\" (UID: \"7f8d3509-ea3b-40a3-b828-f4a966032c37\") " pod="opendatahub/test-curl-11262-299ddb"
Nov 02 03:57:50.745101 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:57:50.745067    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcw9p\" (UniqueName: \"kubernetes.io/projected/7f8d3509-ea3b-40a3-b828-f4a966032c37-kube-api-access-dcw9p\") pod \"test-curl-11262-299ddb\" (UID: \"7f8d3509-ea3b-40a3-b828-f4a966032c37\") " pod="opendatahub/test-curl-11262-299ddb"
Nov 02 03:57:50.805209 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:57:50.805134    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcw9p\" (UniqueName: \"kubernetes.io/projected/7f8d3509-ea3b-40a3-b828-f4a966032c37-kube-api-access-dcw9p\") pod \"test-curl-11262-299ddb\" (UID: \"7f8d3509-ea3b-40a3-b828-f4a966032c37\") " pod="opendatahub/test-curl-11262-299ddb"
Nov 02 03:57:50.854455 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:57:50.854426    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11262-299ddb"
Nov 02 03:57:51.228553 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:57:51.228522    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11262-299ddb"]
Nov 02 03:57:51.228929 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:57:51.228906    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7f8d3509_ea3b_40a3_b828_f4a966032c37.slice/crio-59c963b67be017dbe06398f2f0cf0145f23fcc0f1af0043af7c9ad60fd7d5f46 WatchSource:0}: Error finding container 59c963b67be017dbe06398f2f0cf0145f23fcc0f1af0043af7c9ad60fd7d5f46: Status 404 returned error can't find the container with id 59c963b67be017dbe06398f2f0cf0145f23fcc0f1af0043af7c9ad60fd7d5f46
Nov 02 03:57:51.665413 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:57:51.665305    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11262-299ddb" event={"ID":"7f8d3509-ea3b-40a3-b828-f4a966032c37","Type":"ContainerStarted","Data":"7e897283ed2c249f741c68c7277ad42584412621e0f4d35a7b0cbef2df5704f7"}
Nov 02 03:57:51.665413 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:57:51.665358    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11262-299ddb" event={"ID":"7f8d3509-ea3b-40a3-b828-f4a966032c37","Type":"ContainerStarted","Data":"59c963b67be017dbe06398f2f0cf0145f23fcc0f1af0043af7c9ad60fd7d5f46"}
Nov 02 03:57:51.780358 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:57:51.780306    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11262-299ddb" podStartSLOduration=1.780292991 podStartE2EDuration="1.780292991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:57:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:57:51.777614685 +0000 UTC m=+4212.356222066" watchObservedRunningTime="2026-11-02 03:57:51.780292991 +0000 UTC m=+4212.358900371"
Nov 02 03:57:53.227140 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:57:53.227106    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11262-299ddb"]
Nov 02 03:57:53.233309 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:57:53.233280    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11262-299ddb"]
Nov 02 03:57:53.681999 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:57:53.681893    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11262-299ddb" podUID="7f8d3509-ea3b-40a3-b828-f4a966032c37" containerName="test-curl-11262-299ddb" containerID="cri-o://7e897283ed2c249f741c68c7277ad42584412621e0f4d35a7b0cbef2df5704f7" gracePeriod=2
Nov 02 03:57:55.930107 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:57:55.930079    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11262-299ddb"
Nov 02 03:57:56.003145 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:57:56.003120    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f8d3509-ea3b-40a3-b828-f4a966032c37-kube-api-access-dcw9p\" (UniqueName: \"kubernetes.io/projected/7f8d3509-ea3b-40a3-b828-f4a966032c37-kube-api-access-dcw9p\") pod \"7f8d3509-ea3b-40a3-b828-f4a966032c37\" (UID: \"7f8d3509-ea3b-40a3-b828-f4a966032c37\") "
Nov 02 03:57:56.005152 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:57:56.005124    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f8d3509-ea3b-40a3-b828-f4a966032c37-kube-api-access-dcw9p" pod "7f8d3509-ea3b-40a3-b828-f4a966032c37" (UID: "7f8d3509-ea3b-40a3-b828-f4a966032c37"). InnerVolumeSpecName "kube-api-access-dcw9p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:57:56.105073 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:57:56.105041    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcw9p\" (UniqueName: \"kubernetes.io/projected/7f8d3509-ea3b-40a3-b828-f4a966032c37-kube-api-access-dcw9p\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:57:56.711194 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:57:56.711161    4496 generic.go:356] "Generic (PLEG): container finished" podID="7f8d3509-ea3b-40a3-b828-f4a966032c37" containerID="7e897283ed2c249f741c68c7277ad42584412621e0f4d35a7b0cbef2df5704f7" exitCode=137
Nov 02 03:57:56.711378 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:57:56.711203    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11262-299ddb"
Nov 02 03:57:56.711378 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:57:56.711259    4496 scope.go:122] "RemoveContainer" containerID="7e897283ed2c249f741c68c7277ad42584412621e0f4d35a7b0cbef2df5704f7"
Nov 02 03:57:56.722106 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:57:56.722086    4496 scope.go:122] "RemoveContainer" containerID="7e897283ed2c249f741c68c7277ad42584412621e0f4d35a7b0cbef2df5704f7"
Nov 02 03:57:56.722358 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:57:56.722338    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e897283ed2c249f741c68c7277ad42584412621e0f4d35a7b0cbef2df5704f7\": container with ID starting with 7e897283ed2c249f741c68c7277ad42584412621e0f4d35a7b0cbef2df5704f7 not found: ID does not exist" containerID="7e897283ed2c249f741c68c7277ad42584412621e0f4d35a7b0cbef2df5704f7"
Nov 02 03:57:56.722413 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:57:56.722366    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e897283ed2c249f741c68c7277ad42584412621e0f4d35a7b0cbef2df5704f7"} err="failed to get container status \"7e897283ed2c249f741c68c7277ad42584412621e0f4d35a7b0cbef2df5704f7\": rpc error: code = NotFound desc = could not find container \"7e897283ed2c249f741c68c7277ad42584412621e0f4d35a7b0cbef2df5704f7\": container with ID starting with 7e897283ed2c249f741c68c7277ad42584412621e0f4d35a7b0cbef2df5704f7 not found: ID does not exist"
Nov 02 03:57:56.752269 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:57:56.752225    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-147104/auth-test-model-147104-kserve-74f4568577-85cnx" podUID="4d10c8e8-9717-4cdc-a7df-229ab28ae02a" containerName="main" containerID="cri-o://33a2a5e1488ba81c27ea28fa1b801d59f7f619bc85969005bbc4bfd2b8dc501e" gracePeriod=45
Nov 02 03:57:56.772144 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:57:56.771988    4496 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11262-299ddb\" is forbidden: User \"system:node:ip-10-0-133-163.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-163.us-west-1.compute.internal' and this object" podUID="7f8d3509-ea3b-40a3-b828-f4a966032c37" pod="opendatahub/test-curl-11262-299ddb"
Nov 02 03:57:57.038208 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:57:57.038181    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-147104/auth-test-model-147104-kserve-74f4568577-85cnx"
Nov 02 03:57:57.090258 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:57:57.090217    4496 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11262-299ddb\" is forbidden: User \"system:node:ip-10-0-133-163.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-163.us-west-1.compute.internal' and this object" podUID="7f8d3509-ea3b-40a3-b828-f4a966032c37" pod="opendatahub/test-curl-11262-299ddb"
Nov 02 03:57:57.114175 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:57:57.114145    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d10c8e8-9717-4cdc-a7df-229ab28ae02a-home\" (UniqueName: \"kubernetes.io/empty-dir/4d10c8e8-9717-4cdc-a7df-229ab28ae02a-home\") pod \"4d10c8e8-9717-4cdc-a7df-229ab28ae02a\" (UID: \"4d10c8e8-9717-4cdc-a7df-229ab28ae02a\") "
Nov 02 03:57:57.114324 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:57:57.114232    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d10c8e8-9717-4cdc-a7df-229ab28ae02a-kube-api-access-c4zb2\" (UniqueName: \"kubernetes.io/projected/4d10c8e8-9717-4cdc-a7df-229ab28ae02a-kube-api-access-c4zb2\") pod \"4d10c8e8-9717-4cdc-a7df-229ab28ae02a\" (UID: \"4d10c8e8-9717-4cdc-a7df-229ab28ae02a\") "
Nov 02 03:57:57.114372 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:57:57.114326    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d10c8e8-9717-4cdc-a7df-229ab28ae02a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d10c8e8-9717-4cdc-a7df-229ab28ae02a-model-cache\") pod \"4d10c8e8-9717-4cdc-a7df-229ab28ae02a\" (UID: \"4d10c8e8-9717-4cdc-a7df-229ab28ae02a\") "
Nov 02 03:57:57.114420 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:57:57.114392    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d10c8e8-9717-4cdc-a7df-229ab28ae02a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d10c8e8-9717-4cdc-a7df-229ab28ae02a-dshm\") pod \"4d10c8e8-9717-4cdc-a7df-229ab28ae02a\" (UID: \"4d10c8e8-9717-4cdc-a7df-229ab28ae02a\") "
Nov 02 03:57:57.114462 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:57:57.114430    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d10c8e8-9717-4cdc-a7df-229ab28ae02a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d10c8e8-9717-4cdc-a7df-229ab28ae02a-tmp-dir\") pod \"4d10c8e8-9717-4cdc-a7df-229ab28ae02a\" (UID: \"4d10c8e8-9717-4cdc-a7df-229ab28ae02a\") "
Nov 02 03:57:57.114462 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:57:57.114452    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d10c8e8-9717-4cdc-a7df-229ab28ae02a-home" pod "4d10c8e8-9717-4cdc-a7df-229ab28ae02a" (UID: "4d10c8e8-9717-4cdc-a7df-229ab28ae02a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:57:57.114556 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:57:57.114500    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d10c8e8-9717-4cdc-a7df-229ab28ae02a-tls-certs\" (UniqueName: \"kubernetes.io/secret/4d10c8e8-9717-4cdc-a7df-229ab28ae02a-tls-certs\") pod \"4d10c8e8-9717-4cdc-a7df-229ab28ae02a\" (UID: \"4d10c8e8-9717-4cdc-a7df-229ab28ae02a\") "
Nov 02 03:57:57.114631 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:57:57.114602    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d10c8e8-9717-4cdc-a7df-229ab28ae02a-model-cache" pod "4d10c8e8-9717-4cdc-a7df-229ab28ae02a" (UID: "4d10c8e8-9717-4cdc-a7df-229ab28ae02a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:57:57.114792 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:57:57.114768    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d10c8e8-9717-4cdc-a7df-229ab28ae02a-tmp-dir" pod "4d10c8e8-9717-4cdc-a7df-229ab28ae02a" (UID: "4d10c8e8-9717-4cdc-a7df-229ab28ae02a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:57:57.115032 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:57:57.114998    4496 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d10c8e8-9717-4cdc-a7df-229ab28ae02a-tmp-dir\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:57:57.115111 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:57:57.115053    4496 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d10c8e8-9717-4cdc-a7df-229ab28ae02a-home\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:57:57.115111 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:57:57.115069    4496 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d10c8e8-9717-4cdc-a7df-229ab28ae02a-model-cache\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:57:57.116557 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:57:57.116531    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d10c8e8-9717-4cdc-a7df-229ab28ae02a-tls-certs" pod "4d10c8e8-9717-4cdc-a7df-229ab28ae02a" (UID: "4d10c8e8-9717-4cdc-a7df-229ab28ae02a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 03:57:57.116647 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:57:57.116567    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d10c8e8-9717-4cdc-a7df-229ab28ae02a-dshm" pod "4d10c8e8-9717-4cdc-a7df-229ab28ae02a" (UID: "4d10c8e8-9717-4cdc-a7df-229ab28ae02a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:57:57.116647 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:57:57.116584    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d10c8e8-9717-4cdc-a7df-229ab28ae02a-kube-api-access-c4zb2" pod "4d10c8e8-9717-4cdc-a7df-229ab28ae02a" (UID: "4d10c8e8-9717-4cdc-a7df-229ab28ae02a"). InnerVolumeSpecName "kube-api-access-c4zb2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:57:57.216056 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:57:57.216025    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4zb2\" (UniqueName: \"kubernetes.io/projected/4d10c8e8-9717-4cdc-a7df-229ab28ae02a-kube-api-access-c4zb2\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:57:57.216056 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:57:57.216060    4496 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d10c8e8-9717-4cdc-a7df-229ab28ae02a-dshm\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:57:57.216274 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:57:57.216074    4496 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d10c8e8-9717-4cdc-a7df-229ab28ae02a-tls-certs\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:57:57.577952 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:57:57.577907    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f8d3509-ea3b-40a3-b828-f4a966032c37" path="/var/lib/kubelet/pods/7f8d3509-ea3b-40a3-b828-f4a966032c37/volumes"
Nov 02 03:57:57.720424 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:57:57.720387    4496 generic.go:356] "Generic (PLEG): container finished" podID="4d10c8e8-9717-4cdc-a7df-229ab28ae02a" containerID="33a2a5e1488ba81c27ea28fa1b801d59f7f619bc85969005bbc4bfd2b8dc501e" exitCode=0
Nov 02 03:57:57.720621 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:57:57.720431    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-147104/auth-test-model-147104-kserve-74f4568577-85cnx"
Nov 02 03:57:57.720621 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:57:57.720466    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-147104/auth-test-model-147104-kserve-74f4568577-85cnx" event={"ID":"4d10c8e8-9717-4cdc-a7df-229ab28ae02a","Type":"ContainerDied","Data":"33a2a5e1488ba81c27ea28fa1b801d59f7f619bc85969005bbc4bfd2b8dc501e"}
Nov 02 03:57:57.720621 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:57:57.720508    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-147104/auth-test-model-147104-kserve-74f4568577-85cnx" event={"ID":"4d10c8e8-9717-4cdc-a7df-229ab28ae02a","Type":"ContainerDied","Data":"e56c6a7ffb19baee64cca515cda8508f6d76a402737be96e5c8370c5ff724d3d"}
Nov 02 03:57:57.720621 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:57:57.720525    4496 scope.go:122] "RemoveContainer" containerID="33a2a5e1488ba81c27ea28fa1b801d59f7f619bc85969005bbc4bfd2b8dc501e"
Nov 02 03:57:57.730624 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:57:57.730608    4496 scope.go:122] "RemoveContainer" containerID="33a2a5e1488ba81c27ea28fa1b801d59f7f619bc85969005bbc4bfd2b8dc501e"
Nov 02 03:57:57.730908 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:57:57.730883    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33a2a5e1488ba81c27ea28fa1b801d59f7f619bc85969005bbc4bfd2b8dc501e\": container with ID starting with 33a2a5e1488ba81c27ea28fa1b801d59f7f619bc85969005bbc4bfd2b8dc501e not found: ID does not exist" containerID="33a2a5e1488ba81c27ea28fa1b801d59f7f619bc85969005bbc4bfd2b8dc501e"
Nov 02 03:57:57.730971 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:57:57.730917    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33a2a5e1488ba81c27ea28fa1b801d59f7f619bc85969005bbc4bfd2b8dc501e"} err="failed to get container status \"33a2a5e1488ba81c27ea28fa1b801d59f7f619bc85969005bbc4bfd2b8dc501e\": rpc error: code = NotFound desc = could not find container \"33a2a5e1488ba81c27ea28fa1b801d59f7f619bc85969005bbc4bfd2b8dc501e\": container with ID starting with 33a2a5e1488ba81c27ea28fa1b801d59f7f619bc85969005bbc4bfd2b8dc501e not found: ID does not exist"
Nov 02 03:57:57.887595 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:57:57.887519    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-147104/auth-test-model-147104-kserve-74f4568577-85cnx"]
Nov 02 03:57:57.939988 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:57:57.939948    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-147104/auth-test-model-147104-kserve-74f4568577-85cnx"]
Nov 02 03:57:59.575566 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:57:59.575529    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d10c8e8-9717-4cdc-a7df-229ab28ae02a" path="/var/lib/kubelet/pods/4d10c8e8-9717-4cdc-a7df-229ab28ae02a/volumes"
Nov 02 03:58:03.533985 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:58:03.533949    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 03:58:33.175815 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:58:33.175777    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-3b27a5/auth-test-model-3b27a5-kserve-74c5647cbf-zfzcs"]
Nov 02 03:58:33.263213 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:58:33.263171    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11256-c2b7ec"]
Nov 02 03:58:33.263870 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:58:33.263854    4496 cpu_manager.go:414] "containerMap: removing container" podUID="4d10c8e8-9717-4cdc-a7df-229ab28ae02a" containerName="main"
Nov 02 03:58:33.263915 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:58:33.263871    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d10c8e8-9717-4cdc-a7df-229ab28ae02a" containerName="main"
Nov 02 03:58:33.263915 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:58:33.263905    4496 cpu_manager.go:414] "containerMap: removing container" podUID="7f8d3509-ea3b-40a3-b828-f4a966032c37" containerName="test-curl-11262-299ddb"
Nov 02 03:58:33.263915 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:58:33.263911    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f8d3509-ea3b-40a3-b828-f4a966032c37" containerName="test-curl-11262-299ddb"
Nov 02 03:58:33.264053 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:58:33.264042    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f8d3509-ea3b-40a3-b828-f4a966032c37" containerName="test-curl-11262-299ddb"
Nov 02 03:58:33.264107 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:58:33.264058    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d10c8e8-9717-4cdc-a7df-229ab28ae02a" containerName="main"
Nov 02 03:58:33.267354 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:58:33.267335    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11256-c2b7ec"
Nov 02 03:58:33.320302 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:58:33.320263    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-t4r8t\""
Nov 02 03:58:33.321928 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:58:33.321896    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11256-c2b7ec"]
Nov 02 03:58:33.348178 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:58:33.348140    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psqfb\" (UniqueName: \"kubernetes.io/projected/f9f1febb-170d-4196-a0e2-260e383ef9a1-kube-api-access-psqfb\") pod \"test-curl-11256-c2b7ec\" (UID: \"f9f1febb-170d-4196-a0e2-260e383ef9a1\") " pod="opendatahub/test-curl-11256-c2b7ec"
Nov 02 03:58:33.449578 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:58:33.449490    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psqfb\" (UniqueName: \"kubernetes.io/projected/f9f1febb-170d-4196-a0e2-260e383ef9a1-kube-api-access-psqfb\") pod \"test-curl-11256-c2b7ec\" (UID: \"f9f1febb-170d-4196-a0e2-260e383ef9a1\") " pod="opendatahub/test-curl-11256-c2b7ec"
Nov 02 03:58:33.506735 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:58:33.506702    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-psqfb\" (UniqueName: \"kubernetes.io/projected/f9f1febb-170d-4196-a0e2-260e383ef9a1-kube-api-access-psqfb\") pod \"test-curl-11256-c2b7ec\" (UID: \"f9f1febb-170d-4196-a0e2-260e383ef9a1\") " pod="opendatahub/test-curl-11256-c2b7ec"
Nov 02 03:58:33.528389 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:58:33.528355    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 03:58:33.580671 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:58:33.580641    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11256-c2b7ec"
Nov 02 03:58:33.963716 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:58:33.963688    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf9f1febb_170d_4196_a0e2_260e383ef9a1.slice/crio-6d7632a9e4de7be79951bacb920c16a11699becdd5664ac6b47d73be7d6d71b1 WatchSource:0}: Error finding container 6d7632a9e4de7be79951bacb920c16a11699becdd5664ac6b47d73be7d6d71b1: Status 404 returned error can't find the container with id 6d7632a9e4de7be79951bacb920c16a11699becdd5664ac6b47d73be7d6d71b1
Nov 02 03:58:33.963923 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:58:33.963899    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11256-c2b7ec"]
Nov 02 03:58:34.004812 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:58:34.004781    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11256-c2b7ec" event={"ID":"f9f1febb-170d-4196-a0e2-260e383ef9a1","Type":"ContainerStarted","Data":"6d7632a9e4de7be79951bacb920c16a11699becdd5664ac6b47d73be7d6d71b1"}
Nov 02 03:58:35.013551 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:58:35.013510    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11256-c2b7ec" event={"ID":"f9f1febb-170d-4196-a0e2-260e383ef9a1","Type":"ContainerStarted","Data":"ea128ade8c88d451ab14217099ea448f744e3a8c632272d82e0b2614c3868ae3"}
Nov 02 03:58:35.128307 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:58:35.128257    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11256-c2b7ec" podStartSLOduration=2.128243884 podStartE2EDuration="2.128243884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:58:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:58:35.127298405 +0000 UTC m=+4255.705905785" watchObservedRunningTime="2026-11-02 03:58:35.128243884 +0000 UTC m=+4255.706851265"
Nov 02 03:58:36.504378 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:58:36.504342    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11256-c2b7ec"]
Nov 02 03:58:36.508931 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:58:36.508900    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11256-c2b7ec"]
Nov 02 03:58:37.027395 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:58:37.027352    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11256-c2b7ec" podUID="f9f1febb-170d-4196-a0e2-260e383ef9a1" containerName="test-curl-11256-c2b7ec" containerID="cri-o://ea128ade8c88d451ab14217099ea448f744e3a8c632272d82e0b2614c3868ae3" gracePeriod=2
Nov 02 03:58:37.079409 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:58:37.079374    4496 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11256-c2b7ec\" is forbidden: User \"system:node:ip-10-0-133-163.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-163.us-west-1.compute.internal' and this object" podUID="f9f1febb-170d-4196-a0e2-260e383ef9a1" pod="opendatahub/test-curl-11256-c2b7ec"
Nov 02 03:58:39.240356 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:58:39.240331    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11256-c2b7ec"
Nov 02 03:58:39.323879 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:58:39.323799    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9f1febb-170d-4196-a0e2-260e383ef9a1-kube-api-access-psqfb\" (UniqueName: \"kubernetes.io/projected/f9f1febb-170d-4196-a0e2-260e383ef9a1-kube-api-access-psqfb\") pod \"f9f1febb-170d-4196-a0e2-260e383ef9a1\" (UID: \"f9f1febb-170d-4196-a0e2-260e383ef9a1\") "
Nov 02 03:58:39.325869 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:58:39.325845    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9f1febb-170d-4196-a0e2-260e383ef9a1-kube-api-access-psqfb" pod "f9f1febb-170d-4196-a0e2-260e383ef9a1" (UID: "f9f1febb-170d-4196-a0e2-260e383ef9a1"). InnerVolumeSpecName "kube-api-access-psqfb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:58:39.425640 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:58:39.425608    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psqfb\" (UniqueName: \"kubernetes.io/projected/f9f1febb-170d-4196-a0e2-260e383ef9a1-kube-api-access-psqfb\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:58:39.575217 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:58:39.575145    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9f1febb-170d-4196-a0e2-260e383ef9a1" path="/var/lib/kubelet/pods/f9f1febb-170d-4196-a0e2-260e383ef9a1/volumes"
Nov 02 03:58:40.051066 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:58:40.051004    4496 generic.go:356] "Generic (PLEG): container finished" podID="f9f1febb-170d-4196-a0e2-260e383ef9a1" containerID="ea128ade8c88d451ab14217099ea448f744e3a8c632272d82e0b2614c3868ae3" exitCode=137
Nov 02 03:58:40.051269 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:58:40.051073    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11256-c2b7ec"
Nov 02 03:58:40.051269 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:58:40.051084    4496 scope.go:122] "RemoveContainer" containerID="ea128ade8c88d451ab14217099ea448f744e3a8c632272d82e0b2614c3868ae3"
Nov 02 03:58:40.061215 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:58:40.061191    4496 scope.go:122] "RemoveContainer" containerID="ea128ade8c88d451ab14217099ea448f744e3a8c632272d82e0b2614c3868ae3"
Nov 02 03:58:40.061487 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:58:40.061465    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea128ade8c88d451ab14217099ea448f744e3a8c632272d82e0b2614c3868ae3\": container with ID starting with ea128ade8c88d451ab14217099ea448f744e3a8c632272d82e0b2614c3868ae3 not found: ID does not exist" containerID="ea128ade8c88d451ab14217099ea448f744e3a8c632272d82e0b2614c3868ae3"
Nov 02 03:58:40.061554 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:58:40.061494    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea128ade8c88d451ab14217099ea448f744e3a8c632272d82e0b2614c3868ae3"} err="failed to get container status \"ea128ade8c88d451ab14217099ea448f744e3a8c632272d82e0b2614c3868ae3\": rpc error: code = NotFound desc = could not find container \"ea128ade8c88d451ab14217099ea448f744e3a8c632272d82e0b2614c3868ae3\": container with ID starting with ea128ade8c88d451ab14217099ea448f744e3a8c632272d82e0b2614c3868ae3 not found: ID does not exist"
Nov 02 03:58:47.688623 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:58:47.688589    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-61c297-7bf69564c7-8p7fp"]
Nov 02 03:58:47.689275 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:58:47.689260    4496 cpu_manager.go:414] "containerMap: removing container" podUID="f9f1febb-170d-4196-a0e2-260e383ef9a1" containerName="test-curl-11256-c2b7ec"
Nov 02 03:58:47.689335 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:58:47.689276    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9f1febb-170d-4196-a0e2-260e383ef9a1" containerName="test-curl-11256-c2b7ec"
Nov 02 03:58:47.689403 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:58:47.689393    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9f1febb-170d-4196-a0e2-260e383ef9a1" containerName="test-curl-11256-c2b7ec"
Nov 02 03:58:47.694597 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:58:47.694578    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-61c297-7bf69564c7-8p7fp"
Nov 02 03:58:47.729378 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:58:47.729344    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1249045d-1bca-4013-a9f7-f3c139923ded-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-61c297-7bf69564c7-8p7fp\" (UID: \"1249045d-1bca-4013-a9f7-f3c139923ded\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-61c297-7bf69564c7-8p7fp"
Nov 02 03:58:47.729527 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:58:47.729424    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n45rv\" (UniqueName: \"kubernetes.io/projected/1249045d-1bca-4013-a9f7-f3c139923ded-kube-api-access-n45rv\") pod \"maas-api-e2e-extmod-a-61c297-7bf69564c7-8p7fp\" (UID: \"1249045d-1bca-4013-a9f7-f3c139923ded\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-61c297-7bf69564c7-8p7fp"
Nov 02 03:58:47.729585 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:58:47.729557    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1249045d-1bca-4013-a9f7-f3c139923ded-metrics-certs\") pod \"maas-api-e2e-extmod-a-61c297-7bf69564c7-8p7fp\" (UID: \"1249045d-1bca-4013-a9f7-f3c139923ded\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-61c297-7bf69564c7-8p7fp"
Nov 02 03:58:47.729627 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:58:47.729592    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1249045d-1bca-4013-a9f7-f3c139923ded-maas-api-tls\") pod \"maas-api-e2e-extmod-a-61c297-7bf69564c7-8p7fp\" (UID: \"1249045d-1bca-4013-a9f7-f3c139923ded\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-61c297-7bf69564c7-8p7fp"
Nov 02 03:58:47.748147 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:58:47.748116    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-61c297-serving-cert\""
Nov 02 03:58:47.750269 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:58:47.750242    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-61c297-7bf69564c7-8p7fp"]
Nov 02 03:58:47.830561 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:58:47.830529    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1249045d-1bca-4013-a9f7-f3c139923ded-metrics-certs\") pod \"maas-api-e2e-extmod-a-61c297-7bf69564c7-8p7fp\" (UID: \"1249045d-1bca-4013-a9f7-f3c139923ded\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-61c297-7bf69564c7-8p7fp"
Nov 02 03:58:47.830756 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:58:47.830591    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1249045d-1bca-4013-a9f7-f3c139923ded-maas-api-tls\") pod \"maas-api-e2e-extmod-a-61c297-7bf69564c7-8p7fp\" (UID: \"1249045d-1bca-4013-a9f7-f3c139923ded\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-61c297-7bf69564c7-8p7fp"
Nov 02 03:58:47.830756 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:58:47.830680    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1249045d-1bca-4013-a9f7-f3c139923ded-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-61c297-7bf69564c7-8p7fp\" (UID: \"1249045d-1bca-4013-a9f7-f3c139923ded\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-61c297-7bf69564c7-8p7fp"
Nov 02 03:58:47.830756 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:58:47.830713    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n45rv\" (UniqueName: \"kubernetes.io/projected/1249045d-1bca-4013-a9f7-f3c139923ded-kube-api-access-n45rv\") pod \"maas-api-e2e-extmod-a-61c297-7bf69564c7-8p7fp\" (UID: \"1249045d-1bca-4013-a9f7-f3c139923ded\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-61c297-7bf69564c7-8p7fp"
Nov 02 03:58:47.830941 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:58:47.830758    4496 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-61c297-serving-cert: secret "maas-api-e2e-extmod-a-61c297-serving-cert" not found
Nov 02 03:58:47.830941 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:58:47.830845    4496 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1249045d-1bca-4013-a9f7-f3c139923ded-maas-api-tls podName:1249045d-1bca-4013-a9f7-f3c139923ded nodeName:}" failed. No retries permitted until 2026-11-02 03:58:48.330823103 +0000 UTC m=+4268.909430480 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1249045d-1bca-4013-a9f7-f3c139923ded-maas-api-tls") pod "maas-api-e2e-extmod-a-61c297-7bf69564c7-8p7fp" (UID: "1249045d-1bca-4013-a9f7-f3c139923ded") : secret "maas-api-e2e-extmod-a-61c297-serving-cert" not found
Nov 02 03:58:47.831722 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:58:47.831698    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1249045d-1bca-4013-a9f7-f3c139923ded-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-61c297-7bf69564c7-8p7fp\" (UID: \"1249045d-1bca-4013-a9f7-f3c139923ded\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-61c297-7bf69564c7-8p7fp"
Nov 02 03:58:47.834411 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:58:47.834385    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1249045d-1bca-4013-a9f7-f3c139923ded-metrics-certs\") pod \"maas-api-e2e-extmod-a-61c297-7bf69564c7-8p7fp\" (UID: \"1249045d-1bca-4013-a9f7-f3c139923ded\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-61c297-7bf69564c7-8p7fp"
Nov 02 03:58:47.888866 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:58:47.888833    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n45rv\" (UniqueName: \"kubernetes.io/projected/1249045d-1bca-4013-a9f7-f3c139923ded-kube-api-access-n45rv\") pod \"maas-api-e2e-extmod-a-61c297-7bf69564c7-8p7fp\" (UID: \"1249045d-1bca-4013-a9f7-f3c139923ded\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-61c297-7bf69564c7-8p7fp"
Nov 02 03:58:48.196989 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:58:48.196948    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-3b27a5/auth-test-model-3b27a5-kserve-74c5647cbf-zfzcs" podUID="9ad2c2f7-2198-4ec7-bd1c-bcf5ab3cc455" containerName="main" containerID="cri-o://28f8457572157facb0dcf23d145a9256e11ba8f6d63a0b3c3722bac6245ef7d4" gracePeriod=45
Nov 02 03:58:48.335896 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:58:48.335862    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1249045d-1bca-4013-a9f7-f3c139923ded-maas-api-tls\") pod \"maas-api-e2e-extmod-a-61c297-7bf69564c7-8p7fp\" (UID: \"1249045d-1bca-4013-a9f7-f3c139923ded\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-61c297-7bf69564c7-8p7fp"
Nov 02 03:58:48.338186 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:58:48.338158    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1249045d-1bca-4013-a9f7-f3c139923ded-maas-api-tls\") pod \"maas-api-e2e-extmod-a-61c297-7bf69564c7-8p7fp\" (UID: \"1249045d-1bca-4013-a9f7-f3c139923ded\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-61c297-7bf69564c7-8p7fp"
Nov 02 03:58:48.474600 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:58:48.474571    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-3b27a5/auth-test-model-3b27a5-kserve-74c5647cbf-zfzcs"
Nov 02 03:58:48.537986 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:58:48.537957    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ad2c2f7-2198-4ec7-bd1c-bcf5ab3cc455-tls-certs\" (UniqueName: \"kubernetes.io/secret/9ad2c2f7-2198-4ec7-bd1c-bcf5ab3cc455-tls-certs\") pod \"9ad2c2f7-2198-4ec7-bd1c-bcf5ab3cc455\" (UID: \"9ad2c2f7-2198-4ec7-bd1c-bcf5ab3cc455\") "
Nov 02 03:58:48.538167 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:58:48.538007    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ad2c2f7-2198-4ec7-bd1c-bcf5ab3cc455-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ad2c2f7-2198-4ec7-bd1c-bcf5ab3cc455-dshm\") pod \"9ad2c2f7-2198-4ec7-bd1c-bcf5ab3cc455\" (UID: \"9ad2c2f7-2198-4ec7-bd1c-bcf5ab3cc455\") "
Nov 02 03:58:48.538167 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:58:48.538127    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ad2c2f7-2198-4ec7-bd1c-bcf5ab3cc455-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ad2c2f7-2198-4ec7-bd1c-bcf5ab3cc455-tmp-dir\") pod \"9ad2c2f7-2198-4ec7-bd1c-bcf5ab3cc455\" (UID: \"9ad2c2f7-2198-4ec7-bd1c-bcf5ab3cc455\") "
Nov 02 03:58:48.538290 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:58:48.538166    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ad2c2f7-2198-4ec7-bd1c-bcf5ab3cc455-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ad2c2f7-2198-4ec7-bd1c-bcf5ab3cc455-model-cache\") pod \"9ad2c2f7-2198-4ec7-bd1c-bcf5ab3cc455\" (UID: \"9ad2c2f7-2198-4ec7-bd1c-bcf5ab3cc455\") "
Nov 02 03:58:48.538290 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:58:48.538216    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ad2c2f7-2198-4ec7-bd1c-bcf5ab3cc455-kube-api-access-f9xzd\" (UniqueName: \"kubernetes.io/projected/9ad2c2f7-2198-4ec7-bd1c-bcf5ab3cc455-kube-api-access-f9xzd\") pod \"9ad2c2f7-2198-4ec7-bd1c-bcf5ab3cc455\" (UID: \"9ad2c2f7-2198-4ec7-bd1c-bcf5ab3cc455\") "
Nov 02 03:58:48.538290 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:58:48.538269    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ad2c2f7-2198-4ec7-bd1c-bcf5ab3cc455-home\" (UniqueName: \"kubernetes.io/empty-dir/9ad2c2f7-2198-4ec7-bd1c-bcf5ab3cc455-home\") pod \"9ad2c2f7-2198-4ec7-bd1c-bcf5ab3cc455\" (UID: \"9ad2c2f7-2198-4ec7-bd1c-bcf5ab3cc455\") "
Nov 02 03:58:48.539004 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:58:48.538495    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ad2c2f7-2198-4ec7-bd1c-bcf5ab3cc455-tmp-dir" pod "9ad2c2f7-2198-4ec7-bd1c-bcf5ab3cc455" (UID: "9ad2c2f7-2198-4ec7-bd1c-bcf5ab3cc455"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:58:48.539004 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:58:48.538737    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ad2c2f7-2198-4ec7-bd1c-bcf5ab3cc455-home" pod "9ad2c2f7-2198-4ec7-bd1c-bcf5ab3cc455" (UID: "9ad2c2f7-2198-4ec7-bd1c-bcf5ab3cc455"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:58:48.539004 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:58:48.538957    4496 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ad2c2f7-2198-4ec7-bd1c-bcf5ab3cc455-tmp-dir\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:58:48.539004 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:58:48.538978    4496 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ad2c2f7-2198-4ec7-bd1c-bcf5ab3cc455-home\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:58:48.539542 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:58:48.539515    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ad2c2f7-2198-4ec7-bd1c-bcf5ab3cc455-model-cache" pod "9ad2c2f7-2198-4ec7-bd1c-bcf5ab3cc455" (UID: "9ad2c2f7-2198-4ec7-bd1c-bcf5ab3cc455"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:58:48.540525 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:58:48.540498    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ad2c2f7-2198-4ec7-bd1c-bcf5ab3cc455-tls-certs" pod "9ad2c2f7-2198-4ec7-bd1c-bcf5ab3cc455" (UID: "9ad2c2f7-2198-4ec7-bd1c-bcf5ab3cc455"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 03:58:48.540607 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:58:48.540558    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ad2c2f7-2198-4ec7-bd1c-bcf5ab3cc455-dshm" pod "9ad2c2f7-2198-4ec7-bd1c-bcf5ab3cc455" (UID: "9ad2c2f7-2198-4ec7-bd1c-bcf5ab3cc455"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 03:58:48.540798 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:58:48.540777    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ad2c2f7-2198-4ec7-bd1c-bcf5ab3cc455-kube-api-access-f9xzd" pod "9ad2c2f7-2198-4ec7-bd1c-bcf5ab3cc455" (UID: "9ad2c2f7-2198-4ec7-bd1c-bcf5ab3cc455"). InnerVolumeSpecName "kube-api-access-f9xzd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 03:58:48.609733 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:58:48.609409    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-61c297-7bf69564c7-8p7fp"
Nov 02 03:58:48.643058 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:58:48.640986    4496 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ad2c2f7-2198-4ec7-bd1c-bcf5ab3cc455-tls-certs\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:58:48.643058 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:58:48.641033    4496 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ad2c2f7-2198-4ec7-bd1c-bcf5ab3cc455-dshm\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:58:48.643058 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:58:48.641052    4496 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ad2c2f7-2198-4ec7-bd1c-bcf5ab3cc455-model-cache\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:58:48.643058 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:58:48.641068    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9xzd\" (UniqueName: \"kubernetes.io/projected/9ad2c2f7-2198-4ec7-bd1c-bcf5ab3cc455-kube-api-access-f9xzd\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 03:58:48.989911 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:58:48.989880    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-61c297-7bf69564c7-8p7fp"]
Nov 02 03:58:48.990314 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:58:48.990254    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1249045d_1bca_4013_a9f7_f3c139923ded.slice/crio-e25941d1910aebbfdfbca69a415bdda9d1a7b463a2ded4f1183c5de3a1ddf2d7 WatchSource:0}: Error finding container e25941d1910aebbfdfbca69a415bdda9d1a7b463a2ded4f1183c5de3a1ddf2d7: Status 404 returned error can't find the container with id e25941d1910aebbfdfbca69a415bdda9d1a7b463a2ded4f1183c5de3a1ddf2d7
Nov 02 03:58:48.991634 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:58:48.991611    4496 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 03:58:49.125741 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:58:49.125649    4496 generic.go:356] "Generic (PLEG): container finished" podID="9ad2c2f7-2198-4ec7-bd1c-bcf5ab3cc455" containerID="28f8457572157facb0dcf23d145a9256e11ba8f6d63a0b3c3722bac6245ef7d4" exitCode=0
Nov 02 03:58:49.125741 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:58:49.125691    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-3b27a5/auth-test-model-3b27a5-kserve-74c5647cbf-zfzcs"
Nov 02 03:58:49.125741 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:58:49.125729    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-3b27a5/auth-test-model-3b27a5-kserve-74c5647cbf-zfzcs" event={"ID":"9ad2c2f7-2198-4ec7-bd1c-bcf5ab3cc455","Type":"ContainerDied","Data":"28f8457572157facb0dcf23d145a9256e11ba8f6d63a0b3c3722bac6245ef7d4"}
Nov 02 03:58:49.126039 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:58:49.125778    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-3b27a5/auth-test-model-3b27a5-kserve-74c5647cbf-zfzcs" event={"ID":"9ad2c2f7-2198-4ec7-bd1c-bcf5ab3cc455","Type":"ContainerDied","Data":"02d9e9983315d4328dc371ad29c7d62487f0521acbc954673720689365fa0587"}
Nov 02 03:58:49.126039 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:58:49.125801    4496 scope.go:122] "RemoveContainer" containerID="28f8457572157facb0dcf23d145a9256e11ba8f6d63a0b3c3722bac6245ef7d4"
Nov 02 03:58:49.127302 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:58:49.127279    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-61c297-7bf69564c7-8p7fp" event={"ID":"1249045d-1bca-4013-a9f7-f3c139923ded","Type":"ContainerStarted","Data":"e25941d1910aebbfdfbca69a415bdda9d1a7b463a2ded4f1183c5de3a1ddf2d7"}
Nov 02 03:58:49.136193 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:58:49.136170    4496 scope.go:122] "RemoveContainer" containerID="28f8457572157facb0dcf23d145a9256e11ba8f6d63a0b3c3722bac6245ef7d4"
Nov 02 03:58:49.136478 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:58:49.136459    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28f8457572157facb0dcf23d145a9256e11ba8f6d63a0b3c3722bac6245ef7d4\": container with ID starting with 28f8457572157facb0dcf23d145a9256e11ba8f6d63a0b3c3722bac6245ef7d4 not found: ID does not exist" containerID="28f8457572157facb0dcf23d145a9256e11ba8f6d63a0b3c3722bac6245ef7d4"
Nov 02 03:58:49.136555 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:58:49.136489    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28f8457572157facb0dcf23d145a9256e11ba8f6d63a0b3c3722bac6245ef7d4"} err="failed to get container status \"28f8457572157facb0dcf23d145a9256e11ba8f6d63a0b3c3722bac6245ef7d4\": rpc error: code = NotFound desc = could not find container \"28f8457572157facb0dcf23d145a9256e11ba8f6d63a0b3c3722bac6245ef7d4\": container with ID starting with 28f8457572157facb0dcf23d145a9256e11ba8f6d63a0b3c3722bac6245ef7d4 not found: ID does not exist"
Nov 02 03:58:49.293543 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:58:49.293505    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-3b27a5/auth-test-model-3b27a5-kserve-74c5647cbf-zfzcs"]
Nov 02 03:58:49.346400 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:58:49.346360    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-3b27a5/auth-test-model-3b27a5-kserve-74c5647cbf-zfzcs"]
Nov 02 03:58:49.578336 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:58:49.578301    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ad2c2f7-2198-4ec7-bd1c-bcf5ab3cc455" path="/var/lib/kubelet/pods/9ad2c2f7-2198-4ec7-bd1c-bcf5ab3cc455/volumes"
Nov 02 03:58:50.137339 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:58:50.137303    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-61c297-7bf69564c7-8p7fp" event={"ID":"1249045d-1bca-4013-a9f7-f3c139923ded","Type":"ContainerStarted","Data":"1e3f1bef6ec9f81799a3456b48dfff5490a7deeaa7ec024f3215c08ea2f2ad5a"}
Nov 02 03:58:50.137774 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:58:50.137453    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-61c297-7bf69564c7-8p7fp"
Nov 02 03:58:50.255279 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:58:50.255227    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-61c297-7bf69564c7-8p7fp" podStartSLOduration=2.319511741 podStartE2EDuration="3.255213637s" totalImagesPullingTime="935.701896ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:58:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:58:50.253050681 +0000 UTC m=+4270.831658058" watchObservedRunningTime="2026-11-02 03:58:50.255213637 +0000 UTC m=+4270.833821017"
Nov 02 03:58:56.151218 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:58:56.151187    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-61c297-7bf69564c7-8p7fp"
Nov 02 03:58:57.782561 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:58:57.782527    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-9b7f43-745986bb4b-wc44k"]
Nov 02 03:58:57.783225 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:58:57.783209    4496 cpu_manager.go:414] "containerMap: removing container" podUID="9ad2c2f7-2198-4ec7-bd1c-bcf5ab3cc455" containerName="main"
Nov 02 03:58:57.783225 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:58:57.783226    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ad2c2f7-2198-4ec7-bd1c-bcf5ab3cc455" containerName="main"
Nov 02 03:58:57.783355 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:58:57.783346    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ad2c2f7-2198-4ec7-bd1c-bcf5ab3cc455" containerName="main"
Nov 02 03:58:57.787003 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:58:57.786981    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9b7f43-745986bb4b-wc44k"
Nov 02 03:58:57.793063 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:58:57.793032    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-9b7f43-745986bb4b-wc44k"]
Nov 02 03:58:57.840351 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:58:57.840307    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-9b7f43-serving-cert\""
Nov 02 03:58:57.855306 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:58:57.855272    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ce4cf66-9b27-4031-aa2b-b2430b012b97-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-9b7f43-745986bb4b-wc44k\" (UID: \"3ce4cf66-9b27-4031-aa2b-b2430b012b97\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9b7f43-745986bb4b-wc44k"
Nov 02 03:58:57.855458 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:58:57.855371    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5l67d\" (UniqueName: \"kubernetes.io/projected/3ce4cf66-9b27-4031-aa2b-b2430b012b97-kube-api-access-5l67d\") pod \"maas-api-e2e-extmod-a-9b7f43-745986bb4b-wc44k\" (UID: \"3ce4cf66-9b27-4031-aa2b-b2430b012b97\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9b7f43-745986bb4b-wc44k"
Nov 02 03:58:57.855506 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:58:57.855477    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ce4cf66-9b27-4031-aa2b-b2430b012b97-maas-api-tls\") pod \"maas-api-e2e-extmod-a-9b7f43-745986bb4b-wc44k\" (UID: \"3ce4cf66-9b27-4031-aa2b-b2430b012b97\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9b7f43-745986bb4b-wc44k"
Nov 02 03:58:57.855678 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:58:57.855640    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ce4cf66-9b27-4031-aa2b-b2430b012b97-metrics-certs\") pod \"maas-api-e2e-extmod-a-9b7f43-745986bb4b-wc44k\" (UID: \"3ce4cf66-9b27-4031-aa2b-b2430b012b97\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9b7f43-745986bb4b-wc44k"
Nov 02 03:58:57.956824 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:58:57.956787    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ce4cf66-9b27-4031-aa2b-b2430b012b97-metrics-certs\") pod \"maas-api-e2e-extmod-a-9b7f43-745986bb4b-wc44k\" (UID: \"3ce4cf66-9b27-4031-aa2b-b2430b012b97\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9b7f43-745986bb4b-wc44k"
Nov 02 03:58:57.956988 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:58:57.956849    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ce4cf66-9b27-4031-aa2b-b2430b012b97-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-9b7f43-745986bb4b-wc44k\" (UID: \"3ce4cf66-9b27-4031-aa2b-b2430b012b97\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9b7f43-745986bb4b-wc44k"
Nov 02 03:58:57.956988 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:58:57.956891    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5l67d\" (UniqueName: \"kubernetes.io/projected/3ce4cf66-9b27-4031-aa2b-b2430b012b97-kube-api-access-5l67d\") pod \"maas-api-e2e-extmod-a-9b7f43-745986bb4b-wc44k\" (UID: \"3ce4cf66-9b27-4031-aa2b-b2430b012b97\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9b7f43-745986bb4b-wc44k"
Nov 02 03:58:57.956988 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:58:57.956971    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ce4cf66-9b27-4031-aa2b-b2430b012b97-maas-api-tls\") pod \"maas-api-e2e-extmod-a-9b7f43-745986bb4b-wc44k\" (UID: \"3ce4cf66-9b27-4031-aa2b-b2430b012b97\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9b7f43-745986bb4b-wc44k"
Nov 02 03:58:57.957187 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:58:57.957166    4496 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-9b7f43-serving-cert: secret "maas-api-e2e-extmod-a-9b7f43-serving-cert" not found
Nov 02 03:58:57.957248 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:58:57.957237    4496 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ce4cf66-9b27-4031-aa2b-b2430b012b97-maas-api-tls podName:3ce4cf66-9b27-4031-aa2b-b2430b012b97 nodeName:}" failed. No retries permitted until 2026-11-02 03:58:58.457217087 +0000 UTC m=+4279.035824447 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3ce4cf66-9b27-4031-aa2b-b2430b012b97-maas-api-tls") pod "maas-api-e2e-extmod-a-9b7f43-745986bb4b-wc44k" (UID: "3ce4cf66-9b27-4031-aa2b-b2430b012b97") : secret "maas-api-e2e-extmod-a-9b7f43-serving-cert" not found
Nov 02 03:58:57.957970 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:58:57.957941    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ce4cf66-9b27-4031-aa2b-b2430b012b97-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-9b7f43-745986bb4b-wc44k\" (UID: \"3ce4cf66-9b27-4031-aa2b-b2430b012b97\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9b7f43-745986bb4b-wc44k"
Nov 02 03:58:57.960626 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:58:57.960598    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ce4cf66-9b27-4031-aa2b-b2430b012b97-metrics-certs\") pod \"maas-api-e2e-extmod-a-9b7f43-745986bb4b-wc44k\" (UID: \"3ce4cf66-9b27-4031-aa2b-b2430b012b97\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9b7f43-745986bb4b-wc44k"
Nov 02 03:58:58.017152 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:58:58.017126    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5l67d\" (UniqueName: \"kubernetes.io/projected/3ce4cf66-9b27-4031-aa2b-b2430b012b97-kube-api-access-5l67d\") pod \"maas-api-e2e-extmod-a-9b7f43-745986bb4b-wc44k\" (UID: \"3ce4cf66-9b27-4031-aa2b-b2430b012b97\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9b7f43-745986bb4b-wc44k"
Nov 02 03:58:58.463780 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:58:58.463734    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ce4cf66-9b27-4031-aa2b-b2430b012b97-maas-api-tls\") pod \"maas-api-e2e-extmod-a-9b7f43-745986bb4b-wc44k\" (UID: \"3ce4cf66-9b27-4031-aa2b-b2430b012b97\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9b7f43-745986bb4b-wc44k"
Nov 02 03:58:58.466328 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:58:58.466296    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ce4cf66-9b27-4031-aa2b-b2430b012b97-maas-api-tls\") pod \"maas-api-e2e-extmod-a-9b7f43-745986bb4b-wc44k\" (UID: \"3ce4cf66-9b27-4031-aa2b-b2430b012b97\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9b7f43-745986bb4b-wc44k"
Nov 02 03:58:58.703591 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:58:58.703557    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9b7f43-745986bb4b-wc44k"
Nov 02 03:58:59.089592 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:58:59.089561    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-9b7f43-745986bb4b-wc44k"]
Nov 02 03:58:59.090341 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:58:59.090316    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ce4cf66_9b27_4031_aa2b_b2430b012b97.slice/crio-e4d858b87b96c01c0dbbe2e6d9982cf9a2a898fd9b329d5079d4b860e32fca32 WatchSource:0}: Error finding container e4d858b87b96c01c0dbbe2e6d9982cf9a2a898fd9b329d5079d4b860e32fca32: Status 404 returned error can't find the container with id e4d858b87b96c01c0dbbe2e6d9982cf9a2a898fd9b329d5079d4b860e32fca32
Nov 02 03:58:59.205921 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:58:59.205890    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9b7f43-745986bb4b-wc44k" event={"ID":"3ce4cf66-9b27-4031-aa2b-b2430b012b97","Type":"ContainerStarted","Data":"e4d858b87b96c01c0dbbe2e6d9982cf9a2a898fd9b329d5079d4b860e32fca32"}
Nov 02 03:59:00.214467 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:59:00.214369    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9b7f43-745986bb4b-wc44k" event={"ID":"3ce4cf66-9b27-4031-aa2b-b2430b012b97","Type":"ContainerStarted","Data":"9ff6415704e0df45d4c0db7eb1fcfb7d89cf10b01891d17d917b31d0394807a0"}
Nov 02 03:59:00.214467 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:59:00.214434    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9b7f43-745986bb4b-wc44k"
Nov 02 03:59:00.334372 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:59:00.334312    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9b7f43-745986bb4b-wc44k" podStartSLOduration=2.532291001 podStartE2EDuration="3.334294498s" totalImagesPullingTime="802.003497ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:58:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:59:00.329126183 +0000 UTC m=+4280.907733563" watchObservedRunningTime="2026-11-02 03:59:00.334294498 +0000 UTC m=+4280.912901880"
Nov 02 03:59:03.527583 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:59:03.527548    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 03:59:06.228244 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:59:06.228208    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9b7f43-745986bb4b-wc44k"
Nov 02 03:59:33.541672 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:59:33.541631    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 03:59:37.024493 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:59:37.024456    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-ae1315-695b4db8d5-hwgpm"]
Nov 02 03:59:37.029248 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:59:37.029225    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ae1315-695b4db8d5-hwgpm"
Nov 02 03:59:37.086245 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:59:37.086218    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-ae1315-695b4db8d5-hwgpm"]
Nov 02 03:59:37.125215 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:59:37.125183    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc4e9a96-bcbe-4993-9509-3987dcedff81-maas-api-tls\") pod \"maas-api-e2e-extmod-a-ae1315-695b4db8d5-hwgpm\" (UID: \"fc4e9a96-bcbe-4993-9509-3987dcedff81\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ae1315-695b4db8d5-hwgpm"
Nov 02 03:59:37.125410 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:59:37.125387    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc4e9a96-bcbe-4993-9509-3987dcedff81-metrics-certs\") pod \"maas-api-e2e-extmod-a-ae1315-695b4db8d5-hwgpm\" (UID: \"fc4e9a96-bcbe-4993-9509-3987dcedff81\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ae1315-695b4db8d5-hwgpm"
Nov 02 03:59:37.125472 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:59:37.125427    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc4e9a96-bcbe-4993-9509-3987dcedff81-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-ae1315-695b4db8d5-hwgpm\" (UID: \"fc4e9a96-bcbe-4993-9509-3987dcedff81\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ae1315-695b4db8d5-hwgpm"
Nov 02 03:59:37.125472 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:59:37.125447    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpctr\" (UniqueName: \"kubernetes.io/projected/fc4e9a96-bcbe-4993-9509-3987dcedff81-kube-api-access-gpctr\") pod \"maas-api-e2e-extmod-a-ae1315-695b4db8d5-hwgpm\" (UID: \"fc4e9a96-bcbe-4993-9509-3987dcedff81\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ae1315-695b4db8d5-hwgpm"
Nov 02 03:59:37.132254 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:59:37.132232    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-ae1315-serving-cert\""
Nov 02 03:59:37.226258 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:59:37.226219    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc4e9a96-bcbe-4993-9509-3987dcedff81-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-ae1315-695b4db8d5-hwgpm\" (UID: \"fc4e9a96-bcbe-4993-9509-3987dcedff81\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ae1315-695b4db8d5-hwgpm"
Nov 02 03:59:37.226443 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:59:37.226264    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpctr\" (UniqueName: \"kubernetes.io/projected/fc4e9a96-bcbe-4993-9509-3987dcedff81-kube-api-access-gpctr\") pod \"maas-api-e2e-extmod-a-ae1315-695b4db8d5-hwgpm\" (UID: \"fc4e9a96-bcbe-4993-9509-3987dcedff81\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ae1315-695b4db8d5-hwgpm"
Nov 02 03:59:37.226443 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:59:37.226358    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc4e9a96-bcbe-4993-9509-3987dcedff81-maas-api-tls\") pod \"maas-api-e2e-extmod-a-ae1315-695b4db8d5-hwgpm\" (UID: \"fc4e9a96-bcbe-4993-9509-3987dcedff81\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ae1315-695b4db8d5-hwgpm"
Nov 02 03:59:37.226563 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:59:37.226506    4496 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-ae1315-serving-cert: secret "maas-api-e2e-extmod-a-ae1315-serving-cert" not found
Nov 02 03:59:37.226622 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:59:37.226574    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc4e9a96-bcbe-4993-9509-3987dcedff81-metrics-certs\") pod \"maas-api-e2e-extmod-a-ae1315-695b4db8d5-hwgpm\" (UID: \"fc4e9a96-bcbe-4993-9509-3987dcedff81\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ae1315-695b4db8d5-hwgpm"
Nov 02 03:59:37.226689 ip-10-0-133-163 kubenswrapper[4496]: E1102 03:59:37.226673    4496 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc4e9a96-bcbe-4993-9509-3987dcedff81-maas-api-tls podName:fc4e9a96-bcbe-4993-9509-3987dcedff81 nodeName:}" failed. No retries permitted until 2026-11-02 03:59:37.726647832 +0000 UTC m=+4318.305255192 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fc4e9a96-bcbe-4993-9509-3987dcedff81-maas-api-tls") pod "maas-api-e2e-extmod-a-ae1315-695b4db8d5-hwgpm" (UID: "fc4e9a96-bcbe-4993-9509-3987dcedff81") : secret "maas-api-e2e-extmod-a-ae1315-serving-cert" not found
Nov 02 03:59:37.226878 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:59:37.226856    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc4e9a96-bcbe-4993-9509-3987dcedff81-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-ae1315-695b4db8d5-hwgpm\" (UID: \"fc4e9a96-bcbe-4993-9509-3987dcedff81\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ae1315-695b4db8d5-hwgpm"
Nov 02 03:59:37.228913 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:59:37.228892    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc4e9a96-bcbe-4993-9509-3987dcedff81-metrics-certs\") pod \"maas-api-e2e-extmod-a-ae1315-695b4db8d5-hwgpm\" (UID: \"fc4e9a96-bcbe-4993-9509-3987dcedff81\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ae1315-695b4db8d5-hwgpm"
Nov 02 03:59:37.289129 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:59:37.289046    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpctr\" (UniqueName: \"kubernetes.io/projected/fc4e9a96-bcbe-4993-9509-3987dcedff81-kube-api-access-gpctr\") pod \"maas-api-e2e-extmod-a-ae1315-695b4db8d5-hwgpm\" (UID: \"fc4e9a96-bcbe-4993-9509-3987dcedff81\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ae1315-695b4db8d5-hwgpm"
Nov 02 03:59:37.731890 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:59:37.731864    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc4e9a96-bcbe-4993-9509-3987dcedff81-maas-api-tls\") pod \"maas-api-e2e-extmod-a-ae1315-695b4db8d5-hwgpm\" (UID: \"fc4e9a96-bcbe-4993-9509-3987dcedff81\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ae1315-695b4db8d5-hwgpm"
Nov 02 03:59:37.734324 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:59:37.734291    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc4e9a96-bcbe-4993-9509-3987dcedff81-maas-api-tls\") pod \"maas-api-e2e-extmod-a-ae1315-695b4db8d5-hwgpm\" (UID: \"fc4e9a96-bcbe-4993-9509-3987dcedff81\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ae1315-695b4db8d5-hwgpm"
Nov 02 03:59:37.945698 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:59:37.945666    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ae1315-695b4db8d5-hwgpm"
Nov 02 03:59:38.331563 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:59:38.331531    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-ae1315-695b4db8d5-hwgpm"]
Nov 02 03:59:38.332987 ip-10-0-133-163 kubenswrapper[4496]: W1102 03:59:38.332959    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc4e9a96_bcbe_4993_9509_3987dcedff81.slice/crio-82ba66e84a71bcf63760f1421c13b6355dbb5adb529e5208734596e2f64f4c43 WatchSource:0}: Error finding container 82ba66e84a71bcf63760f1421c13b6355dbb5adb529e5208734596e2f64f4c43: Status 404 returned error can't find the container with id 82ba66e84a71bcf63760f1421c13b6355dbb5adb529e5208734596e2f64f4c43
Nov 02 03:59:38.502520 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:59:38.502476    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ae1315-695b4db8d5-hwgpm" event={"ID":"fc4e9a96-bcbe-4993-9509-3987dcedff81","Type":"ContainerStarted","Data":"82ba66e84a71bcf63760f1421c13b6355dbb5adb529e5208734596e2f64f4c43"}
Nov 02 03:59:39.512335 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:59:39.512296    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ae1315-695b4db8d5-hwgpm" event={"ID":"fc4e9a96-bcbe-4993-9509-3987dcedff81","Type":"ContainerStarted","Data":"3da6267f8c6d3f0f95deddaa93f6fb9ac576afcb159ae457a5357796c9f23f9a"}
Nov 02 03:59:39.512882 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:59:39.512391    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ae1315-695b4db8d5-hwgpm"
Nov 02 03:59:39.629978 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:59:39.629918    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ae1315-695b4db8d5-hwgpm" podStartSLOduration=2.832645363 podStartE2EDuration="3.629901311s" totalImagesPullingTime="797.255948ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:59:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:59:39.627333669 +0000 UTC m=+4320.205941050" watchObservedRunningTime="2026-11-02 03:59:39.629901311 +0000 UTC m=+4320.208508695"
Nov 02 03:59:45.525998 ip-10-0-133-163 kubenswrapper[4496]: I1102 03:59:45.525969    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ae1315-695b4db8d5-hwgpm"
Nov 02 04:00:00.247265 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:00:00.247224    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-61c297-29893200-45xd9"]
Nov 02 04:00:00.253692 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:00:00.253662    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-61c297-29893200-45xd9"
Nov 02 04:00:00.327006 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:00:00.326974    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9z5bk\""
Nov 02 04:00:00.391936 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:00:00.385151    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-61c297-29893200-45xd9"]
Nov 02 04:00:00.402076 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:00:00.402006    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85c3ba82-04d0-48f7-9da2-849ee385edc7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-61c297-29893200-45xd9\" (UID: \"85c3ba82-04d0-48f7-9da2-849ee385edc7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-61c297-29893200-45xd9"
Nov 02 04:00:00.402441 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:00:00.402413    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fndrg\" (UniqueName: \"kubernetes.io/projected/85c3ba82-04d0-48f7-9da2-849ee385edc7-kube-api-access-fndrg\") pod \"maas-api-key-cleanup-e2e-extmod-a-61c297-29893200-45xd9\" (UID: \"85c3ba82-04d0-48f7-9da2-849ee385edc7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-61c297-29893200-45xd9"
Nov 02 04:00:00.504114 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:00:00.503994    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85c3ba82-04d0-48f7-9da2-849ee385edc7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-61c297-29893200-45xd9\" (UID: \"85c3ba82-04d0-48f7-9da2-849ee385edc7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-61c297-29893200-45xd9"
Nov 02 04:00:00.504114 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:00:00.504086    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fndrg\" (UniqueName: \"kubernetes.io/projected/85c3ba82-04d0-48f7-9da2-849ee385edc7-kube-api-access-fndrg\") pod \"maas-api-key-cleanup-e2e-extmod-a-61c297-29893200-45xd9\" (UID: \"85c3ba82-04d0-48f7-9da2-849ee385edc7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-61c297-29893200-45xd9"
Nov 02 04:00:00.504712 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:00:00.504684    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85c3ba82-04d0-48f7-9da2-849ee385edc7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-61c297-29893200-45xd9\" (UID: \"85c3ba82-04d0-48f7-9da2-849ee385edc7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-61c297-29893200-45xd9"
Nov 02 04:00:00.563717 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:00:00.563682    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fndrg\" (UniqueName: \"kubernetes.io/projected/85c3ba82-04d0-48f7-9da2-849ee385edc7-kube-api-access-fndrg\") pod \"maas-api-key-cleanup-e2e-extmod-a-61c297-29893200-45xd9\" (UID: \"85c3ba82-04d0-48f7-9da2-849ee385edc7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-61c297-29893200-45xd9"
Nov 02 04:00:00.568926 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:00:00.568903    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-61c297-29893200-45xd9"
Nov 02 04:00:00.950322 ip-10-0-133-163 kubenswrapper[4496]: W1102 04:00:00.950290    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85c3ba82_04d0_48f7_9da2_849ee385edc7.slice/crio-fd6a73e9879a86e99995e487d2b783f664790e27d15940eab94fe98000897cbb WatchSource:0}: Error finding container fd6a73e9879a86e99995e487d2b783f664790e27d15940eab94fe98000897cbb: Status 404 returned error can't find the container with id fd6a73e9879a86e99995e487d2b783f664790e27d15940eab94fe98000897cbb
Nov 02 04:00:00.951206 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:00:00.951179    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-61c297-29893200-45xd9"]
Nov 02 04:00:01.690378 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:00:01.690290    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-61c297-29893200-45xd9" event={"ID":"85c3ba82-04d0-48f7-9da2-849ee385edc7","Type":"ContainerStarted","Data":"5043be8b9c9eb74de7b99fe08286b6b706d81b31f3f230096921615c0837aa64"}
Nov 02 04:00:01.690378 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:00:01.690345    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-61c297-29893200-45xd9" event={"ID":"85c3ba82-04d0-48f7-9da2-849ee385edc7","Type":"ContainerStarted","Data":"fd6a73e9879a86e99995e487d2b783f664790e27d15940eab94fe98000897cbb"}
Nov 02 04:00:01.806509 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:00:01.806456    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-61c297-29893200-45xd9" podStartSLOduration=1.806441996 podStartE2EDuration="1.806441996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 04:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 04:00:01.804862155 +0000 UTC m=+4342.383469537" watchObservedRunningTime="2026-11-02 04:00:01.806441996 +0000 UTC m=+4342.385049375"
Nov 02 04:00:02.699173 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:00:02.699142    4496 generic.go:356] "Generic (PLEG): container finished" podID="85c3ba82-04d0-48f7-9da2-849ee385edc7" containerID="5043be8b9c9eb74de7b99fe08286b6b706d81b31f3f230096921615c0837aa64" exitCode=60
Nov 02 04:00:02.699629 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:00:02.699224    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-61c297-29893200-45xd9" event={"ID":"85c3ba82-04d0-48f7-9da2-849ee385edc7","Type":"ContainerDied","Data":"5043be8b9c9eb74de7b99fe08286b6b706d81b31f3f230096921615c0837aa64"}
Nov 02 04:00:02.699629 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:00:02.699606    4496 scope.go:122] "RemoveContainer" containerID="5043be8b9c9eb74de7b99fe08286b6b706d81b31f3f230096921615c0837aa64"
Nov 02 04:00:03.528050 ip-10-0-133-163 kubenswrapper[4496]: E1102 04:00:03.527993    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 04:00:03.709292 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:00:03.709258    4496 generic.go:356] "Generic (PLEG): container finished" podID="85c3ba82-04d0-48f7-9da2-849ee385edc7" containerID="663b6a43b0262d81b5d0441ec119a5c31e194638c8984f7bd2d9367b6f2aa368" exitCode=60
Nov 02 04:00:03.709854 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:00:03.709296    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-61c297-29893200-45xd9" event={"ID":"85c3ba82-04d0-48f7-9da2-849ee385edc7","Type":"ContainerDied","Data":"663b6a43b0262d81b5d0441ec119a5c31e194638c8984f7bd2d9367b6f2aa368"}
Nov 02 04:00:03.709854 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:00:03.709362    4496 scope.go:122] "RemoveContainer" containerID="5043be8b9c9eb74de7b99fe08286b6b706d81b31f3f230096921615c0837aa64"
Nov 02 04:00:03.709854 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:00:03.709674    4496 scope.go:122] "RemoveContainer" containerID="663b6a43b0262d81b5d0441ec119a5c31e194638c8984f7bd2d9367b6f2aa368"
Nov 02 04:00:03.710043 ip-10-0-133-163 kubenswrapper[4496]: E1102 04:00:03.709998    4496 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-61c297-29893200-45xd9_odh-ai-gateway-infra(85c3ba82-04d0-48f7-9da2-849ee385edc7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-61c297-29893200-45xd9" podUID="85c3ba82-04d0-48f7-9da2-849ee385edc7"
Nov 02 04:00:04.720087 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:00:04.720055    4496 scope.go:122] "RemoveContainer" containerID="663b6a43b0262d81b5d0441ec119a5c31e194638c8984f7bd2d9367b6f2aa368"
Nov 02 04:00:04.720472 ip-10-0-133-163 kubenswrapper[4496]: E1102 04:00:04.720355    4496 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-61c297-29893200-45xd9_odh-ai-gateway-infra(85c3ba82-04d0-48f7-9da2-849ee385edc7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-61c297-29893200-45xd9" podUID="85c3ba82-04d0-48f7-9da2-849ee385edc7"
Nov 02 04:00:15.566826 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:00:15.566787    4496 scope.go:122] "RemoveContainer" containerID="663b6a43b0262d81b5d0441ec119a5c31e194638c8984f7bd2d9367b6f2aa368"
Nov 02 04:00:15.924307 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:00:15.924271    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-61f3b6-6c5cdc8df5-sr7j5"]
Nov 02 04:00:15.929075 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:00:15.929053    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-61f3b6-6c5cdc8df5-sr7j5"
Nov 02 04:00:15.982805 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:00:15.982774    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-61f3b6-serving-cert\""
Nov 02 04:00:15.995473 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:00:15.995444    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7fcc039-906f-40fd-8ffa-c244e4c2494c-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-61f3b6-6c5cdc8df5-sr7j5\" (UID: \"a7fcc039-906f-40fd-8ffa-c244e4c2494c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-61f3b6-6c5cdc8df5-sr7j5"
Nov 02 04:00:15.995600 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:00:15.995486    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7fcc039-906f-40fd-8ffa-c244e4c2494c-maas-api-tls\") pod \"maas-api-e2e-extmod-a-61f3b6-6c5cdc8df5-sr7j5\" (UID: \"a7fcc039-906f-40fd-8ffa-c244e4c2494c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-61f3b6-6c5cdc8df5-sr7j5"
Nov 02 04:00:15.995675 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:00:15.995646    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nszxs\" (UniqueName: \"kubernetes.io/projected/a7fcc039-906f-40fd-8ffa-c244e4c2494c-kube-api-access-nszxs\") pod \"maas-api-e2e-extmod-a-61f3b6-6c5cdc8df5-sr7j5\" (UID: \"a7fcc039-906f-40fd-8ffa-c244e4c2494c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-61f3b6-6c5cdc8df5-sr7j5"
Nov 02 04:00:15.995723 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:00:15.995700    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7fcc039-906f-40fd-8ffa-c244e4c2494c-metrics-certs\") pod \"maas-api-e2e-extmod-a-61f3b6-6c5cdc8df5-sr7j5\" (UID: \"a7fcc039-906f-40fd-8ffa-c244e4c2494c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-61f3b6-6c5cdc8df5-sr7j5"
Nov 02 04:00:16.016109 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:00:16.016081    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-61f3b6-6c5cdc8df5-sr7j5"]
Nov 02 04:00:16.097179 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:00:16.097098    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nszxs\" (UniqueName: \"kubernetes.io/projected/a7fcc039-906f-40fd-8ffa-c244e4c2494c-kube-api-access-nszxs\") pod \"maas-api-e2e-extmod-a-61f3b6-6c5cdc8df5-sr7j5\" (UID: \"a7fcc039-906f-40fd-8ffa-c244e4c2494c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-61f3b6-6c5cdc8df5-sr7j5"
Nov 02 04:00:16.097179 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:00:16.097156    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7fcc039-906f-40fd-8ffa-c244e4c2494c-metrics-certs\") pod \"maas-api-e2e-extmod-a-61f3b6-6c5cdc8df5-sr7j5\" (UID: \"a7fcc039-906f-40fd-8ffa-c244e4c2494c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-61f3b6-6c5cdc8df5-sr7j5"
Nov 02 04:00:16.097386 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:00:16.097333    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7fcc039-906f-40fd-8ffa-c244e4c2494c-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-61f3b6-6c5cdc8df5-sr7j5\" (UID: \"a7fcc039-906f-40fd-8ffa-c244e4c2494c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-61f3b6-6c5cdc8df5-sr7j5"
Nov 02 04:00:16.097386 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:00:16.097364    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7fcc039-906f-40fd-8ffa-c244e4c2494c-maas-api-tls\") pod \"maas-api-e2e-extmod-a-61f3b6-6c5cdc8df5-sr7j5\" (UID: \"a7fcc039-906f-40fd-8ffa-c244e4c2494c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-61f3b6-6c5cdc8df5-sr7j5"
Nov 02 04:00:16.097528 ip-10-0-133-163 kubenswrapper[4496]: E1102 04:00:16.097513    4496 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-61f3b6-serving-cert: secret "maas-api-e2e-extmod-a-61f3b6-serving-cert" not found
Nov 02 04:00:16.097620 ip-10-0-133-163 kubenswrapper[4496]: E1102 04:00:16.097607    4496 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7fcc039-906f-40fd-8ffa-c244e4c2494c-maas-api-tls podName:a7fcc039-906f-40fd-8ffa-c244e4c2494c nodeName:}" failed. No retries permitted until 2026-11-02 04:00:16.597583594 +0000 UTC m=+4357.176190956 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a7fcc039-906f-40fd-8ffa-c244e4c2494c-maas-api-tls") pod "maas-api-e2e-extmod-a-61f3b6-6c5cdc8df5-sr7j5" (UID: "a7fcc039-906f-40fd-8ffa-c244e4c2494c") : secret "maas-api-e2e-extmod-a-61f3b6-serving-cert" not found
Nov 02 04:00:16.098065 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:00:16.098039    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7fcc039-906f-40fd-8ffa-c244e4c2494c-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-61f3b6-6c5cdc8df5-sr7j5\" (UID: \"a7fcc039-906f-40fd-8ffa-c244e4c2494c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-61f3b6-6c5cdc8df5-sr7j5"
Nov 02 04:00:16.099561 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:00:16.099543    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7fcc039-906f-40fd-8ffa-c244e4c2494c-metrics-certs\") pod \"maas-api-e2e-extmod-a-61f3b6-6c5cdc8df5-sr7j5\" (UID: \"a7fcc039-906f-40fd-8ffa-c244e4c2494c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-61f3b6-6c5cdc8df5-sr7j5"
Nov 02 04:00:16.155997 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:00:16.155966    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nszxs\" (UniqueName: \"kubernetes.io/projected/a7fcc039-906f-40fd-8ffa-c244e4c2494c-kube-api-access-nszxs\") pod \"maas-api-e2e-extmod-a-61f3b6-6c5cdc8df5-sr7j5\" (UID: \"a7fcc039-906f-40fd-8ffa-c244e4c2494c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-61f3b6-6c5cdc8df5-sr7j5"
Nov 02 04:00:16.602448 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:00:16.602413    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7fcc039-906f-40fd-8ffa-c244e4c2494c-maas-api-tls\") pod \"maas-api-e2e-extmod-a-61f3b6-6c5cdc8df5-sr7j5\" (UID: \"a7fcc039-906f-40fd-8ffa-c244e4c2494c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-61f3b6-6c5cdc8df5-sr7j5"
Nov 02 04:00:16.604831 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:00:16.604811    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7fcc039-906f-40fd-8ffa-c244e4c2494c-maas-api-tls\") pod \"maas-api-e2e-extmod-a-61f3b6-6c5cdc8df5-sr7j5\" (UID: \"a7fcc039-906f-40fd-8ffa-c244e4c2494c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-61f3b6-6c5cdc8df5-sr7j5"
Nov 02 04:00:16.814483 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:00:16.814446    4496 generic.go:356] "Generic (PLEG): container finished" podID="85c3ba82-04d0-48f7-9da2-849ee385edc7" containerID="3e33ad32d4c826fb1ffae49f0d18693340447b85bb24bbc64fe3cb522deeb5ac" exitCode=60
Nov 02 04:00:16.814661 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:00:16.814521    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-61c297-29893200-45xd9" event={"ID":"85c3ba82-04d0-48f7-9da2-849ee385edc7","Type":"ContainerDied","Data":"3e33ad32d4c826fb1ffae49f0d18693340447b85bb24bbc64fe3cb522deeb5ac"}
Nov 02 04:00:16.814661 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:00:16.814602    4496 scope.go:122] "RemoveContainer" containerID="663b6a43b0262d81b5d0441ec119a5c31e194638c8984f7bd2d9367b6f2aa368"
Nov 02 04:00:16.815066 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:00:16.815047    4496 scope.go:122] "RemoveContainer" containerID="3e33ad32d4c826fb1ffae49f0d18693340447b85bb24bbc64fe3cb522deeb5ac"
Nov 02 04:00:16.815376 ip-10-0-133-163 kubenswrapper[4496]: E1102 04:00:16.815348    4496 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-61c297-29893200-45xd9_odh-ai-gateway-infra(85c3ba82-04d0-48f7-9da2-849ee385edc7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-61c297-29893200-45xd9" podUID="85c3ba82-04d0-48f7-9da2-849ee385edc7"
Nov 02 04:00:16.843387 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:00:16.843351    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-61f3b6-6c5cdc8df5-sr7j5"
Nov 02 04:00:17.225254 ip-10-0-133-163 kubenswrapper[4496]: W1102 04:00:17.225218    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7fcc039_906f_40fd_8ffa_c244e4c2494c.slice/crio-c5c7b02ac75632e54d322f4d2bae5215b52c6c2afbfc274ae16497e83316ed33 WatchSource:0}: Error finding container c5c7b02ac75632e54d322f4d2bae5215b52c6c2afbfc274ae16497e83316ed33: Status 404 returned error can't find the container with id c5c7b02ac75632e54d322f4d2bae5215b52c6c2afbfc274ae16497e83316ed33
Nov 02 04:00:17.225469 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:00:17.225260    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-61f3b6-6c5cdc8df5-sr7j5"]
Nov 02 04:00:17.823101 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:00:17.823068    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-61f3b6-6c5cdc8df5-sr7j5" event={"ID":"a7fcc039-906f-40fd-8ffa-c244e4c2494c","Type":"ContainerStarted","Data":"c5c7b02ac75632e54d322f4d2bae5215b52c6c2afbfc274ae16497e83316ed33"}
Nov 02 04:00:17.945289 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:00:17.945253    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-61c297-29893200-45xd9"]
Nov 02 04:00:18.117660 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:00:18.117631    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-61c297-29893200-45xd9"
Nov 02 04:00:18.226062 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:00:18.226034    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85c3ba82-04d0-48f7-9da2-849ee385edc7-kube-api-access-fndrg\" (UniqueName: \"kubernetes.io/projected/85c3ba82-04d0-48f7-9da2-849ee385edc7-kube-api-access-fndrg\") pod \"85c3ba82-04d0-48f7-9da2-849ee385edc7\" (UID: \"85c3ba82-04d0-48f7-9da2-849ee385edc7\") "
Nov 02 04:00:18.226203 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:00:18.226179    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85c3ba82-04d0-48f7-9da2-849ee385edc7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85c3ba82-04d0-48f7-9da2-849ee385edc7-maas-api-service-ca\") pod \"85c3ba82-04d0-48f7-9da2-849ee385edc7\" (UID: \"85c3ba82-04d0-48f7-9da2-849ee385edc7\") "
Nov 02 04:00:18.226570 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:00:18.226543    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85c3ba82-04d0-48f7-9da2-849ee385edc7-maas-api-service-ca" pod "85c3ba82-04d0-48f7-9da2-849ee385edc7" (UID: "85c3ba82-04d0-48f7-9da2-849ee385edc7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 04:00:18.226770 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:00:18.226754    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85c3ba82-04d0-48f7-9da2-849ee385edc7-maas-api-service-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:00:18.228580 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:00:18.228557    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85c3ba82-04d0-48f7-9da2-849ee385edc7-kube-api-access-fndrg" pod "85c3ba82-04d0-48f7-9da2-849ee385edc7" (UID: "85c3ba82-04d0-48f7-9da2-849ee385edc7"). InnerVolumeSpecName "kube-api-access-fndrg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 04:00:18.328308 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:00:18.328273    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fndrg\" (UniqueName: \"kubernetes.io/projected/85c3ba82-04d0-48f7-9da2-849ee385edc7-kube-api-access-fndrg\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:00:18.834274 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:00:18.834225    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-61f3b6-6c5cdc8df5-sr7j5" event={"ID":"a7fcc039-906f-40fd-8ffa-c244e4c2494c","Type":"ContainerStarted","Data":"ee662f489d2f38911fa7d500beb801ed96b03e123658b157aedf1860837f7467"}
Nov 02 04:00:18.834767 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:00:18.834475    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-61f3b6-6c5cdc8df5-sr7j5"
Nov 02 04:00:18.836334 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:00:18.836300    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-61c297-29893200-45xd9" event={"ID":"85c3ba82-04d0-48f7-9da2-849ee385edc7","Type":"ContainerDied","Data":"fd6a73e9879a86e99995e487d2b783f664790e27d15940eab94fe98000897cbb"}
Nov 02 04:00:18.836334 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:00:18.836317    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-61c297-29893200-45xd9"
Nov 02 04:00:18.836539 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:00:18.836360    4496 scope.go:122] "RemoveContainer" containerID="3e33ad32d4c826fb1ffae49f0d18693340447b85bb24bbc64fe3cb522deeb5ac"
Nov 02 04:00:18.956907 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:00:18.956849    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-61f3b6-6c5cdc8df5-sr7j5" podStartSLOduration=3.00385844 podStartE2EDuration="3.956831922s" totalImagesPullingTime="952.973482ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 04:00:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 04:00:18.953450508 +0000 UTC m=+4359.532057889" watchObservedRunningTime="2026-11-02 04:00:18.956831922 +0000 UTC m=+4359.535439307"
Nov 02 04:00:19.069305 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:00:19.069269    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-61c297-29893200-45xd9"]
Nov 02 04:00:19.078144 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:00:19.075696    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-61c297-29893200-45xd9"]
Nov 02 04:00:19.578157 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:00:19.578125    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85c3ba82-04d0-48f7-9da2-849ee385edc7" path="/var/lib/kubelet/pods/85c3ba82-04d0-48f7-9da2-849ee385edc7/volumes"
Nov 02 04:00:24.851675 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:00:24.851588    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-61f3b6-6c5cdc8df5-sr7j5"
Nov 02 04:00:33.526957 ip-10-0-133-163 kubenswrapper[4496]: E1102 04:00:33.526923    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 04:01:03.528529 ip-10-0-133-163 kubenswrapper[4496]: E1102 04:01:03.528488    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 04:01:33.534851 ip-10-0-133-163 kubenswrapper[4496]: E1102 04:01:33.528161    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 04:01:58.870743 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:01:58.870659    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-61f3b6-6c5cdc8df5-sr7j5"]
Nov 02 04:01:58.871158 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:01:58.870893    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-61f3b6-6c5cdc8df5-sr7j5" podUID="a7fcc039-906f-40fd-8ffa-c244e4c2494c" containerName="maas-api" containerID="cri-o://ee662f489d2f38911fa7d500beb801ed96b03e123658b157aedf1860837f7467" gracePeriod=30
Nov 02 04:01:59.150468 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:01:59.150438    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-61f3b6-6c5cdc8df5-sr7j5"
Nov 02 04:01:59.254319 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:01:59.254286    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7fcc039-906f-40fd-8ffa-c244e4c2494c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7fcc039-906f-40fd-8ffa-c244e4c2494c-openshift-service-ca\") pod \"a7fcc039-906f-40fd-8ffa-c244e4c2494c\" (UID: \"a7fcc039-906f-40fd-8ffa-c244e4c2494c\") "
Nov 02 04:01:59.254534 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:01:59.254354    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7fcc039-906f-40fd-8ffa-c244e4c2494c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7fcc039-906f-40fd-8ffa-c244e4c2494c-maas-api-tls\") pod \"a7fcc039-906f-40fd-8ffa-c244e4c2494c\" (UID: \"a7fcc039-906f-40fd-8ffa-c244e4c2494c\") "
Nov 02 04:01:59.254534 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:01:59.254382    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7fcc039-906f-40fd-8ffa-c244e4c2494c-kube-api-access-nszxs\" (UniqueName: \"kubernetes.io/projected/a7fcc039-906f-40fd-8ffa-c244e4c2494c-kube-api-access-nszxs\") pod \"a7fcc039-906f-40fd-8ffa-c244e4c2494c\" (UID: \"a7fcc039-906f-40fd-8ffa-c244e4c2494c\") "
Nov 02 04:01:59.254534 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:01:59.254473    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7fcc039-906f-40fd-8ffa-c244e4c2494c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7fcc039-906f-40fd-8ffa-c244e4c2494c-metrics-certs\") pod \"a7fcc039-906f-40fd-8ffa-c244e4c2494c\" (UID: \"a7fcc039-906f-40fd-8ffa-c244e4c2494c\") "
Nov 02 04:01:59.254748 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:01:59.254716    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7fcc039-906f-40fd-8ffa-c244e4c2494c-openshift-service-ca" pod "a7fcc039-906f-40fd-8ffa-c244e4c2494c" (UID: "a7fcc039-906f-40fd-8ffa-c244e4c2494c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 04:01:59.255160 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:01:59.255133    4496 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7fcc039-906f-40fd-8ffa-c244e4c2494c-openshift-service-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:01:59.256694 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:01:59.256662    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7fcc039-906f-40fd-8ffa-c244e4c2494c-kube-api-access-nszxs" pod "a7fcc039-906f-40fd-8ffa-c244e4c2494c" (UID: "a7fcc039-906f-40fd-8ffa-c244e4c2494c"). InnerVolumeSpecName "kube-api-access-nszxs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 04:01:59.256821 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:01:59.256721    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7fcc039-906f-40fd-8ffa-c244e4c2494c-maas-api-tls" pod "a7fcc039-906f-40fd-8ffa-c244e4c2494c" (UID: "a7fcc039-906f-40fd-8ffa-c244e4c2494c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 04:01:59.256872 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:01:59.256847    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7fcc039-906f-40fd-8ffa-c244e4c2494c-metrics-certs" pod "a7fcc039-906f-40fd-8ffa-c244e4c2494c" (UID: "a7fcc039-906f-40fd-8ffa-c244e4c2494c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 04:01:59.356634 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:01:59.356595    4496 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7fcc039-906f-40fd-8ffa-c244e4c2494c-metrics-certs\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:01:59.356634 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:01:59.356628    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7fcc039-906f-40fd-8ffa-c244e4c2494c-maas-api-tls\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:01:59.356634 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:01:59.356638    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nszxs\" (UniqueName: \"kubernetes.io/projected/a7fcc039-906f-40fd-8ffa-c244e4c2494c-kube-api-access-nszxs\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:01:59.623825 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:01:59.623788    4496 generic.go:356] "Generic (PLEG): container finished" podID="a7fcc039-906f-40fd-8ffa-c244e4c2494c" containerID="ee662f489d2f38911fa7d500beb801ed96b03e123658b157aedf1860837f7467" exitCode=0
Nov 02 04:01:59.623974 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:01:59.623828    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-61f3b6-6c5cdc8df5-sr7j5"
Nov 02 04:01:59.623974 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:01:59.623865    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-61f3b6-6c5cdc8df5-sr7j5" event={"ID":"a7fcc039-906f-40fd-8ffa-c244e4c2494c","Type":"ContainerDied","Data":"ee662f489d2f38911fa7d500beb801ed96b03e123658b157aedf1860837f7467"}
Nov 02 04:01:59.623974 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:01:59.623912    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-61f3b6-6c5cdc8df5-sr7j5" event={"ID":"a7fcc039-906f-40fd-8ffa-c244e4c2494c","Type":"ContainerDied","Data":"c5c7b02ac75632e54d322f4d2bae5215b52c6c2afbfc274ae16497e83316ed33"}
Nov 02 04:01:59.623974 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:01:59.623936    4496 scope.go:122] "RemoveContainer" containerID="ee662f489d2f38911fa7d500beb801ed96b03e123658b157aedf1860837f7467"
Nov 02 04:01:59.634566 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:01:59.634549    4496 scope.go:122] "RemoveContainer" containerID="ee662f489d2f38911fa7d500beb801ed96b03e123658b157aedf1860837f7467"
Nov 02 04:01:59.634854 ip-10-0-133-163 kubenswrapper[4496]: E1102 04:01:59.634828    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee662f489d2f38911fa7d500beb801ed96b03e123658b157aedf1860837f7467\": container with ID starting with ee662f489d2f38911fa7d500beb801ed96b03e123658b157aedf1860837f7467 not found: ID does not exist" containerID="ee662f489d2f38911fa7d500beb801ed96b03e123658b157aedf1860837f7467"
Nov 02 04:01:59.634964 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:01:59.634862    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee662f489d2f38911fa7d500beb801ed96b03e123658b157aedf1860837f7467"} err="failed to get container status \"ee662f489d2f38911fa7d500beb801ed96b03e123658b157aedf1860837f7467\": rpc error: code = NotFound desc = could not find container \"ee662f489d2f38911fa7d500beb801ed96b03e123658b157aedf1860837f7467\": container with ID starting with ee662f489d2f38911fa7d500beb801ed96b03e123658b157aedf1860837f7467 not found: ID does not exist"
Nov 02 04:01:59.792743 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:01:59.792706    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-61f3b6-6c5cdc8df5-sr7j5"]
Nov 02 04:01:59.796079 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:01:59.796050    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-61f3b6-6c5cdc8df5-sr7j5"]
Nov 02 04:02:01.576501 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:01.576463    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7fcc039-906f-40fd-8ffa-c244e4c2494c" path="/var/lib/kubelet/pods/a7fcc039-906f-40fd-8ffa-c244e4c2494c/volumes"
Nov 02 04:02:03.534988 ip-10-0-133-163 kubenswrapper[4496]: E1102 04:02:03.534948    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 04:02:04.871282 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:04.871246    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-61c297-7bf69564c7-8p7fp"]
Nov 02 04:02:04.871806 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:04.871439    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-61c297-7bf69564c7-8p7fp" podUID="1249045d-1bca-4013-a9f7-f3c139923ded" containerName="maas-api" containerID="cri-o://1e3f1bef6ec9f81799a3456b48dfff5490a7deeaa7ec024f3215c08ea2f2ad5a" gracePeriod=30
Nov 02 04:02:05.141814 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:05.141750    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-61c297-7bf69564c7-8p7fp"
Nov 02 04:02:05.227522 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:05.227493    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1249045d-1bca-4013-a9f7-f3c139923ded-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1249045d-1bca-4013-a9f7-f3c139923ded-maas-api-tls\") pod \"1249045d-1bca-4013-a9f7-f3c139923ded\" (UID: \"1249045d-1bca-4013-a9f7-f3c139923ded\") "
Nov 02 04:02:05.227667 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:05.227620    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1249045d-1bca-4013-a9f7-f3c139923ded-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1249045d-1bca-4013-a9f7-f3c139923ded-openshift-service-ca\") pod \"1249045d-1bca-4013-a9f7-f3c139923ded\" (UID: \"1249045d-1bca-4013-a9f7-f3c139923ded\") "
Nov 02 04:02:05.227667 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:05.227660    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1249045d-1bca-4013-a9f7-f3c139923ded-kube-api-access-n45rv\" (UniqueName: \"kubernetes.io/projected/1249045d-1bca-4013-a9f7-f3c139923ded-kube-api-access-n45rv\") pod \"1249045d-1bca-4013-a9f7-f3c139923ded\" (UID: \"1249045d-1bca-4013-a9f7-f3c139923ded\") "
Nov 02 04:02:05.227780 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:05.227675    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1249045d-1bca-4013-a9f7-f3c139923ded-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1249045d-1bca-4013-a9f7-f3c139923ded-metrics-certs\") pod \"1249045d-1bca-4013-a9f7-f3c139923ded\" (UID: \"1249045d-1bca-4013-a9f7-f3c139923ded\") "
Nov 02 04:02:05.228098 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:05.228067    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1249045d-1bca-4013-a9f7-f3c139923ded-openshift-service-ca" pod "1249045d-1bca-4013-a9f7-f3c139923ded" (UID: "1249045d-1bca-4013-a9f7-f3c139923ded"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 04:02:05.229659 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:05.229627    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1249045d-1bca-4013-a9f7-f3c139923ded-maas-api-tls" pod "1249045d-1bca-4013-a9f7-f3c139923ded" (UID: "1249045d-1bca-4013-a9f7-f3c139923ded"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 04:02:05.229769 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:05.229708    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1249045d-1bca-4013-a9f7-f3c139923ded-metrics-certs" pod "1249045d-1bca-4013-a9f7-f3c139923ded" (UID: "1249045d-1bca-4013-a9f7-f3c139923ded"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 04:02:05.229769 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:05.229749    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1249045d-1bca-4013-a9f7-f3c139923ded-kube-api-access-n45rv" pod "1249045d-1bca-4013-a9f7-f3c139923ded" (UID: "1249045d-1bca-4013-a9f7-f3c139923ded"). InnerVolumeSpecName "kube-api-access-n45rv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 04:02:05.329497 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:05.329463    4496 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1249045d-1bca-4013-a9f7-f3c139923ded-openshift-service-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:02:05.329497 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:05.329493    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n45rv\" (UniqueName: \"kubernetes.io/projected/1249045d-1bca-4013-a9f7-f3c139923ded-kube-api-access-n45rv\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:02:05.329711 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:05.329507    4496 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1249045d-1bca-4013-a9f7-f3c139923ded-metrics-certs\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:02:05.329711 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:05.329521    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1249045d-1bca-4013-a9f7-f3c139923ded-maas-api-tls\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:02:05.671871 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:05.671824    4496 generic.go:356] "Generic (PLEG): container finished" podID="1249045d-1bca-4013-a9f7-f3c139923ded" containerID="1e3f1bef6ec9f81799a3456b48dfff5490a7deeaa7ec024f3215c08ea2f2ad5a" exitCode=0
Nov 02 04:02:05.672079 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:05.671873    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-61c297-7bf69564c7-8p7fp"
Nov 02 04:02:05.672079 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:05.671882    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-61c297-7bf69564c7-8p7fp" event={"ID":"1249045d-1bca-4013-a9f7-f3c139923ded","Type":"ContainerDied","Data":"1e3f1bef6ec9f81799a3456b48dfff5490a7deeaa7ec024f3215c08ea2f2ad5a"}
Nov 02 04:02:05.672079 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:05.671945    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-61c297-7bf69564c7-8p7fp" event={"ID":"1249045d-1bca-4013-a9f7-f3c139923ded","Type":"ContainerDied","Data":"e25941d1910aebbfdfbca69a415bdda9d1a7b463a2ded4f1183c5de3a1ddf2d7"}
Nov 02 04:02:05.672079 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:05.671969    4496 scope.go:122] "RemoveContainer" containerID="1e3f1bef6ec9f81799a3456b48dfff5490a7deeaa7ec024f3215c08ea2f2ad5a"
Nov 02 04:02:05.684051 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:05.684010    4496 scope.go:122] "RemoveContainer" containerID="1e3f1bef6ec9f81799a3456b48dfff5490a7deeaa7ec024f3215c08ea2f2ad5a"
Nov 02 04:02:05.684316 ip-10-0-133-163 kubenswrapper[4496]: E1102 04:02:05.684290    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e3f1bef6ec9f81799a3456b48dfff5490a7deeaa7ec024f3215c08ea2f2ad5a\": container with ID starting with 1e3f1bef6ec9f81799a3456b48dfff5490a7deeaa7ec024f3215c08ea2f2ad5a not found: ID does not exist" containerID="1e3f1bef6ec9f81799a3456b48dfff5490a7deeaa7ec024f3215c08ea2f2ad5a"
Nov 02 04:02:05.684409 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:05.684329    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e3f1bef6ec9f81799a3456b48dfff5490a7deeaa7ec024f3215c08ea2f2ad5a"} err="failed to get container status \"1e3f1bef6ec9f81799a3456b48dfff5490a7deeaa7ec024f3215c08ea2f2ad5a\": rpc error: code = NotFound desc = could not find container \"1e3f1bef6ec9f81799a3456b48dfff5490a7deeaa7ec024f3215c08ea2f2ad5a\": container with ID starting with 1e3f1bef6ec9f81799a3456b48dfff5490a7deeaa7ec024f3215c08ea2f2ad5a not found: ID does not exist"
Nov 02 04:02:05.845381 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:05.845301    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-61c297-7bf69564c7-8p7fp"]
Nov 02 04:02:05.850778 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:05.850743    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-61c297-7bf69564c7-8p7fp"]
Nov 02 04:02:07.575409 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:07.575373    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1249045d-1bca-4013-a9f7-f3c139923ded" path="/var/lib/kubelet/pods/1249045d-1bca-4013-a9f7-f3c139923ded/volumes"
Nov 02 04:02:11.116637 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:11.116554    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-9b7f43-745986bb4b-wc44k"]
Nov 02 04:02:11.117445 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:11.116750    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9b7f43-745986bb4b-wc44k" podUID="3ce4cf66-9b27-4031-aa2b-b2430b012b97" containerName="maas-api" containerID="cri-o://9ff6415704e0df45d4c0db7eb1fcfb7d89cf10b01891d17d917b31d0394807a0" gracePeriod=30
Nov 02 04:02:11.222862 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:11.222818    4496 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9b7f43-745986bb4b-wc44k" podUID="3ce4cf66-9b27-4031-aa2b-b2430b012b97" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.1.22:8443/health\": dial tcp 10.134.1.22:8443: connect: connection refused"
Nov 02 04:02:11.410491 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:11.410464    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9b7f43-745986bb4b-wc44k"
Nov 02 04:02:11.506128 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:11.506101    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ce4cf66-9b27-4031-aa2b-b2430b012b97-kube-api-access-5l67d\" (UniqueName: \"kubernetes.io/projected/3ce4cf66-9b27-4031-aa2b-b2430b012b97-kube-api-access-5l67d\") pod \"3ce4cf66-9b27-4031-aa2b-b2430b012b97\" (UID: \"3ce4cf66-9b27-4031-aa2b-b2430b012b97\") "
Nov 02 04:02:11.506293 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:11.506196    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ce4cf66-9b27-4031-aa2b-b2430b012b97-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ce4cf66-9b27-4031-aa2b-b2430b012b97-maas-api-tls\") pod \"3ce4cf66-9b27-4031-aa2b-b2430b012b97\" (UID: \"3ce4cf66-9b27-4031-aa2b-b2430b012b97\") "
Nov 02 04:02:11.506345 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:11.506320    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ce4cf66-9b27-4031-aa2b-b2430b012b97-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ce4cf66-9b27-4031-aa2b-b2430b012b97-openshift-service-ca\") pod \"3ce4cf66-9b27-4031-aa2b-b2430b012b97\" (UID: \"3ce4cf66-9b27-4031-aa2b-b2430b012b97\") "
Nov 02 04:02:11.506396 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:11.506345    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ce4cf66-9b27-4031-aa2b-b2430b012b97-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ce4cf66-9b27-4031-aa2b-b2430b012b97-metrics-certs\") pod \"3ce4cf66-9b27-4031-aa2b-b2430b012b97\" (UID: \"3ce4cf66-9b27-4031-aa2b-b2430b012b97\") "
Nov 02 04:02:11.506727 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:11.506697    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ce4cf66-9b27-4031-aa2b-b2430b012b97-openshift-service-ca" pod "3ce4cf66-9b27-4031-aa2b-b2430b012b97" (UID: "3ce4cf66-9b27-4031-aa2b-b2430b012b97"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 04:02:11.508497 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:11.508469    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ce4cf66-9b27-4031-aa2b-b2430b012b97-metrics-certs" pod "3ce4cf66-9b27-4031-aa2b-b2430b012b97" (UID: "3ce4cf66-9b27-4031-aa2b-b2430b012b97"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 04:02:11.508664 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:11.508646    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ce4cf66-9b27-4031-aa2b-b2430b012b97-maas-api-tls" pod "3ce4cf66-9b27-4031-aa2b-b2430b012b97" (UID: "3ce4cf66-9b27-4031-aa2b-b2430b012b97"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 04:02:11.508747 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:11.508701    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ce4cf66-9b27-4031-aa2b-b2430b012b97-kube-api-access-5l67d" pod "3ce4cf66-9b27-4031-aa2b-b2430b012b97" (UID: "3ce4cf66-9b27-4031-aa2b-b2430b012b97"). InnerVolumeSpecName "kube-api-access-5l67d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 04:02:11.607918 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:11.607884    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5l67d\" (UniqueName: \"kubernetes.io/projected/3ce4cf66-9b27-4031-aa2b-b2430b012b97-kube-api-access-5l67d\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:02:11.607918 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:11.607910    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ce4cf66-9b27-4031-aa2b-b2430b012b97-maas-api-tls\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:02:11.607918 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:11.607921    4496 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ce4cf66-9b27-4031-aa2b-b2430b012b97-openshift-service-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:02:11.608168 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:11.607931    4496 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ce4cf66-9b27-4031-aa2b-b2430b012b97-metrics-certs\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:02:11.715585 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:11.715495    4496 generic.go:356] "Generic (PLEG): container finished" podID="3ce4cf66-9b27-4031-aa2b-b2430b012b97" containerID="9ff6415704e0df45d4c0db7eb1fcfb7d89cf10b01891d17d917b31d0394807a0" exitCode=0
Nov 02 04:02:11.715585 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:11.715555    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9b7f43-745986bb4b-wc44k"
Nov 02 04:02:11.715585 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:11.715555    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9b7f43-745986bb4b-wc44k" event={"ID":"3ce4cf66-9b27-4031-aa2b-b2430b012b97","Type":"ContainerDied","Data":"9ff6415704e0df45d4c0db7eb1fcfb7d89cf10b01891d17d917b31d0394807a0"}
Nov 02 04:02:11.715882 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:11.715623    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9b7f43-745986bb4b-wc44k" event={"ID":"3ce4cf66-9b27-4031-aa2b-b2430b012b97","Type":"ContainerDied","Data":"e4d858b87b96c01c0dbbe2e6d9982cf9a2a898fd9b329d5079d4b860e32fca32"}
Nov 02 04:02:11.715882 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:11.715657    4496 scope.go:122] "RemoveContainer" containerID="9ff6415704e0df45d4c0db7eb1fcfb7d89cf10b01891d17d917b31d0394807a0"
Nov 02 04:02:11.726597 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:11.726579    4496 scope.go:122] "RemoveContainer" containerID="9ff6415704e0df45d4c0db7eb1fcfb7d89cf10b01891d17d917b31d0394807a0"
Nov 02 04:02:11.726923 ip-10-0-133-163 kubenswrapper[4496]: E1102 04:02:11.726901    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ff6415704e0df45d4c0db7eb1fcfb7d89cf10b01891d17d917b31d0394807a0\": container with ID starting with 9ff6415704e0df45d4c0db7eb1fcfb7d89cf10b01891d17d917b31d0394807a0 not found: ID does not exist" containerID="9ff6415704e0df45d4c0db7eb1fcfb7d89cf10b01891d17d917b31d0394807a0"
Nov 02 04:02:11.726989 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:11.726931    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ff6415704e0df45d4c0db7eb1fcfb7d89cf10b01891d17d917b31d0394807a0"} err="failed to get container status \"9ff6415704e0df45d4c0db7eb1fcfb7d89cf10b01891d17d917b31d0394807a0\": rpc error: code = NotFound desc = could not find container \"9ff6415704e0df45d4c0db7eb1fcfb7d89cf10b01891d17d917b31d0394807a0\": container with ID starting with 9ff6415704e0df45d4c0db7eb1fcfb7d89cf10b01891d17d917b31d0394807a0 not found: ID does not exist"
Nov 02 04:02:11.884957 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:11.884915    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-9b7f43-745986bb4b-wc44k"]
Nov 02 04:02:11.888305 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:11.888277    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-9b7f43-745986bb4b-wc44k"]
Nov 02 04:02:13.574869 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:13.574835    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ce4cf66-9b27-4031-aa2b-b2430b012b97" path="/var/lib/kubelet/pods/3ce4cf66-9b27-4031-aa2b-b2430b012b97/volumes"
Nov 02 04:02:33.488046 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:33.487995    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-ee0fed-c75c4d6c9-fbd9s"]
Nov 02 04:02:33.527078 ip-10-0-133-163 kubenswrapper[4496]: E1102 04:02:33.527043    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 04:02:33.595615 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:33.595517    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ee0fed-66c7bc7579-6b2zf"]
Nov 02 04:02:37.714597 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:37.714556    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ee0fed-58b9d75758-2smtj"]
Nov 02 04:02:37.715055 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:37.714824    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ee0fed-58b9d75758-2smtj" podUID="9c4dbc8c-dcaf-4864-bcd7-0a8346d04d5b" containerName="maas-api" containerID="cri-o://df8ee3a621836d426c45a7bce3b3e70c7b1ea4384a54b72b5dd1084b9a1a66ca" gracePeriod=30
Nov 02 04:02:37.917475 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:37.917439    4496 generic.go:356] "Generic (PLEG): container finished" podID="9c4dbc8c-dcaf-4864-bcd7-0a8346d04d5b" containerID="df8ee3a621836d426c45a7bce3b3e70c7b1ea4384a54b72b5dd1084b9a1a66ca" exitCode=0
Nov 02 04:02:37.917638 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:37.917522    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ee0fed-58b9d75758-2smtj" event={"ID":"9c4dbc8c-dcaf-4864-bcd7-0a8346d04d5b","Type":"ContainerDied","Data":"df8ee3a621836d426c45a7bce3b3e70c7b1ea4384a54b72b5dd1084b9a1a66ca"}
Nov 02 04:02:37.984100 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:37.984075    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ee0fed-58b9d75758-2smtj"
Nov 02 04:02:38.115528 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:38.115497    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c4dbc8c-dcaf-4864-bcd7-0a8346d04d5b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c4dbc8c-dcaf-4864-bcd7-0a8346d04d5b-openshift-service-ca\") pod \"9c4dbc8c-dcaf-4864-bcd7-0a8346d04d5b\" (UID: \"9c4dbc8c-dcaf-4864-bcd7-0a8346d04d5b\") "
Nov 02 04:02:38.115720 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:38.115543    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c4dbc8c-dcaf-4864-bcd7-0a8346d04d5b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c4dbc8c-dcaf-4864-bcd7-0a8346d04d5b-metrics-certs\") pod \"9c4dbc8c-dcaf-4864-bcd7-0a8346d04d5b\" (UID: \"9c4dbc8c-dcaf-4864-bcd7-0a8346d04d5b\") "
Nov 02 04:02:38.115720 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:38.115573    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c4dbc8c-dcaf-4864-bcd7-0a8346d04d5b-kube-api-access-5cqmb\" (UniqueName: \"kubernetes.io/projected/9c4dbc8c-dcaf-4864-bcd7-0a8346d04d5b-kube-api-access-5cqmb\") pod \"9c4dbc8c-dcaf-4864-bcd7-0a8346d04d5b\" (UID: \"9c4dbc8c-dcaf-4864-bcd7-0a8346d04d5b\") "
Nov 02 04:02:38.115865 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:38.115799    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c4dbc8c-dcaf-4864-bcd7-0a8346d04d5b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c4dbc8c-dcaf-4864-bcd7-0a8346d04d5b-maas-api-tls\") pod \"9c4dbc8c-dcaf-4864-bcd7-0a8346d04d5b\" (UID: \"9c4dbc8c-dcaf-4864-bcd7-0a8346d04d5b\") "
Nov 02 04:02:38.115929 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:38.115906    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c4dbc8c-dcaf-4864-bcd7-0a8346d04d5b-openshift-service-ca" pod "9c4dbc8c-dcaf-4864-bcd7-0a8346d04d5b" (UID: "9c4dbc8c-dcaf-4864-bcd7-0a8346d04d5b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 04:02:38.116245 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:38.116222    4496 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c4dbc8c-dcaf-4864-bcd7-0a8346d04d5b-openshift-service-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:02:38.118618 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:38.118589    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c4dbc8c-dcaf-4864-bcd7-0a8346d04d5b-metrics-certs" pod "9c4dbc8c-dcaf-4864-bcd7-0a8346d04d5b" (UID: "9c4dbc8c-dcaf-4864-bcd7-0a8346d04d5b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 04:02:38.118788 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:38.118766    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c4dbc8c-dcaf-4864-bcd7-0a8346d04d5b-kube-api-access-5cqmb" pod "9c4dbc8c-dcaf-4864-bcd7-0a8346d04d5b" (UID: "9c4dbc8c-dcaf-4864-bcd7-0a8346d04d5b"). InnerVolumeSpecName "kube-api-access-5cqmb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 04:02:38.122173 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:38.122145    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c4dbc8c-dcaf-4864-bcd7-0a8346d04d5b-maas-api-tls" pod "9c4dbc8c-dcaf-4864-bcd7-0a8346d04d5b" (UID: "9c4dbc8c-dcaf-4864-bcd7-0a8346d04d5b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 04:02:38.217666 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:38.217624    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c4dbc8c-dcaf-4864-bcd7-0a8346d04d5b-maas-api-tls\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:02:38.217666 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:38.217657    4496 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c4dbc8c-dcaf-4864-bcd7-0a8346d04d5b-metrics-certs\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:02:38.217874 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:38.217671    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cqmb\" (UniqueName: \"kubernetes.io/projected/9c4dbc8c-dcaf-4864-bcd7-0a8346d04d5b-kube-api-access-5cqmb\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:02:38.489091 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:38.489055    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ee0fed-c75c4d6c9-fbd9s" podUID="9887dae0-cba1-4148-bce6-936ff305a1a9" containerName="payload-processing" containerID="cri-o://f67649bdc1ce376990d78ef0569898b047ef25db7d1e55cc7797cc1def170f5b" gracePeriod=25
Nov 02 04:02:38.601039 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:38.600571    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ee0fed-66c7bc7579-6b2zf" podUID="1955f7a8-3b12-45cd-8059-7d701db7ac37" containerName="payload-pre-processing" containerID="cri-o://016ed7e6f9aab48ab13f89b14adc5e8bc05d6f92fff4e21d68b86fd473c48e3e" gracePeriod=25
Nov 02 04:02:38.926116 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:38.926010    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ee0fed-58b9d75758-2smtj"
Nov 02 04:02:38.926116 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:38.926044    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ee0fed-58b9d75758-2smtj" event={"ID":"9c4dbc8c-dcaf-4864-bcd7-0a8346d04d5b","Type":"ContainerDied","Data":"f5e3f360fa25855ede3c9ed9e03a78df24cacb88a36bf1682d33c5ae1d86730c"}
Nov 02 04:02:38.926638 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:38.926150    4496 scope.go:122] "RemoveContainer" containerID="df8ee3a621836d426c45a7bce3b3e70c7b1ea4384a54b72b5dd1084b9a1a66ca"
Nov 02 04:02:39.103136 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:39.103095    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ee0fed-58b9d75758-2smtj"]
Nov 02 04:02:39.154039 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:39.153996    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ee0fed-58b9d75758-2smtj"]
Nov 02 04:02:39.574762 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:39.574725    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c4dbc8c-dcaf-4864-bcd7-0a8346d04d5b" path="/var/lib/kubelet/pods/9c4dbc8c-dcaf-4864-bcd7-0a8346d04d5b/volumes"
Nov 02 04:02:40.491342 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:40.491311    4496 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hww8q_5c797b08-a314-44ed-8026-8a0ce1f6c36f/ovn-acl-logging/0.log"
Nov 02 04:02:40.507657 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:40.507633    4496 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hww8q_5c797b08-a314-44ed-8026-8a0ce1f6c36f/ovn-acl-logging/0.log"
Nov 02 04:02:44.818891 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:44.818856    4496 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ee0fed-66c7bc7579-6b2zf" podUID="1955f7a8-3b12-45cd-8059-7d701db7ac37" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 04:02:44.819401 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:44.818849    4496 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ee0fed-c75c4d6c9-fbd9s" podUID="9887dae0-cba1-4148-bce6-936ff305a1a9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 04:02:48.999447 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:48.999411    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-ae1315-695b4db8d5-hwgpm"]
Nov 02 04:02:48.999882 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:48.999639    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ae1315-695b4db8d5-hwgpm" podUID="fc4e9a96-bcbe-4993-9509-3987dcedff81" containerName="maas-api" containerID="cri-o://3da6267f8c6d3f0f95deddaa93f6fb9ac576afcb159ae457a5357796c9f23f9a" gracePeriod=30
Nov 02 04:02:49.266935 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:49.266872    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ae1315-695b4db8d5-hwgpm"
Nov 02 04:02:49.347778 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:49.347742    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc4e9a96-bcbe-4993-9509-3987dcedff81-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc4e9a96-bcbe-4993-9509-3987dcedff81-openshift-service-ca\") pod \"fc4e9a96-bcbe-4993-9509-3987dcedff81\" (UID: \"fc4e9a96-bcbe-4993-9509-3987dcedff81\") "
Nov 02 04:02:49.347964 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:49.347936    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc4e9a96-bcbe-4993-9509-3987dcedff81-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc4e9a96-bcbe-4993-9509-3987dcedff81-maas-api-tls\") pod \"fc4e9a96-bcbe-4993-9509-3987dcedff81\" (UID: \"fc4e9a96-bcbe-4993-9509-3987dcedff81\") "
Nov 02 04:02:49.348106 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:49.348087    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc4e9a96-bcbe-4993-9509-3987dcedff81-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc4e9a96-bcbe-4993-9509-3987dcedff81-metrics-certs\") pod \"fc4e9a96-bcbe-4993-9509-3987dcedff81\" (UID: \"fc4e9a96-bcbe-4993-9509-3987dcedff81\") "
Nov 02 04:02:49.348156 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:49.348126    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc4e9a96-bcbe-4993-9509-3987dcedff81-openshift-service-ca" pod "fc4e9a96-bcbe-4993-9509-3987dcedff81" (UID: "fc4e9a96-bcbe-4993-9509-3987dcedff81"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 04:02:49.348194 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:49.348136    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc4e9a96-bcbe-4993-9509-3987dcedff81-kube-api-access-gpctr\" (UniqueName: \"kubernetes.io/projected/fc4e9a96-bcbe-4993-9509-3987dcedff81-kube-api-access-gpctr\") pod \"fc4e9a96-bcbe-4993-9509-3987dcedff81\" (UID: \"fc4e9a96-bcbe-4993-9509-3987dcedff81\") "
Nov 02 04:02:49.348538 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:49.348516    4496 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc4e9a96-bcbe-4993-9509-3987dcedff81-openshift-service-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:02:49.350195 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:49.350170    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc4e9a96-bcbe-4993-9509-3987dcedff81-metrics-certs" pod "fc4e9a96-bcbe-4993-9509-3987dcedff81" (UID: "fc4e9a96-bcbe-4993-9509-3987dcedff81"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 04:02:49.350295 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:49.350210    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc4e9a96-bcbe-4993-9509-3987dcedff81-maas-api-tls" pod "fc4e9a96-bcbe-4993-9509-3987dcedff81" (UID: "fc4e9a96-bcbe-4993-9509-3987dcedff81"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 04:02:49.350295 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:49.350285    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc4e9a96-bcbe-4993-9509-3987dcedff81-kube-api-access-gpctr" pod "fc4e9a96-bcbe-4993-9509-3987dcedff81" (UID: "fc4e9a96-bcbe-4993-9509-3987dcedff81"). InnerVolumeSpecName "kube-api-access-gpctr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 04:02:49.449692 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:49.449658    4496 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc4e9a96-bcbe-4993-9509-3987dcedff81-metrics-certs\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:02:49.449692 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:49.449689    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpctr\" (UniqueName: \"kubernetes.io/projected/fc4e9a96-bcbe-4993-9509-3987dcedff81-kube-api-access-gpctr\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:02:49.449692 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:49.449700    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc4e9a96-bcbe-4993-9509-3987dcedff81-maas-api-tls\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:02:50.011357 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:50.011318    4496 generic.go:356] "Generic (PLEG): container finished" podID="fc4e9a96-bcbe-4993-9509-3987dcedff81" containerID="3da6267f8c6d3f0f95deddaa93f6fb9ac576afcb159ae457a5357796c9f23f9a" exitCode=0
Nov 02 04:02:50.011756 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:50.011388    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ae1315-695b4db8d5-hwgpm"
Nov 02 04:02:50.011756 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:50.011398    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ae1315-695b4db8d5-hwgpm" event={"ID":"fc4e9a96-bcbe-4993-9509-3987dcedff81","Type":"ContainerDied","Data":"3da6267f8c6d3f0f95deddaa93f6fb9ac576afcb159ae457a5357796c9f23f9a"}
Nov 02 04:02:50.011756 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:50.011454    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ae1315-695b4db8d5-hwgpm" event={"ID":"fc4e9a96-bcbe-4993-9509-3987dcedff81","Type":"ContainerDied","Data":"82ba66e84a71bcf63760f1421c13b6355dbb5adb529e5208734596e2f64f4c43"}
Nov 02 04:02:50.011756 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:50.011476    4496 scope.go:122] "RemoveContainer" containerID="3da6267f8c6d3f0f95deddaa93f6fb9ac576afcb159ae457a5357796c9f23f9a"
Nov 02 04:02:50.021645 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:50.021623    4496 scope.go:122] "RemoveContainer" containerID="3da6267f8c6d3f0f95deddaa93f6fb9ac576afcb159ae457a5357796c9f23f9a"
Nov 02 04:02:50.021886 ip-10-0-133-163 kubenswrapper[4496]: E1102 04:02:50.021866    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3da6267f8c6d3f0f95deddaa93f6fb9ac576afcb159ae457a5357796c9f23f9a\": container with ID starting with 3da6267f8c6d3f0f95deddaa93f6fb9ac576afcb159ae457a5357796c9f23f9a not found: ID does not exist" containerID="3da6267f8c6d3f0f95deddaa93f6fb9ac576afcb159ae457a5357796c9f23f9a"
Nov 02 04:02:50.021937 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:50.021893    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3da6267f8c6d3f0f95deddaa93f6fb9ac576afcb159ae457a5357796c9f23f9a"} err="failed to get container status \"3da6267f8c6d3f0f95deddaa93f6fb9ac576afcb159ae457a5357796c9f23f9a\": rpc error: code = NotFound desc = could not find container \"3da6267f8c6d3f0f95deddaa93f6fb9ac576afcb159ae457a5357796c9f23f9a\": container with ID starting with 3da6267f8c6d3f0f95deddaa93f6fb9ac576afcb159ae457a5357796c9f23f9a not found: ID does not exist"
Nov 02 04:02:50.178852 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:50.178818    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-ae1315-695b4db8d5-hwgpm"]
Nov 02 04:02:50.229274 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:50.229239    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-ae1315-695b4db8d5-hwgpm"]
Nov 02 04:02:51.575600 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:51.575564    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc4e9a96-bcbe-4993-9509-3987dcedff81" path="/var/lib/kubelet/pods/fc4e9a96-bcbe-4993-9509-3987dcedff81/volumes"
Nov 02 04:02:54.818299 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:54.818261    4496 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ee0fed-c75c4d6c9-fbd9s" podUID="9887dae0-cba1-4148-bce6-936ff305a1a9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 04:02:54.818790 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:54.818261    4496 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ee0fed-66c7bc7579-6b2zf" podUID="1955f7a8-3b12-45cd-8059-7d701db7ac37" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 04:02:58.772926 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:58.772892    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ee0fed-c75c4d6c9-fbd9s"
Nov 02 04:02:58.860653 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:58.860629    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ee0fed-66c7bc7579-6b2zf"
Nov 02 04:02:58.863648 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:58.863627    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9887dae0-cba1-4148-bce6-936ff305a1a9-kube-api-access-jl57p\" (UniqueName: \"kubernetes.io/projected/9887dae0-cba1-4148-bce6-936ff305a1a9-kube-api-access-jl57p\") pod \"9887dae0-cba1-4148-bce6-936ff305a1a9\" (UID: \"9887dae0-cba1-4148-bce6-936ff305a1a9\") "
Nov 02 04:02:58.863786 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:58.863766    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9887dae0-cba1-4148-bce6-936ff305a1a9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9887dae0-cba1-4148-bce6-936ff305a1a9-plugins-config-volume\") pod \"9887dae0-cba1-4148-bce6-936ff305a1a9\" (UID: \"9887dae0-cba1-4148-bce6-936ff305a1a9\") "
Nov 02 04:02:58.864125 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:58.864104    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9887dae0-cba1-4148-bce6-936ff305a1a9-plugins-config-volume" pod "9887dae0-cba1-4148-bce6-936ff305a1a9" (UID: "9887dae0-cba1-4148-bce6-936ff305a1a9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 04:02:58.865785 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:58.865760    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9887dae0-cba1-4148-bce6-936ff305a1a9-kube-api-access-jl57p" pod "9887dae0-cba1-4148-bce6-936ff305a1a9" (UID: "9887dae0-cba1-4148-bce6-936ff305a1a9"). InnerVolumeSpecName "kube-api-access-jl57p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 04:02:58.964845 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:58.964814    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1955f7a8-3b12-45cd-8059-7d701db7ac37-kube-api-access-9mmkm\" (UniqueName: \"kubernetes.io/projected/1955f7a8-3b12-45cd-8059-7d701db7ac37-kube-api-access-9mmkm\") pod \"1955f7a8-3b12-45cd-8059-7d701db7ac37\" (UID: \"1955f7a8-3b12-45cd-8059-7d701db7ac37\") "
Nov 02 04:02:58.965064 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:58.964855    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1955f7a8-3b12-45cd-8059-7d701db7ac37-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1955f7a8-3b12-45cd-8059-7d701db7ac37-plugins-config-volume\") pod \"1955f7a8-3b12-45cd-8059-7d701db7ac37\" (UID: \"1955f7a8-3b12-45cd-8059-7d701db7ac37\") "
Nov 02 04:02:58.965359 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:58.965325    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1955f7a8-3b12-45cd-8059-7d701db7ac37-plugins-config-volume" pod "1955f7a8-3b12-45cd-8059-7d701db7ac37" (UID: "1955f7a8-3b12-45cd-8059-7d701db7ac37"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 04:02:58.965482 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:58.965408    4496 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9887dae0-cba1-4148-bce6-936ff305a1a9-plugins-config-volume\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:02:58.965482 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:58.965430    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jl57p\" (UniqueName: \"kubernetes.io/projected/9887dae0-cba1-4148-bce6-936ff305a1a9-kube-api-access-jl57p\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:02:58.967588 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:58.967568    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1955f7a8-3b12-45cd-8059-7d701db7ac37-kube-api-access-9mmkm" pod "1955f7a8-3b12-45cd-8059-7d701db7ac37" (UID: "1955f7a8-3b12-45cd-8059-7d701db7ac37"). InnerVolumeSpecName "kube-api-access-9mmkm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 04:02:59.066803 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:59.066721    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mmkm\" (UniqueName: \"kubernetes.io/projected/1955f7a8-3b12-45cd-8059-7d701db7ac37-kube-api-access-9mmkm\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:02:59.066803 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:59.066760    4496 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1955f7a8-3b12-45cd-8059-7d701db7ac37-plugins-config-volume\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:02:59.083833 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:59.083796    4496 generic.go:356] "Generic (PLEG): container finished" podID="1955f7a8-3b12-45cd-8059-7d701db7ac37" containerID="016ed7e6f9aab48ab13f89b14adc5e8bc05d6f92fff4e21d68b86fd473c48e3e" exitCode=0
Nov 02 04:02:59.083991 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:59.083841    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ee0fed-66c7bc7579-6b2zf"
Nov 02 04:02:59.083991 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:59.083878    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ee0fed-66c7bc7579-6b2zf" event={"ID":"1955f7a8-3b12-45cd-8059-7d701db7ac37","Type":"ContainerDied","Data":"016ed7e6f9aab48ab13f89b14adc5e8bc05d6f92fff4e21d68b86fd473c48e3e"}
Nov 02 04:02:59.083991 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:59.083968    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ee0fed-66c7bc7579-6b2zf" event={"ID":"1955f7a8-3b12-45cd-8059-7d701db7ac37","Type":"ContainerDied","Data":"5c302a0091a677bc658523fd3bb14bbea1f7507ffe1767af5a7fadd2b2de0545"}
Nov 02 04:02:59.084187 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:59.083996    4496 scope.go:122] "RemoveContainer" containerID="016ed7e6f9aab48ab13f89b14adc5e8bc05d6f92fff4e21d68b86fd473c48e3e"
Nov 02 04:02:59.085776 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:59.085742    4496 generic.go:356] "Generic (PLEG): container finished" podID="9887dae0-cba1-4148-bce6-936ff305a1a9" containerID="f67649bdc1ce376990d78ef0569898b047ef25db7d1e55cc7797cc1def170f5b" exitCode=0
Nov 02 04:02:59.085903 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:59.085782    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ee0fed-c75c4d6c9-fbd9s"
Nov 02 04:02:59.085903 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:59.085773    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ee0fed-c75c4d6c9-fbd9s" event={"ID":"9887dae0-cba1-4148-bce6-936ff305a1a9","Type":"ContainerDied","Data":"f67649bdc1ce376990d78ef0569898b047ef25db7d1e55cc7797cc1def170f5b"}
Nov 02 04:02:59.086066 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:59.085898    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ee0fed-c75c4d6c9-fbd9s" event={"ID":"9887dae0-cba1-4148-bce6-936ff305a1a9","Type":"ContainerDied","Data":"9aea61bbe1fdf2aea02c4481fa446d916ca7b2970326f04c8e74bab72752f8f3"}
Nov 02 04:02:59.095712 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:59.095616    4496 scope.go:122] "RemoveContainer" containerID="016ed7e6f9aab48ab13f89b14adc5e8bc05d6f92fff4e21d68b86fd473c48e3e"
Nov 02 04:02:59.095988 ip-10-0-133-163 kubenswrapper[4496]: E1102 04:02:59.095965    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"016ed7e6f9aab48ab13f89b14adc5e8bc05d6f92fff4e21d68b86fd473c48e3e\": container with ID starting with 016ed7e6f9aab48ab13f89b14adc5e8bc05d6f92fff4e21d68b86fd473c48e3e not found: ID does not exist" containerID="016ed7e6f9aab48ab13f89b14adc5e8bc05d6f92fff4e21d68b86fd473c48e3e"
Nov 02 04:02:59.096102 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:59.095999    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"016ed7e6f9aab48ab13f89b14adc5e8bc05d6f92fff4e21d68b86fd473c48e3e"} err="failed to get container status \"016ed7e6f9aab48ab13f89b14adc5e8bc05d6f92fff4e21d68b86fd473c48e3e\": rpc error: code = NotFound desc = could not find container \"016ed7e6f9aab48ab13f89b14adc5e8bc05d6f92fff4e21d68b86fd473c48e3e\": container with ID starting with 016ed7e6f9aab48ab13f89b14adc5e8bc05d6f92fff4e21d68b86fd473c48e3e not found: ID does not exist"
Nov 02 04:02:59.096102 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:59.096038    4496 scope.go:122] "RemoveContainer" containerID="f67649bdc1ce376990d78ef0569898b047ef25db7d1e55cc7797cc1def170f5b"
Nov 02 04:02:59.105848 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:59.105831    4496 scope.go:122] "RemoveContainer" containerID="f67649bdc1ce376990d78ef0569898b047ef25db7d1e55cc7797cc1def170f5b"
Nov 02 04:02:59.106120 ip-10-0-133-163 kubenswrapper[4496]: E1102 04:02:59.106099    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f67649bdc1ce376990d78ef0569898b047ef25db7d1e55cc7797cc1def170f5b\": container with ID starting with f67649bdc1ce376990d78ef0569898b047ef25db7d1e55cc7797cc1def170f5b not found: ID does not exist" containerID="f67649bdc1ce376990d78ef0569898b047ef25db7d1e55cc7797cc1def170f5b"
Nov 02 04:02:59.106170 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:59.106130    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f67649bdc1ce376990d78ef0569898b047ef25db7d1e55cc7797cc1def170f5b"} err="failed to get container status \"f67649bdc1ce376990d78ef0569898b047ef25db7d1e55cc7797cc1def170f5b\": rpc error: code = NotFound desc = could not find container \"f67649bdc1ce376990d78ef0569898b047ef25db7d1e55cc7797cc1def170f5b\": container with ID starting with f67649bdc1ce376990d78ef0569898b047ef25db7d1e55cc7797cc1def170f5b not found: ID does not exist"
Nov 02 04:02:59.253315 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:59.253280    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-ee0fed-c75c4d6c9-fbd9s"]
Nov 02 04:02:59.260247 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:59.260215    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-ee0fed-c75c4d6c9-fbd9s"]
Nov 02 04:02:59.375156 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:59.375066    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ee0fed-66c7bc7579-6b2zf"]
Nov 02 04:02:59.379214 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:59.379182    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ee0fed-66c7bc7579-6b2zf"]
Nov 02 04:02:59.576289 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:59.576254    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1955f7a8-3b12-45cd-8059-7d701db7ac37" path="/var/lib/kubelet/pods/1955f7a8-3b12-45cd-8059-7d701db7ac37/volumes"
Nov 02 04:02:59.576768 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:02:59.576743    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9887dae0-cba1-4148-bce6-936ff305a1a9" path="/var/lib/kubelet/pods/9887dae0-cba1-4148-bce6-936ff305a1a9/volumes"
Nov 02 04:03:03.528536 ip-10-0-133-163 kubenswrapper[4496]: E1102 04:03:03.528498    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 04:03:13.483704 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:13.483666    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-a7cf93-859f5d856c-qgchk"]
Nov 02 04:03:13.591995 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:13.591943    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a7cf93-7b78677459-fvrpv"]
Nov 02 04:03:18.484553 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:18.484500    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-a7cf93-859f5d856c-qgchk" podUID="ab5ecfc3-592a-47ca-8acd-000d982544d5" containerName="payload-processing" containerID="cri-o://26fea63cd9c0d4c790fc0e2f4cabc35cfd11907f32dc291eb15159ada15992f3" gracePeriod=25
Nov 02 04:03:18.593220 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:18.593138    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a7cf93-7b78677459-fvrpv" podUID="8ea63372-147c-486f-99d4-5d2a5390db50" containerName="payload-pre-processing" containerID="cri-o://90c9b4d908742ce3d2ba56515696ae5d598541b0e6fe318587bead83fc34efff" gracePeriod=25
Nov 02 04:03:19.861256 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:19.861210    4496 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-a7cf93-859f5d856c-qgchk" podUID="ab5ecfc3-592a-47ca-8acd-000d982544d5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 04:03:19.861689 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:19.861273    4496 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a7cf93-7b78677459-fvrpv" podUID="8ea63372-147c-486f-99d4-5d2a5390db50" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 04:03:29.862287 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:29.862182    4496 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a7cf93-7b78677459-fvrpv" podUID="8ea63372-147c-486f-99d4-5d2a5390db50" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 04:03:29.862287 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:29.862267    4496 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-a7cf93-859f5d856c-qgchk" podUID="ab5ecfc3-592a-47ca-8acd-000d982544d5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 04:03:33.527131 ip-10-0-133-163 kubenswrapper[4496]: E1102 04:03:33.527094    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 04:03:38.827093 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:38.827061    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-a7cf93-859f5d856c-qgchk"
Nov 02 04:03:38.862187 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:38.862162    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a7cf93-7b78677459-fvrpv"
Nov 02 04:03:38.931596 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:38.931560    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab5ecfc3-592a-47ca-8acd-000d982544d5-kube-api-access-rb89v\" (UniqueName: \"kubernetes.io/projected/ab5ecfc3-592a-47ca-8acd-000d982544d5-kube-api-access-rb89v\") pod \"ab5ecfc3-592a-47ca-8acd-000d982544d5\" (UID: \"ab5ecfc3-592a-47ca-8acd-000d982544d5\") "
Nov 02 04:03:38.931783 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:38.931763    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab5ecfc3-592a-47ca-8acd-000d982544d5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab5ecfc3-592a-47ca-8acd-000d982544d5-plugins-config-volume\") pod \"ab5ecfc3-592a-47ca-8acd-000d982544d5\" (UID: \"ab5ecfc3-592a-47ca-8acd-000d982544d5\") "
Nov 02 04:03:38.932143 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:38.932122    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab5ecfc3-592a-47ca-8acd-000d982544d5-plugins-config-volume" pod "ab5ecfc3-592a-47ca-8acd-000d982544d5" (UID: "ab5ecfc3-592a-47ca-8acd-000d982544d5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 04:03:38.932260 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:38.932239    4496 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab5ecfc3-592a-47ca-8acd-000d982544d5-plugins-config-volume\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:03:38.933784 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:38.933759    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab5ecfc3-592a-47ca-8acd-000d982544d5-kube-api-access-rb89v" pod "ab5ecfc3-592a-47ca-8acd-000d982544d5" (UID: "ab5ecfc3-592a-47ca-8acd-000d982544d5"). InnerVolumeSpecName "kube-api-access-rb89v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 04:03:39.033430 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:39.033341    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ea63372-147c-486f-99d4-5d2a5390db50-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ea63372-147c-486f-99d4-5d2a5390db50-plugins-config-volume\") pod \"8ea63372-147c-486f-99d4-5d2a5390db50\" (UID: \"8ea63372-147c-486f-99d4-5d2a5390db50\") "
Nov 02 04:03:39.033613 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:39.033439    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ea63372-147c-486f-99d4-5d2a5390db50-kube-api-access-n9lsx\" (UniqueName: \"kubernetes.io/projected/8ea63372-147c-486f-99d4-5d2a5390db50-kube-api-access-n9lsx\") pod \"8ea63372-147c-486f-99d4-5d2a5390db50\" (UID: \"8ea63372-147c-486f-99d4-5d2a5390db50\") "
Nov 02 04:03:39.033724 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:39.033701    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ea63372-147c-486f-99d4-5d2a5390db50-plugins-config-volume" pod "8ea63372-147c-486f-99d4-5d2a5390db50" (UID: "8ea63372-147c-486f-99d4-5d2a5390db50"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 04:03:39.033870 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:39.033856    4496 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ea63372-147c-486f-99d4-5d2a5390db50-plugins-config-volume\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:03:39.033937 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:39.033872    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rb89v\" (UniqueName: \"kubernetes.io/projected/ab5ecfc3-592a-47ca-8acd-000d982544d5-kube-api-access-rb89v\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:03:39.036088 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:39.036058    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ea63372-147c-486f-99d4-5d2a5390db50-kube-api-access-n9lsx" pod "8ea63372-147c-486f-99d4-5d2a5390db50" (UID: "8ea63372-147c-486f-99d4-5d2a5390db50"). InnerVolumeSpecName "kube-api-access-n9lsx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 04:03:39.135158 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:39.135124    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9lsx\" (UniqueName: \"kubernetes.io/projected/8ea63372-147c-486f-99d4-5d2a5390db50-kube-api-access-n9lsx\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:03:39.417766 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:39.417671    4496 generic.go:356] "Generic (PLEG): container finished" podID="ab5ecfc3-592a-47ca-8acd-000d982544d5" containerID="26fea63cd9c0d4c790fc0e2f4cabc35cfd11907f32dc291eb15159ada15992f3" exitCode=0
Nov 02 04:03:39.417766 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:39.417715    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-a7cf93-859f5d856c-qgchk"
Nov 02 04:03:39.417766 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:39.417755    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-a7cf93-859f5d856c-qgchk" event={"ID":"ab5ecfc3-592a-47ca-8acd-000d982544d5","Type":"ContainerDied","Data":"26fea63cd9c0d4c790fc0e2f4cabc35cfd11907f32dc291eb15159ada15992f3"}
Nov 02 04:03:39.418097 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:39.417838    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-a7cf93-859f5d856c-qgchk" event={"ID":"ab5ecfc3-592a-47ca-8acd-000d982544d5","Type":"ContainerDied","Data":"b8c178eac4a11f3cf48d4b852928b5a5d4001bf22962eb913d68d80a97765aea"}
Nov 02 04:03:39.418097 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:39.417871    4496 scope.go:122] "RemoveContainer" containerID="26fea63cd9c0d4c790fc0e2f4cabc35cfd11907f32dc291eb15159ada15992f3"
Nov 02 04:03:39.421566 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:39.421533    4496 generic.go:356] "Generic (PLEG): container finished" podID="8ea63372-147c-486f-99d4-5d2a5390db50" containerID="90c9b4d908742ce3d2ba56515696ae5d598541b0e6fe318587bead83fc34efff" exitCode=0
Nov 02 04:03:39.421712 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:39.421597    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a7cf93-7b78677459-fvrpv" event={"ID":"8ea63372-147c-486f-99d4-5d2a5390db50","Type":"ContainerDied","Data":"90c9b4d908742ce3d2ba56515696ae5d598541b0e6fe318587bead83fc34efff"}
Nov 02 04:03:39.421712 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:39.421613    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a7cf93-7b78677459-fvrpv"
Nov 02 04:03:39.421712 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:39.421638    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a7cf93-7b78677459-fvrpv" event={"ID":"8ea63372-147c-486f-99d4-5d2a5390db50","Type":"ContainerDied","Data":"a227278b5878dd053e4e454021268e845a9851de77b382f513aec54fa930bf01"}
Nov 02 04:03:39.432710 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:39.432692    4496 scope.go:122] "RemoveContainer" containerID="26fea63cd9c0d4c790fc0e2f4cabc35cfd11907f32dc291eb15159ada15992f3"
Nov 02 04:03:39.432998 ip-10-0-133-163 kubenswrapper[4496]: E1102 04:03:39.432977    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26fea63cd9c0d4c790fc0e2f4cabc35cfd11907f32dc291eb15159ada15992f3\": container with ID starting with 26fea63cd9c0d4c790fc0e2f4cabc35cfd11907f32dc291eb15159ada15992f3 not found: ID does not exist" containerID="26fea63cd9c0d4c790fc0e2f4cabc35cfd11907f32dc291eb15159ada15992f3"
Nov 02 04:03:39.433086 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:39.433008    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26fea63cd9c0d4c790fc0e2f4cabc35cfd11907f32dc291eb15159ada15992f3"} err="failed to get container status \"26fea63cd9c0d4c790fc0e2f4cabc35cfd11907f32dc291eb15159ada15992f3\": rpc error: code = NotFound desc = could not find container \"26fea63cd9c0d4c790fc0e2f4cabc35cfd11907f32dc291eb15159ada15992f3\": container with ID starting with 26fea63cd9c0d4c790fc0e2f4cabc35cfd11907f32dc291eb15159ada15992f3 not found: ID does not exist"
Nov 02 04:03:39.433086 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:39.433046    4496 scope.go:122] "RemoveContainer" containerID="90c9b4d908742ce3d2ba56515696ae5d598541b0e6fe318587bead83fc34efff"
Nov 02 04:03:39.442240 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:39.442220    4496 scope.go:122] "RemoveContainer" containerID="90c9b4d908742ce3d2ba56515696ae5d598541b0e6fe318587bead83fc34efff"
Nov 02 04:03:39.442497 ip-10-0-133-163 kubenswrapper[4496]: E1102 04:03:39.442480    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90c9b4d908742ce3d2ba56515696ae5d598541b0e6fe318587bead83fc34efff\": container with ID starting with 90c9b4d908742ce3d2ba56515696ae5d598541b0e6fe318587bead83fc34efff not found: ID does not exist" containerID="90c9b4d908742ce3d2ba56515696ae5d598541b0e6fe318587bead83fc34efff"
Nov 02 04:03:39.442554 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:39.442503    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90c9b4d908742ce3d2ba56515696ae5d598541b0e6fe318587bead83fc34efff"} err="failed to get container status \"90c9b4d908742ce3d2ba56515696ae5d598541b0e6fe318587bead83fc34efff\": rpc error: code = NotFound desc = could not find container \"90c9b4d908742ce3d2ba56515696ae5d598541b0e6fe318587bead83fc34efff\": container with ID starting with 90c9b4d908742ce3d2ba56515696ae5d598541b0e6fe318587bead83fc34efff not found: ID does not exist"
Nov 02 04:03:39.587443 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:39.587413    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-a7cf93-859f5d856c-qgchk"]
Nov 02 04:03:39.590473 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:39.590450    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-a7cf93-859f5d856c-qgchk"]
Nov 02 04:03:39.699881 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:39.699792    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a7cf93-7b78677459-fvrpv"]
Nov 02 04:03:39.704198 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:39.704169    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a7cf93-7b78677459-fvrpv"]
Nov 02 04:03:41.577772 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:41.577712    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ea63372-147c-486f-99d4-5d2a5390db50" path="/var/lib/kubelet/pods/8ea63372-147c-486f-99d4-5d2a5390db50/volumes"
Nov 02 04:03:41.578393 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:41.578371    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab5ecfc3-592a-47ca-8acd-000d982544d5" path="/var/lib/kubelet/pods/ab5ecfc3-592a-47ca-8acd-000d982544d5/volumes"
Nov 02 04:03:47.007009 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:47.006974    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-7c1bee-84b77485bf-9bk69"]
Nov 02 04:03:47.115140 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:47.115097    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7c1bee-694764dc5-zrthf"]
Nov 02 04:03:52.007555 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:52.007511    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-7c1bee-84b77485bf-9bk69" podUID="d147ea36-e9b2-42de-ac31-de7c5e03fe51" containerName="payload-processing" containerID="cri-o://033f92905c6e23818d5fd5699de333f7c23cd44452c99a0e5cf3f03f30d8bbd3" gracePeriod=25
Nov 02 04:03:52.116037 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:52.115964    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7c1bee-694764dc5-zrthf" podUID="2046cb9f-964f-45f4-9f8a-e427b5818ffc" containerName="payload-pre-processing" containerID="cri-o://82cb4bb8139daaf6a20153fc19c23fc0243490b51055519ba1344df219fbd0d5" gracePeriod=25
Nov 02 04:03:53.291642 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:53.291603    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a7cf93-696b58646-gp5l7"]
Nov 02 04:03:53.292108 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:53.291863    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a7cf93-696b58646-gp5l7" podUID="44122d78-2a1a-43bb-a123-4494b445d939" containerName="maas-api" containerID="cri-o://85e2a219bd0aa448adc75c57577cc3b0d58403a5accca00b8338a43cc004ae87" gracePeriod=30
Nov 02 04:03:53.542496 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:53.542321    4496 generic.go:356] "Generic (PLEG): container finished" podID="44122d78-2a1a-43bb-a123-4494b445d939" containerID="85e2a219bd0aa448adc75c57577cc3b0d58403a5accca00b8338a43cc004ae87" exitCode=0
Nov 02 04:03:53.542496 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:53.542367    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a7cf93-696b58646-gp5l7" event={"ID":"44122d78-2a1a-43bb-a123-4494b445d939","Type":"ContainerDied","Data":"85e2a219bd0aa448adc75c57577cc3b0d58403a5accca00b8338a43cc004ae87"}
Nov 02 04:03:53.542849 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:53.542543    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a7cf93-696b58646-gp5l7" event={"ID":"44122d78-2a1a-43bb-a123-4494b445d939","Type":"ContainerDied","Data":"f96a5c49665aca6fc9655259916f6b67c9212a9b18255f85342f03dec81d2e04"}
Nov 02 04:03:53.542849 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:53.542763    4496 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f96a5c49665aca6fc9655259916f6b67c9212a9b18255f85342f03dec81d2e04"
Nov 02 04:03:53.566863 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:53.566838    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a7cf93-696b58646-gp5l7"
Nov 02 04:03:53.596903 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:53.596873    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44122d78-2a1a-43bb-a123-4494b445d939-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44122d78-2a1a-43bb-a123-4494b445d939-openshift-service-ca\") pod \"44122d78-2a1a-43bb-a123-4494b445d939\" (UID: \"44122d78-2a1a-43bb-a123-4494b445d939\") "
Nov 02 04:03:53.597074 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:53.596966    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44122d78-2a1a-43bb-a123-4494b445d939-kube-api-access-wb5q8\" (UniqueName: \"kubernetes.io/projected/44122d78-2a1a-43bb-a123-4494b445d939-kube-api-access-wb5q8\") pod \"44122d78-2a1a-43bb-a123-4494b445d939\" (UID: \"44122d78-2a1a-43bb-a123-4494b445d939\") "
Nov 02 04:03:53.597074 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:53.597010    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44122d78-2a1a-43bb-a123-4494b445d939-metrics-certs\" (UniqueName: \"kubernetes.io/secret/44122d78-2a1a-43bb-a123-4494b445d939-metrics-certs\") pod \"44122d78-2a1a-43bb-a123-4494b445d939\" (UID: \"44122d78-2a1a-43bb-a123-4494b445d939\") "
Nov 02 04:03:53.597209 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:53.597093    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44122d78-2a1a-43bb-a123-4494b445d939-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44122d78-2a1a-43bb-a123-4494b445d939-maas-api-tls\") pod \"44122d78-2a1a-43bb-a123-4494b445d939\" (UID: \"44122d78-2a1a-43bb-a123-4494b445d939\") "
Nov 02 04:03:53.597273 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:53.597221    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44122d78-2a1a-43bb-a123-4494b445d939-openshift-service-ca" pod "44122d78-2a1a-43bb-a123-4494b445d939" (UID: "44122d78-2a1a-43bb-a123-4494b445d939"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 04:03:53.597568 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:53.597535    4496 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44122d78-2a1a-43bb-a123-4494b445d939-openshift-service-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:03:53.599188 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:53.599160    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44122d78-2a1a-43bb-a123-4494b445d939-metrics-certs" pod "44122d78-2a1a-43bb-a123-4494b445d939" (UID: "44122d78-2a1a-43bb-a123-4494b445d939"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 04:03:53.599311 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:53.599288    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44122d78-2a1a-43bb-a123-4494b445d939-kube-api-access-wb5q8" pod "44122d78-2a1a-43bb-a123-4494b445d939" (UID: "44122d78-2a1a-43bb-a123-4494b445d939"). InnerVolumeSpecName "kube-api-access-wb5q8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 04:03:53.599360 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:53.599341    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44122d78-2a1a-43bb-a123-4494b445d939-maas-api-tls" pod "44122d78-2a1a-43bb-a123-4494b445d939" (UID: "44122d78-2a1a-43bb-a123-4494b445d939"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 04:03:53.698274 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:53.698239    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wb5q8\" (UniqueName: \"kubernetes.io/projected/44122d78-2a1a-43bb-a123-4494b445d939-kube-api-access-wb5q8\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:03:53.698274 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:53.698276    4496 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44122d78-2a1a-43bb-a123-4494b445d939-metrics-certs\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:03:53.698504 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:53.698290    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44122d78-2a1a-43bb-a123-4494b445d939-maas-api-tls\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:03:54.472758 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:54.472724    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-147104-658844d6c9-899t6"]
Nov 02 04:03:54.473215 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:54.472927    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-147104-658844d6c9-899t6" podUID="7f80f210-8056-4a78-a27f-5166ab50c9d0" containerName="maas-api" containerID="cri-o://1d478ed3c1c0cd05b0f9a93864fb833e8e26bb2d1fd8bc3e1672d638b752d65f" gracePeriod=30
Nov 02 04:03:54.552880 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:54.552842    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a7cf93-696b58646-gp5l7"
Nov 02 04:03:54.712861 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:54.712821    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a7cf93-696b58646-gp5l7"]
Nov 02 04:03:54.719483 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:54.719448    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a7cf93-696b58646-gp5l7"]
Nov 02 04:03:54.755401 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:54.755378    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-147104-658844d6c9-899t6"
Nov 02 04:03:54.813472 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:54.813443    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f80f210-8056-4a78-a27f-5166ab50c9d0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f80f210-8056-4a78-a27f-5166ab50c9d0-metrics-certs\") pod \"7f80f210-8056-4a78-a27f-5166ab50c9d0\" (UID: \"7f80f210-8056-4a78-a27f-5166ab50c9d0\") "
Nov 02 04:03:54.813626 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:54.813514    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f80f210-8056-4a78-a27f-5166ab50c9d0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f80f210-8056-4a78-a27f-5166ab50c9d0-maas-api-tls\") pod \"7f80f210-8056-4a78-a27f-5166ab50c9d0\" (UID: \"7f80f210-8056-4a78-a27f-5166ab50c9d0\") "
Nov 02 04:03:54.813626 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:54.813578    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f80f210-8056-4a78-a27f-5166ab50c9d0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f80f210-8056-4a78-a27f-5166ab50c9d0-openshift-service-ca\") pod \"7f80f210-8056-4a78-a27f-5166ab50c9d0\" (UID: \"7f80f210-8056-4a78-a27f-5166ab50c9d0\") "
Nov 02 04:03:54.813755 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:54.813675    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f80f210-8056-4a78-a27f-5166ab50c9d0-kube-api-access-wrct2\" (UniqueName: \"kubernetes.io/projected/7f80f210-8056-4a78-a27f-5166ab50c9d0-kube-api-access-wrct2\") pod \"7f80f210-8056-4a78-a27f-5166ab50c9d0\" (UID: \"7f80f210-8056-4a78-a27f-5166ab50c9d0\") "
Nov 02 04:03:54.813995 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:54.813964    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f80f210-8056-4a78-a27f-5166ab50c9d0-openshift-service-ca" pod "7f80f210-8056-4a78-a27f-5166ab50c9d0" (UID: "7f80f210-8056-4a78-a27f-5166ab50c9d0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 04:03:54.814208 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:54.814187    4496 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f80f210-8056-4a78-a27f-5166ab50c9d0-openshift-service-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:03:54.815503 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:54.815478    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f80f210-8056-4a78-a27f-5166ab50c9d0-metrics-certs" pod "7f80f210-8056-4a78-a27f-5166ab50c9d0" (UID: "7f80f210-8056-4a78-a27f-5166ab50c9d0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 04:03:54.815755 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:54.815726    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f80f210-8056-4a78-a27f-5166ab50c9d0-maas-api-tls" pod "7f80f210-8056-4a78-a27f-5166ab50c9d0" (UID: "7f80f210-8056-4a78-a27f-5166ab50c9d0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 04:03:54.815849 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:54.815732    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f80f210-8056-4a78-a27f-5166ab50c9d0-kube-api-access-wrct2" pod "7f80f210-8056-4a78-a27f-5166ab50c9d0" (UID: "7f80f210-8056-4a78-a27f-5166ab50c9d0"). InnerVolumeSpecName "kube-api-access-wrct2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 04:03:54.915524 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:54.915489    4496 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f80f210-8056-4a78-a27f-5166ab50c9d0-metrics-certs\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:03:54.915524 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:54.915523    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f80f210-8056-4a78-a27f-5166ab50c9d0-maas-api-tls\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:03:54.915736 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:54.915537    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrct2\" (UniqueName: \"kubernetes.io/projected/7f80f210-8056-4a78-a27f-5166ab50c9d0-kube-api-access-wrct2\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:03:55.562678 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:55.562641    4496 generic.go:356] "Generic (PLEG): container finished" podID="7f80f210-8056-4a78-a27f-5166ab50c9d0" containerID="1d478ed3c1c0cd05b0f9a93864fb833e8e26bb2d1fd8bc3e1672d638b752d65f" exitCode=0
Nov 02 04:03:55.563222 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:55.562688    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-147104-658844d6c9-899t6"
Nov 02 04:03:55.563222 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:55.562725    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-147104-658844d6c9-899t6" event={"ID":"7f80f210-8056-4a78-a27f-5166ab50c9d0","Type":"ContainerDied","Data":"1d478ed3c1c0cd05b0f9a93864fb833e8e26bb2d1fd8bc3e1672d638b752d65f"}
Nov 02 04:03:55.563222 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:55.562813    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-147104-658844d6c9-899t6" event={"ID":"7f80f210-8056-4a78-a27f-5166ab50c9d0","Type":"ContainerDied","Data":"9472cdf207c56d1e878d1b97f837b89bc13850035d42423cbb0b668ed039abbc"}
Nov 02 04:03:55.563222 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:55.562842    4496 scope.go:122] "RemoveContainer" containerID="1d478ed3c1c0cd05b0f9a93864fb833e8e26bb2d1fd8bc3e1672d638b752d65f"
Nov 02 04:03:55.576486 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:55.576463    4496 scope.go:122] "RemoveContainer" containerID="1d478ed3c1c0cd05b0f9a93864fb833e8e26bb2d1fd8bc3e1672d638b752d65f"
Nov 02 04:03:55.576785 ip-10-0-133-163 kubenswrapper[4496]: E1102 04:03:55.576764    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d478ed3c1c0cd05b0f9a93864fb833e8e26bb2d1fd8bc3e1672d638b752d65f\": container with ID starting with 1d478ed3c1c0cd05b0f9a93864fb833e8e26bb2d1fd8bc3e1672d638b752d65f not found: ID does not exist" containerID="1d478ed3c1c0cd05b0f9a93864fb833e8e26bb2d1fd8bc3e1672d638b752d65f"
Nov 02 04:03:55.576875 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:55.576792    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d478ed3c1c0cd05b0f9a93864fb833e8e26bb2d1fd8bc3e1672d638b752d65f"} err="failed to get container status \"1d478ed3c1c0cd05b0f9a93864fb833e8e26bb2d1fd8bc3e1672d638b752d65f\": rpc error: code = NotFound desc = could not find container \"1d478ed3c1c0cd05b0f9a93864fb833e8e26bb2d1fd8bc3e1672d638b752d65f\": container with ID starting with 1d478ed3c1c0cd05b0f9a93864fb833e8e26bb2d1fd8bc3e1672d638b752d65f not found: ID does not exist"
Nov 02 04:03:55.577105 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:55.577035    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44122d78-2a1a-43bb-a123-4494b445d939" path="/var/lib/kubelet/pods/44122d78-2a1a-43bb-a123-4494b445d939/volumes"
Nov 02 04:03:55.672749 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:55.672701    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7c1bee-67bb9f8cc7-lx755"]
Nov 02 04:03:55.672987 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:55.672957    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7c1bee-67bb9f8cc7-lx755" podUID="5649b53a-018e-41be-8592-7d828b6ef19f" containerName="maas-api" containerID="cri-o://7403777d5d9b7994f9d441505e956f37eecb904494bd1c7a9e996cc1689ccc26" gracePeriod=30
Nov 02 04:03:55.736814 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:55.736776    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-147104-658844d6c9-899t6"]
Nov 02 04:03:55.740291 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:55.740263    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-147104-658844d6c9-899t6"]
Nov 02 04:03:55.940966 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:55.940932    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7c1bee-67bb9f8cc7-lx755"
Nov 02 04:03:56.028903 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:56.028868    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5649b53a-018e-41be-8592-7d828b6ef19f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5649b53a-018e-41be-8592-7d828b6ef19f-openshift-service-ca\") pod \"5649b53a-018e-41be-8592-7d828b6ef19f\" (UID: \"5649b53a-018e-41be-8592-7d828b6ef19f\") "
Nov 02 04:03:56.029122 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:56.029010    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5649b53a-018e-41be-8592-7d828b6ef19f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5649b53a-018e-41be-8592-7d828b6ef19f-maas-api-tls\") pod \"5649b53a-018e-41be-8592-7d828b6ef19f\" (UID: \"5649b53a-018e-41be-8592-7d828b6ef19f\") "
Nov 02 04:03:56.029122 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:56.029094    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5649b53a-018e-41be-8592-7d828b6ef19f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5649b53a-018e-41be-8592-7d828b6ef19f-metrics-certs\") pod \"5649b53a-018e-41be-8592-7d828b6ef19f\" (UID: \"5649b53a-018e-41be-8592-7d828b6ef19f\") "
Nov 02 04:03:56.029258 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:56.029222    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5649b53a-018e-41be-8592-7d828b6ef19f-kube-api-access-kkbgh\" (UniqueName: \"kubernetes.io/projected/5649b53a-018e-41be-8592-7d828b6ef19f-kube-api-access-kkbgh\") pod \"5649b53a-018e-41be-8592-7d828b6ef19f\" (UID: \"5649b53a-018e-41be-8592-7d828b6ef19f\") "
Nov 02 04:03:56.029258 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:56.029245    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5649b53a-018e-41be-8592-7d828b6ef19f-openshift-service-ca" pod "5649b53a-018e-41be-8592-7d828b6ef19f" (UID: "5649b53a-018e-41be-8592-7d828b6ef19f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 04:03:56.029648 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:56.029627    4496 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5649b53a-018e-41be-8592-7d828b6ef19f-openshift-service-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:03:56.031113 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:56.031087    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5649b53a-018e-41be-8592-7d828b6ef19f-metrics-certs" pod "5649b53a-018e-41be-8592-7d828b6ef19f" (UID: "5649b53a-018e-41be-8592-7d828b6ef19f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 04:03:56.031530 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:56.031509    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5649b53a-018e-41be-8592-7d828b6ef19f-maas-api-tls" pod "5649b53a-018e-41be-8592-7d828b6ef19f" (UID: "5649b53a-018e-41be-8592-7d828b6ef19f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 04:03:56.031606 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:56.031537    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5649b53a-018e-41be-8592-7d828b6ef19f-kube-api-access-kkbgh" pod "5649b53a-018e-41be-8592-7d828b6ef19f" (UID: "5649b53a-018e-41be-8592-7d828b6ef19f"). InnerVolumeSpecName "kube-api-access-kkbgh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 04:03:56.130823 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:56.130745    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkbgh\" (UniqueName: \"kubernetes.io/projected/5649b53a-018e-41be-8592-7d828b6ef19f-kube-api-access-kkbgh\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:03:56.130823 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:56.130776    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5649b53a-018e-41be-8592-7d828b6ef19f-maas-api-tls\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:03:56.130823 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:56.130787    4496 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5649b53a-018e-41be-8592-7d828b6ef19f-metrics-certs\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:03:56.574759 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:56.574722    4496 generic.go:356] "Generic (PLEG): container finished" podID="5649b53a-018e-41be-8592-7d828b6ef19f" containerID="7403777d5d9b7994f9d441505e956f37eecb904494bd1c7a9e996cc1689ccc26" exitCode=0
Nov 02 04:03:56.574759 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:56.574764    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7c1bee-67bb9f8cc7-lx755"
Nov 02 04:03:56.575379 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:56.574800    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7c1bee-67bb9f8cc7-lx755" event={"ID":"5649b53a-018e-41be-8592-7d828b6ef19f","Type":"ContainerDied","Data":"7403777d5d9b7994f9d441505e956f37eecb904494bd1c7a9e996cc1689ccc26"}
Nov 02 04:03:56.575379 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:56.574852    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7c1bee-67bb9f8cc7-lx755" event={"ID":"5649b53a-018e-41be-8592-7d828b6ef19f","Type":"ContainerDied","Data":"5c67376f8abc5814ffe3685d4d0608d15c5fcb1dd5133a57d8a288d00f571c50"}
Nov 02 04:03:56.575379 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:56.574873    4496 scope.go:122] "RemoveContainer" containerID="7403777d5d9b7994f9d441505e956f37eecb904494bd1c7a9e996cc1689ccc26"
Nov 02 04:03:56.587404 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:56.587383    4496 scope.go:122] "RemoveContainer" containerID="7403777d5d9b7994f9d441505e956f37eecb904494bd1c7a9e996cc1689ccc26"
Nov 02 04:03:56.587684 ip-10-0-133-163 kubenswrapper[4496]: E1102 04:03:56.587660    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7403777d5d9b7994f9d441505e956f37eecb904494bd1c7a9e996cc1689ccc26\": container with ID starting with 7403777d5d9b7994f9d441505e956f37eecb904494bd1c7a9e996cc1689ccc26 not found: ID does not exist" containerID="7403777d5d9b7994f9d441505e956f37eecb904494bd1c7a9e996cc1689ccc26"
Nov 02 04:03:56.587789 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:56.587690    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7403777d5d9b7994f9d441505e956f37eecb904494bd1c7a9e996cc1689ccc26"} err="failed to get container status \"7403777d5d9b7994f9d441505e956f37eecb904494bd1c7a9e996cc1689ccc26\": rpc error: code = NotFound desc = could not find container \"7403777d5d9b7994f9d441505e956f37eecb904494bd1c7a9e996cc1689ccc26\": container with ID starting with 7403777d5d9b7994f9d441505e956f37eecb904494bd1c7a9e996cc1689ccc26 not found: ID does not exist"
Nov 02 04:03:56.743578 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:56.743540    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7c1bee-67bb9f8cc7-lx755"]
Nov 02 04:03:56.746554 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:56.746528    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7c1bee-67bb9f8cc7-lx755"]
Nov 02 04:03:56.840005 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:56.839923    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-d94cc3-774b67d9f-knmz4"]
Nov 02 04:03:56.915278 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:56.915240    4496 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-7c1bee-84b77485bf-9bk69" podUID="d147ea36-e9b2-42de-ac31-de7c5e03fe51" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 04:03:56.915491 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:56.915244    4496 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7c1bee-694764dc5-zrthf" podUID="2046cb9f-964f-45f4-9f8a-e427b5818ffc" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 04:03:57.577381 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:57.577340    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5649b53a-018e-41be-8592-7d828b6ef19f" path="/var/lib/kubelet/pods/5649b53a-018e-41be-8592-7d828b6ef19f/volumes"
Nov 02 04:03:57.577984 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:03:57.577965    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f80f210-8056-4a78-a27f-5166ab50c9d0" path="/var/lib/kubelet/pods/7f80f210-8056-4a78-a27f-5166ab50c9d0/volumes"
Nov 02 04:04:01.840427 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:01.840371    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d94cc3-774b67d9f-knmz4" podUID="5215d837-30bb-43d6-81fa-6d50df5e45bf" containerName="payload-processing" containerID="cri-o://4d9282d0eced70db7ca57c22282db6123d057532dd2e1f6dcc3b4781fc299163" gracePeriod=25
Nov 02 04:04:03.525897 ip-10-0-133-163 kubenswrapper[4496]: E1102 04:04:03.525856    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 04:04:06.915328 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:06.915291    4496 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7c1bee-694764dc5-zrthf" podUID="2046cb9f-964f-45f4-9f8a-e427b5818ffc" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 04:04:06.915824 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:06.915388    4496 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-7c1bee-84b77485bf-9bk69" podUID="d147ea36-e9b2-42de-ac31-de7c5e03fe51" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 04:04:09.229247 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:09.229209    4496 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d94cc3-774b67d9f-knmz4" podUID="5215d837-30bb-43d6-81fa-6d50df5e45bf" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 04:04:12.295472 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:12.295445    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-7c1bee-84b77485bf-9bk69"
Nov 02 04:04:12.393924 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:12.393898    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7c1bee-694764dc5-zrthf"
Nov 02 04:04:12.428114 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:12.428080    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d147ea36-e9b2-42de-ac31-de7c5e03fe51-kube-api-access-w4lss\" (UniqueName: \"kubernetes.io/projected/d147ea36-e9b2-42de-ac31-de7c5e03fe51-kube-api-access-w4lss\") pod \"d147ea36-e9b2-42de-ac31-de7c5e03fe51\" (UID: \"d147ea36-e9b2-42de-ac31-de7c5e03fe51\") "
Nov 02 04:04:12.428283 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:12.428183    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d147ea36-e9b2-42de-ac31-de7c5e03fe51-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d147ea36-e9b2-42de-ac31-de7c5e03fe51-plugins-config-volume\") pod \"d147ea36-e9b2-42de-ac31-de7c5e03fe51\" (UID: \"d147ea36-e9b2-42de-ac31-de7c5e03fe51\") "
Nov 02 04:04:12.428568 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:12.428543    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d147ea36-e9b2-42de-ac31-de7c5e03fe51-plugins-config-volume" pod "d147ea36-e9b2-42de-ac31-de7c5e03fe51" (UID: "d147ea36-e9b2-42de-ac31-de7c5e03fe51"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 04:04:12.430341 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:12.430319    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d147ea36-e9b2-42de-ac31-de7c5e03fe51-kube-api-access-w4lss" pod "d147ea36-e9b2-42de-ac31-de7c5e03fe51" (UID: "d147ea36-e9b2-42de-ac31-de7c5e03fe51"). InnerVolumeSpecName "kube-api-access-w4lss". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 04:04:12.529471 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:12.529398    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2046cb9f-964f-45f4-9f8a-e427b5818ffc-kube-api-access-d755f\" (UniqueName: \"kubernetes.io/projected/2046cb9f-964f-45f4-9f8a-e427b5818ffc-kube-api-access-d755f\") pod \"2046cb9f-964f-45f4-9f8a-e427b5818ffc\" (UID: \"2046cb9f-964f-45f4-9f8a-e427b5818ffc\") "
Nov 02 04:04:12.529624 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:12.529580    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2046cb9f-964f-45f4-9f8a-e427b5818ffc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2046cb9f-964f-45f4-9f8a-e427b5818ffc-plugins-config-volume\") pod \"2046cb9f-964f-45f4-9f8a-e427b5818ffc\" (UID: \"2046cb9f-964f-45f4-9f8a-e427b5818ffc\") "
Nov 02 04:04:12.529898 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:12.529872    4496 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d147ea36-e9b2-42de-ac31-de7c5e03fe51-plugins-config-volume\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:04:12.529898 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:12.529890    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4lss\" (UniqueName: \"kubernetes.io/projected/d147ea36-e9b2-42de-ac31-de7c5e03fe51-kube-api-access-w4lss\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:04:12.529898 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:12.529893    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2046cb9f-964f-45f4-9f8a-e427b5818ffc-plugins-config-volume" pod "2046cb9f-964f-45f4-9f8a-e427b5818ffc" (UID: "2046cb9f-964f-45f4-9f8a-e427b5818ffc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 04:04:12.532295 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:12.532264    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2046cb9f-964f-45f4-9f8a-e427b5818ffc-kube-api-access-d755f" pod "2046cb9f-964f-45f4-9f8a-e427b5818ffc" (UID: "2046cb9f-964f-45f4-9f8a-e427b5818ffc"). InnerVolumeSpecName "kube-api-access-d755f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 04:04:12.630636 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:12.630603    4496 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2046cb9f-964f-45f4-9f8a-e427b5818ffc-plugins-config-volume\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:04:12.630636 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:12.630630    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d755f\" (UniqueName: \"kubernetes.io/projected/2046cb9f-964f-45f4-9f8a-e427b5818ffc-kube-api-access-d755f\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:04:12.696818 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:12.696777    4496 generic.go:356] "Generic (PLEG): container finished" podID="d147ea36-e9b2-42de-ac31-de7c5e03fe51" containerID="033f92905c6e23818d5fd5699de333f7c23cd44452c99a0e5cf3f03f30d8bbd3" exitCode=0
Nov 02 04:04:12.697007 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:12.696827    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-7c1bee-84b77485bf-9bk69"
Nov 02 04:04:12.697007 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:12.696869    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-7c1bee-84b77485bf-9bk69" event={"ID":"d147ea36-e9b2-42de-ac31-de7c5e03fe51","Type":"ContainerDied","Data":"033f92905c6e23818d5fd5699de333f7c23cd44452c99a0e5cf3f03f30d8bbd3"}
Nov 02 04:04:12.697007 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:12.696919    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-7c1bee-84b77485bf-9bk69" event={"ID":"d147ea36-e9b2-42de-ac31-de7c5e03fe51","Type":"ContainerDied","Data":"ffc55630fa4142853775e386bce25e5e8b7423082327f7471dea9f8fb44286be"}
Nov 02 04:04:12.697007 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:12.696944    4496 scope.go:122] "RemoveContainer" containerID="033f92905c6e23818d5fd5699de333f7c23cd44452c99a0e5cf3f03f30d8bbd3"
Nov 02 04:04:12.698473 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:12.698448    4496 generic.go:356] "Generic (PLEG): container finished" podID="2046cb9f-964f-45f4-9f8a-e427b5818ffc" containerID="82cb4bb8139daaf6a20153fc19c23fc0243490b51055519ba1344df219fbd0d5" exitCode=0
Nov 02 04:04:12.698590 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:12.698492    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7c1bee-694764dc5-zrthf"
Nov 02 04:04:12.698590 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:12.698519    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7c1bee-694764dc5-zrthf" event={"ID":"2046cb9f-964f-45f4-9f8a-e427b5818ffc","Type":"ContainerDied","Data":"82cb4bb8139daaf6a20153fc19c23fc0243490b51055519ba1344df219fbd0d5"}
Nov 02 04:04:12.698590 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:12.698559    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7c1bee-694764dc5-zrthf" event={"ID":"2046cb9f-964f-45f4-9f8a-e427b5818ffc","Type":"ContainerDied","Data":"59b7f28ae6ec6681da5789fca6635b388417b370e87085c607b566f63089d392"}
Nov 02 04:04:12.708686 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:12.708668    4496 scope.go:122] "RemoveContainer" containerID="033f92905c6e23818d5fd5699de333f7c23cd44452c99a0e5cf3f03f30d8bbd3"
Nov 02 04:04:12.709089 ip-10-0-133-163 kubenswrapper[4496]: E1102 04:04:12.709062    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"033f92905c6e23818d5fd5699de333f7c23cd44452c99a0e5cf3f03f30d8bbd3\": container with ID starting with 033f92905c6e23818d5fd5699de333f7c23cd44452c99a0e5cf3f03f30d8bbd3 not found: ID does not exist" containerID="033f92905c6e23818d5fd5699de333f7c23cd44452c99a0e5cf3f03f30d8bbd3"
Nov 02 04:04:12.709187 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:12.709097    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"033f92905c6e23818d5fd5699de333f7c23cd44452c99a0e5cf3f03f30d8bbd3"} err="failed to get container status \"033f92905c6e23818d5fd5699de333f7c23cd44452c99a0e5cf3f03f30d8bbd3\": rpc error: code = NotFound desc = could not find container \"033f92905c6e23818d5fd5699de333f7c23cd44452c99a0e5cf3f03f30d8bbd3\": container with ID starting with 033f92905c6e23818d5fd5699de333f7c23cd44452c99a0e5cf3f03f30d8bbd3 not found: ID does not exist"
Nov 02 04:04:12.709187 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:12.709125    4496 scope.go:122] "RemoveContainer" containerID="82cb4bb8139daaf6a20153fc19c23fc0243490b51055519ba1344df219fbd0d5"
Nov 02 04:04:12.717802 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:12.717785    4496 scope.go:122] "RemoveContainer" containerID="82cb4bb8139daaf6a20153fc19c23fc0243490b51055519ba1344df219fbd0d5"
Nov 02 04:04:12.718079 ip-10-0-133-163 kubenswrapper[4496]: E1102 04:04:12.718059    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82cb4bb8139daaf6a20153fc19c23fc0243490b51055519ba1344df219fbd0d5\": container with ID starting with 82cb4bb8139daaf6a20153fc19c23fc0243490b51055519ba1344df219fbd0d5 not found: ID does not exist" containerID="82cb4bb8139daaf6a20153fc19c23fc0243490b51055519ba1344df219fbd0d5"
Nov 02 04:04:12.718132 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:12.718085    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82cb4bb8139daaf6a20153fc19c23fc0243490b51055519ba1344df219fbd0d5"} err="failed to get container status \"82cb4bb8139daaf6a20153fc19c23fc0243490b51055519ba1344df219fbd0d5\": rpc error: code = NotFound desc = could not find container \"82cb4bb8139daaf6a20153fc19c23fc0243490b51055519ba1344df219fbd0d5\": container with ID starting with 82cb4bb8139daaf6a20153fc19c23fc0243490b51055519ba1344df219fbd0d5 not found: ID does not exist"
Nov 02 04:04:12.868091 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:12.867984    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7c1bee-694764dc5-zrthf"]
Nov 02 04:04:12.870986 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:12.870964    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7c1bee-694764dc5-zrthf"]
Nov 02 04:04:12.981916 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:12.981882    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-7c1bee-84b77485bf-9bk69"]
Nov 02 04:04:12.984799 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:12.984776    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-7c1bee-84b77485bf-9bk69"]
Nov 02 04:04:13.575424 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:13.575388    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2046cb9f-964f-45f4-9f8a-e427b5818ffc" path="/var/lib/kubelet/pods/2046cb9f-964f-45f4-9f8a-e427b5818ffc/volumes"
Nov 02 04:04:13.576002 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:13.575984    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d147ea36-e9b2-42de-ac31-de7c5e03fe51" path="/var/lib/kubelet/pods/d147ea36-e9b2-42de-ac31-de7c5e03fe51/volumes"
Nov 02 04:04:15.890217 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:15.890176    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-1f6e39-69f9854f95-b4c6w"]
Nov 02 04:04:16.000337 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:16.000302    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-1f6e39-6d697b7999-7nsr8"]
Nov 02 04:04:19.229818 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:19.229780    4496 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d94cc3-774b67d9f-knmz4" podUID="5215d837-30bb-43d6-81fa-6d50df5e45bf" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 04:04:20.890957 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:20.890914    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-1f6e39-69f9854f95-b4c6w" podUID="ccb123ce-4b29-4f58-8201-34433a8bd0b0" containerName="payload-processing" containerID="cri-o://d470216e044b74b66708cf920cfe1d9849b71fdacdaa3f78dc421377e557a3d7" gracePeriod=25
Nov 02 04:04:21.001271 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:21.001229    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1f6e39-6d697b7999-7nsr8" podUID="725d05a7-caa3-4e07-be47-837ea9534075" containerName="payload-pre-processing" containerID="cri-o://c27812b8c5bc2b0f391976b47c5841944ffbc08fb9024f09969f744391103a38" gracePeriod=25
Nov 02 04:04:22.114096 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:22.114039    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d94cc3-774b67d9f-knmz4"
Nov 02 04:04:22.148937 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:22.148900    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5215d837-30bb-43d6-81fa-6d50df5e45bf-kube-api-access-7gxs2\" (UniqueName: \"kubernetes.io/projected/5215d837-30bb-43d6-81fa-6d50df5e45bf-kube-api-access-7gxs2\") pod \"5215d837-30bb-43d6-81fa-6d50df5e45bf\" (UID: \"5215d837-30bb-43d6-81fa-6d50df5e45bf\") "
Nov 02 04:04:22.149096 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:22.149056    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5215d837-30bb-43d6-81fa-6d50df5e45bf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5215d837-30bb-43d6-81fa-6d50df5e45bf-plugins-config-volume\") pod \"5215d837-30bb-43d6-81fa-6d50df5e45bf\" (UID: \"5215d837-30bb-43d6-81fa-6d50df5e45bf\") "
Nov 02 04:04:22.149412 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:22.149389    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5215d837-30bb-43d6-81fa-6d50df5e45bf-plugins-config-volume" pod "5215d837-30bb-43d6-81fa-6d50df5e45bf" (UID: "5215d837-30bb-43d6-81fa-6d50df5e45bf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 04:04:22.149648 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:22.149629    4496 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5215d837-30bb-43d6-81fa-6d50df5e45bf-plugins-config-volume\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:04:22.150995 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:22.150969    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5215d837-30bb-43d6-81fa-6d50df5e45bf-kube-api-access-7gxs2" pod "5215d837-30bb-43d6-81fa-6d50df5e45bf" (UID: "5215d837-30bb-43d6-81fa-6d50df5e45bf"). InnerVolumeSpecName "kube-api-access-7gxs2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 04:04:22.250950 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:22.250906    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gxs2\" (UniqueName: \"kubernetes.io/projected/5215d837-30bb-43d6-81fa-6d50df5e45bf-kube-api-access-7gxs2\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:04:22.770465 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:22.770426    4496 generic.go:356] "Generic (PLEG): container finished" podID="5215d837-30bb-43d6-81fa-6d50df5e45bf" containerID="4d9282d0eced70db7ca57c22282db6123d057532dd2e1f6dcc3b4781fc299163" exitCode=0
Nov 02 04:04:22.770652 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:22.770472    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d94cc3-774b67d9f-knmz4"
Nov 02 04:04:22.770652 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:22.770512    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d94cc3-774b67d9f-knmz4" event={"ID":"5215d837-30bb-43d6-81fa-6d50df5e45bf","Type":"ContainerDied","Data":"4d9282d0eced70db7ca57c22282db6123d057532dd2e1f6dcc3b4781fc299163"}
Nov 02 04:04:22.770652 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:22.770576    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d94cc3-774b67d9f-knmz4" event={"ID":"5215d837-30bb-43d6-81fa-6d50df5e45bf","Type":"ContainerDied","Data":"e81883696cdba08bd1cec8e37933aa24e9e65b0973c325afaab3c70f4c600c9e"}
Nov 02 04:04:22.770652 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:22.770605    4496 scope.go:122] "RemoveContainer" containerID="4d9282d0eced70db7ca57c22282db6123d057532dd2e1f6dcc3b4781fc299163"
Nov 02 04:04:22.783352 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:22.783335    4496 scope.go:122] "RemoveContainer" containerID="4d9282d0eced70db7ca57c22282db6123d057532dd2e1f6dcc3b4781fc299163"
Nov 02 04:04:22.783636 ip-10-0-133-163 kubenswrapper[4496]: E1102 04:04:22.783617    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d9282d0eced70db7ca57c22282db6123d057532dd2e1f6dcc3b4781fc299163\": container with ID starting with 4d9282d0eced70db7ca57c22282db6123d057532dd2e1f6dcc3b4781fc299163 not found: ID does not exist" containerID="4d9282d0eced70db7ca57c22282db6123d057532dd2e1f6dcc3b4781fc299163"
Nov 02 04:04:22.783714 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:22.783647    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d9282d0eced70db7ca57c22282db6123d057532dd2e1f6dcc3b4781fc299163"} err="failed to get container status \"4d9282d0eced70db7ca57c22282db6123d057532dd2e1f6dcc3b4781fc299163\": rpc error: code = NotFound desc = could not find container \"4d9282d0eced70db7ca57c22282db6123d057532dd2e1f6dcc3b4781fc299163\": container with ID starting with 4d9282d0eced70db7ca57c22282db6123d057532dd2e1f6dcc3b4781fc299163 not found: ID does not exist"
Nov 02 04:04:22.938469 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:22.938436    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-d94cc3-774b67d9f-knmz4"]
Nov 02 04:04:22.942159 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:22.942129    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-d94cc3-774b67d9f-knmz4"]
Nov 02 04:04:23.573518 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:23.573487    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5215d837-30bb-43d6-81fa-6d50df5e45bf" path="/var/lib/kubelet/pods/5215d837-30bb-43d6-81fa-6d50df5e45bf/volumes"
Nov 02 04:04:27.763603 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:27.763565    4496 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1f6e39-6d697b7999-7nsr8" podUID="725d05a7-caa3-4e07-be47-837ea9534075" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 04:04:28.770484 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:28.770444    4496 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-1f6e39-69f9854f95-b4c6w" podUID="ccb123ce-4b29-4f58-8201-34433a8bd0b0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 04:04:33.530992 ip-10-0-133-163 kubenswrapper[4496]: E1102 04:04:33.530953    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 04:04:37.763071 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:37.763028    4496 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1f6e39-6d697b7999-7nsr8" podUID="725d05a7-caa3-4e07-be47-837ea9534075" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 04:04:38.770614 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:38.770566    4496 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-1f6e39-69f9854f95-b4c6w" podUID="ccb123ce-4b29-4f58-8201-34433a8bd0b0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 04:04:41.173933 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:41.173908    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-1f6e39-69f9854f95-b4c6w"
Nov 02 04:04:41.272105 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:41.272079    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1f6e39-6d697b7999-7nsr8"
Nov 02 04:04:41.365915 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:41.365889    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccb123ce-4b29-4f58-8201-34433a8bd0b0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccb123ce-4b29-4f58-8201-34433a8bd0b0-plugins-config-volume\") pod \"ccb123ce-4b29-4f58-8201-34433a8bd0b0\" (UID: \"ccb123ce-4b29-4f58-8201-34433a8bd0b0\") "
Nov 02 04:04:41.365915 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:41.365921    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccb123ce-4b29-4f58-8201-34433a8bd0b0-kube-api-access-bp8w8\" (UniqueName: \"kubernetes.io/projected/ccb123ce-4b29-4f58-8201-34433a8bd0b0-kube-api-access-bp8w8\") pod \"ccb123ce-4b29-4f58-8201-34433a8bd0b0\" (UID: \"ccb123ce-4b29-4f58-8201-34433a8bd0b0\") "
Nov 02 04:04:41.366314 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:41.366287    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccb123ce-4b29-4f58-8201-34433a8bd0b0-plugins-config-volume" pod "ccb123ce-4b29-4f58-8201-34433a8bd0b0" (UID: "ccb123ce-4b29-4f58-8201-34433a8bd0b0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 04:04:41.366529 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:41.366504    4496 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccb123ce-4b29-4f58-8201-34433a8bd0b0-plugins-config-volume\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:04:41.368400 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:41.368377    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccb123ce-4b29-4f58-8201-34433a8bd0b0-kube-api-access-bp8w8" pod "ccb123ce-4b29-4f58-8201-34433a8bd0b0" (UID: "ccb123ce-4b29-4f58-8201-34433a8bd0b0"). InnerVolumeSpecName "kube-api-access-bp8w8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 04:04:41.467220 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:41.467148    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/725d05a7-caa3-4e07-be47-837ea9534075-kube-api-access-dv2mw\" (UniqueName: \"kubernetes.io/projected/725d05a7-caa3-4e07-be47-837ea9534075-kube-api-access-dv2mw\") pod \"725d05a7-caa3-4e07-be47-837ea9534075\" (UID: \"725d05a7-caa3-4e07-be47-837ea9534075\") "
Nov 02 04:04:41.467220 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:41.467203    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/725d05a7-caa3-4e07-be47-837ea9534075-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/725d05a7-caa3-4e07-be47-837ea9534075-plugins-config-volume\") pod \"725d05a7-caa3-4e07-be47-837ea9534075\" (UID: \"725d05a7-caa3-4e07-be47-837ea9534075\") "
Nov 02 04:04:41.467557 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:41.467538    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bp8w8\" (UniqueName: \"kubernetes.io/projected/ccb123ce-4b29-4f58-8201-34433a8bd0b0-kube-api-access-bp8w8\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:04:41.467674 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:41.467580    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/725d05a7-caa3-4e07-be47-837ea9534075-plugins-config-volume" pod "725d05a7-caa3-4e07-be47-837ea9534075" (UID: "725d05a7-caa3-4e07-be47-837ea9534075"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 04:04:41.469197 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:41.469178    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/725d05a7-caa3-4e07-be47-837ea9534075-kube-api-access-dv2mw" pod "725d05a7-caa3-4e07-be47-837ea9534075" (UID: "725d05a7-caa3-4e07-be47-837ea9534075"). InnerVolumeSpecName "kube-api-access-dv2mw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 04:04:41.568803 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:41.568775    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dv2mw\" (UniqueName: \"kubernetes.io/projected/725d05a7-caa3-4e07-be47-837ea9534075-kube-api-access-dv2mw\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:04:41.568803 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:41.568801    4496 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/725d05a7-caa3-4e07-be47-837ea9534075-plugins-config-volume\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:04:41.910921 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:41.910882    4496 generic.go:356] "Generic (PLEG): container finished" podID="725d05a7-caa3-4e07-be47-837ea9534075" containerID="c27812b8c5bc2b0f391976b47c5841944ffbc08fb9024f09969f744391103a38" exitCode=0
Nov 02 04:04:41.911108 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:41.910956    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1f6e39-6d697b7999-7nsr8"
Nov 02 04:04:41.911108 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:41.910959    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1f6e39-6d697b7999-7nsr8" event={"ID":"725d05a7-caa3-4e07-be47-837ea9534075","Type":"ContainerDied","Data":"c27812b8c5bc2b0f391976b47c5841944ffbc08fb9024f09969f744391103a38"}
Nov 02 04:04:41.911108 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:41.911038    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1f6e39-6d697b7999-7nsr8" event={"ID":"725d05a7-caa3-4e07-be47-837ea9534075","Type":"ContainerDied","Data":"6cffc3bdd69f67f21c70a3e02d7e01b0b2b82726851bc44d8cce42aa2e7d23b2"}
Nov 02 04:04:41.911108 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:41.911054    4496 scope.go:122] "RemoveContainer" containerID="c27812b8c5bc2b0f391976b47c5841944ffbc08fb9024f09969f744391103a38"
Nov 02 04:04:41.912757 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:41.912734    4496 generic.go:356] "Generic (PLEG): container finished" podID="ccb123ce-4b29-4f58-8201-34433a8bd0b0" containerID="d470216e044b74b66708cf920cfe1d9849b71fdacdaa3f78dc421377e557a3d7" exitCode=0
Nov 02 04:04:41.912896 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:41.912779    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-1f6e39-69f9854f95-b4c6w" event={"ID":"ccb123ce-4b29-4f58-8201-34433a8bd0b0","Type":"ContainerDied","Data":"d470216e044b74b66708cf920cfe1d9849b71fdacdaa3f78dc421377e557a3d7"}
Nov 02 04:04:41.912896 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:41.912805    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-1f6e39-69f9854f95-b4c6w"
Nov 02 04:04:41.912896 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:41.912817    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-1f6e39-69f9854f95-b4c6w" event={"ID":"ccb123ce-4b29-4f58-8201-34433a8bd0b0","Type":"ContainerDied","Data":"98d2e67a0250c09086032a4640b6e181d52c6d78ffd4bdd5d754af7dfaa96d71"}
Nov 02 04:04:41.921593 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:41.921576    4496 scope.go:122] "RemoveContainer" containerID="c27812b8c5bc2b0f391976b47c5841944ffbc08fb9024f09969f744391103a38"
Nov 02 04:04:41.921899 ip-10-0-133-163 kubenswrapper[4496]: E1102 04:04:41.921876    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c27812b8c5bc2b0f391976b47c5841944ffbc08fb9024f09969f744391103a38\": container with ID starting with c27812b8c5bc2b0f391976b47c5841944ffbc08fb9024f09969f744391103a38 not found: ID does not exist" containerID="c27812b8c5bc2b0f391976b47c5841944ffbc08fb9024f09969f744391103a38"
Nov 02 04:04:41.921979 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:41.921906    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c27812b8c5bc2b0f391976b47c5841944ffbc08fb9024f09969f744391103a38"} err="failed to get container status \"c27812b8c5bc2b0f391976b47c5841944ffbc08fb9024f09969f744391103a38\": rpc error: code = NotFound desc = could not find container \"c27812b8c5bc2b0f391976b47c5841944ffbc08fb9024f09969f744391103a38\": container with ID starting with c27812b8c5bc2b0f391976b47c5841944ffbc08fb9024f09969f744391103a38 not found: ID does not exist"
Nov 02 04:04:41.921979 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:41.921923    4496 scope.go:122] "RemoveContainer" containerID="d470216e044b74b66708cf920cfe1d9849b71fdacdaa3f78dc421377e557a3d7"
Nov 02 04:04:41.931997 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:41.931979    4496 scope.go:122] "RemoveContainer" containerID="d470216e044b74b66708cf920cfe1d9849b71fdacdaa3f78dc421377e557a3d7"
Nov 02 04:04:41.932281 ip-10-0-133-163 kubenswrapper[4496]: E1102 04:04:41.932260    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d470216e044b74b66708cf920cfe1d9849b71fdacdaa3f78dc421377e557a3d7\": container with ID starting with d470216e044b74b66708cf920cfe1d9849b71fdacdaa3f78dc421377e557a3d7 not found: ID does not exist" containerID="d470216e044b74b66708cf920cfe1d9849b71fdacdaa3f78dc421377e557a3d7"
Nov 02 04:04:41.932345 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:41.932292    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d470216e044b74b66708cf920cfe1d9849b71fdacdaa3f78dc421377e557a3d7"} err="failed to get container status \"d470216e044b74b66708cf920cfe1d9849b71fdacdaa3f78dc421377e557a3d7\": rpc error: code = NotFound desc = could not find container \"d470216e044b74b66708cf920cfe1d9849b71fdacdaa3f78dc421377e557a3d7\": container with ID starting with d470216e044b74b66708cf920cfe1d9849b71fdacdaa3f78dc421377e557a3d7 not found: ID does not exist"
Nov 02 04:04:42.080486 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:42.080452    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-1f6e39-6d697b7999-7nsr8"]
Nov 02 04:04:42.131154 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:42.131126    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-1f6e39-6d697b7999-7nsr8"]
Nov 02 04:04:42.189123 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:42.189044    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d94cc3-55f675b9d8-46f48"]
Nov 02 04:04:42.189490 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:42.189280    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d94cc3-55f675b9d8-46f48" podUID="2d0cd262-c6a0-4b21-af3c-393ee8051f81" containerName="maas-api" containerID="cri-o://7d8866a2ed44fbe8663451d206fa8e5e26f9559eacb89e93f861871d314e6af3" gracePeriod=30
Nov 02 04:04:42.246988 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:42.246949    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-1f6e39-69f9854f95-b4c6w"]
Nov 02 04:04:42.250410 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:42.250384    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-1f6e39-69f9854f95-b4c6w"]
Nov 02 04:04:42.459880 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:42.459847    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d94cc3-55f675b9d8-46f48"
Nov 02 04:04:42.528335 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:42.528306    4496 scope.go:122] "RemoveContainer" containerID="85e2a219bd0aa448adc75c57577cc3b0d58403a5accca00b8338a43cc004ae87"
Nov 02 04:04:42.538164 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:42.538140    4496 scope.go:122] "RemoveContainer" containerID="7d8866a2ed44fbe8663451d206fa8e5e26f9559eacb89e93f861871d314e6af3"
Nov 02 04:04:42.581487 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:42.581451    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d0cd262-c6a0-4b21-af3c-393ee8051f81-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d0cd262-c6a0-4b21-af3c-393ee8051f81-metrics-certs\") pod \"2d0cd262-c6a0-4b21-af3c-393ee8051f81\" (UID: \"2d0cd262-c6a0-4b21-af3c-393ee8051f81\") "
Nov 02 04:04:42.583128 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:42.583099    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d0cd262-c6a0-4b21-af3c-393ee8051f81-kube-api-access-cgkv8\" (UniqueName: \"kubernetes.io/projected/2d0cd262-c6a0-4b21-af3c-393ee8051f81-kube-api-access-cgkv8\") pod \"2d0cd262-c6a0-4b21-af3c-393ee8051f81\" (UID: \"2d0cd262-c6a0-4b21-af3c-393ee8051f81\") "
Nov 02 04:04:42.583222 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:42.583191    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d0cd262-c6a0-4b21-af3c-393ee8051f81-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d0cd262-c6a0-4b21-af3c-393ee8051f81-maas-api-tls\") pod \"2d0cd262-c6a0-4b21-af3c-393ee8051f81\" (UID: \"2d0cd262-c6a0-4b21-af3c-393ee8051f81\") "
Nov 02 04:04:42.583268 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:42.583249    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d0cd262-c6a0-4b21-af3c-393ee8051f81-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d0cd262-c6a0-4b21-af3c-393ee8051f81-openshift-service-ca\") pod \"2d0cd262-c6a0-4b21-af3c-393ee8051f81\" (UID: \"2d0cd262-c6a0-4b21-af3c-393ee8051f81\") "
Nov 02 04:04:42.585942 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:42.584365    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d0cd262-c6a0-4b21-af3c-393ee8051f81-openshift-service-ca" pod "2d0cd262-c6a0-4b21-af3c-393ee8051f81" (UID: "2d0cd262-c6a0-4b21-af3c-393ee8051f81"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 04:04:42.606810 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:42.606779    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d0cd262-c6a0-4b21-af3c-393ee8051f81-maas-api-tls" pod "2d0cd262-c6a0-4b21-af3c-393ee8051f81" (UID: "2d0cd262-c6a0-4b21-af3c-393ee8051f81"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 04:04:42.608938 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:42.608909    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d0cd262-c6a0-4b21-af3c-393ee8051f81-metrics-certs" pod "2d0cd262-c6a0-4b21-af3c-393ee8051f81" (UID: "2d0cd262-c6a0-4b21-af3c-393ee8051f81"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 04:04:42.613346 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:42.613320    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d0cd262-c6a0-4b21-af3c-393ee8051f81-kube-api-access-cgkv8" pod "2d0cd262-c6a0-4b21-af3c-393ee8051f81" (UID: "2d0cd262-c6a0-4b21-af3c-393ee8051f81"). InnerVolumeSpecName "kube-api-access-cgkv8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 04:04:42.685070 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:42.685033    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgkv8\" (UniqueName: \"kubernetes.io/projected/2d0cd262-c6a0-4b21-af3c-393ee8051f81-kube-api-access-cgkv8\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:04:42.685070 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:42.685071    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d0cd262-c6a0-4b21-af3c-393ee8051f81-maas-api-tls\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:04:42.685337 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:42.685085    4496 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d0cd262-c6a0-4b21-af3c-393ee8051f81-openshift-service-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:04:42.685337 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:42.685099    4496 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d0cd262-c6a0-4b21-af3c-393ee8051f81-metrics-certs\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:04:42.919924 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:42.919886    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d94cc3-55f675b9d8-46f48" event={"ID":"2d0cd262-c6a0-4b21-af3c-393ee8051f81","Type":"ContainerDied","Data":"7d8866a2ed44fbe8663451d206fa8e5e26f9559eacb89e93f861871d314e6af3"}
Nov 02 04:04:42.920078 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:42.919954    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d94cc3-55f675b9d8-46f48" event={"ID":"2d0cd262-c6a0-4b21-af3c-393ee8051f81","Type":"ContainerDied","Data":"941248c46f4a58c6b51e480b6ec09381f0d483d759394ddb40e8b91e19700c15"}
Nov 02 04:04:42.920078 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:42.919968    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d94cc3-55f675b9d8-46f48"
Nov 02 04:04:43.152522 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:43.152480    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d94cc3-55f675b9d8-46f48"]
Nov 02 04:04:43.155980 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:43.155946    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d94cc3-55f675b9d8-46f48"]
Nov 02 04:04:43.574063 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:43.574032    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d0cd262-c6a0-4b21-af3c-393ee8051f81" path="/var/lib/kubelet/pods/2d0cd262-c6a0-4b21-af3c-393ee8051f81/volumes"
Nov 02 04:04:43.574446 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:43.574428    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="725d05a7-caa3-4e07-be47-837ea9534075" path="/var/lib/kubelet/pods/725d05a7-caa3-4e07-be47-837ea9534075/volumes"
Nov 02 04:04:43.574771 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:43.574757    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccb123ce-4b29-4f58-8201-34433a8bd0b0" path="/var/lib/kubelet/pods/ccb123ce-4b29-4f58-8201-34433a8bd0b0/volumes"
Nov 02 04:04:45.382146 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:45.382095    4496 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d94cc3-55f675b9d8-46f48" podUID="2d0cd262-c6a0-4b21-af3c-393ee8051f81" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.237:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 04:04:48.407892 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:48.407855    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-ac7ab4-547f75d6d4-dn58x"]
Nov 02 04:04:48.408408 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:48.408137    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-ac7ab4-547f75d6d4-dn58x" podUID="4304f6cd-6a4b-40cf-985b-d42cb96d0ac9" containerName="maas-api" containerID="cri-o://e6501811c5d8fe951fe82478192c592c3eecd26c9b2fb941ec3f12b34a3a9b98" gracePeriod=30
Nov 02 04:04:48.670827 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:48.670758    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-ac7ab4-547f75d6d4-dn58x"
Nov 02 04:04:48.764390 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:48.764348    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4304f6cd-6a4b-40cf-985b-d42cb96d0ac9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4304f6cd-6a4b-40cf-985b-d42cb96d0ac9-maas-api-tls\") pod \"4304f6cd-6a4b-40cf-985b-d42cb96d0ac9\" (UID: \"4304f6cd-6a4b-40cf-985b-d42cb96d0ac9\") "
Nov 02 04:04:48.764570 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:48.764425    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4304f6cd-6a4b-40cf-985b-d42cb96d0ac9-kube-api-access-bzfsw\" (UniqueName: \"kubernetes.io/projected/4304f6cd-6a4b-40cf-985b-d42cb96d0ac9-kube-api-access-bzfsw\") pod \"4304f6cd-6a4b-40cf-985b-d42cb96d0ac9\" (UID: \"4304f6cd-6a4b-40cf-985b-d42cb96d0ac9\") "
Nov 02 04:04:48.764570 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:48.764562    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4304f6cd-6a4b-40cf-985b-d42cb96d0ac9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4304f6cd-6a4b-40cf-985b-d42cb96d0ac9-metrics-certs\") pod \"4304f6cd-6a4b-40cf-985b-d42cb96d0ac9\" (UID: \"4304f6cd-6a4b-40cf-985b-d42cb96d0ac9\") "
Nov 02 04:04:48.764668 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:48.764595    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4304f6cd-6a4b-40cf-985b-d42cb96d0ac9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4304f6cd-6a4b-40cf-985b-d42cb96d0ac9-openshift-service-ca\") pod \"4304f6cd-6a4b-40cf-985b-d42cb96d0ac9\" (UID: \"4304f6cd-6a4b-40cf-985b-d42cb96d0ac9\") "
Nov 02 04:04:48.765032 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:48.764991    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4304f6cd-6a4b-40cf-985b-d42cb96d0ac9-openshift-service-ca" pod "4304f6cd-6a4b-40cf-985b-d42cb96d0ac9" (UID: "4304f6cd-6a4b-40cf-985b-d42cb96d0ac9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 04:04:48.766639 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:48.766611    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4304f6cd-6a4b-40cf-985b-d42cb96d0ac9-maas-api-tls" pod "4304f6cd-6a4b-40cf-985b-d42cb96d0ac9" (UID: "4304f6cd-6a4b-40cf-985b-d42cb96d0ac9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 04:04:48.766759 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:48.766689    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4304f6cd-6a4b-40cf-985b-d42cb96d0ac9-kube-api-access-bzfsw" pod "4304f6cd-6a4b-40cf-985b-d42cb96d0ac9" (UID: "4304f6cd-6a4b-40cf-985b-d42cb96d0ac9"). InnerVolumeSpecName "kube-api-access-bzfsw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 04:04:48.766759 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:48.766745    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4304f6cd-6a4b-40cf-985b-d42cb96d0ac9-metrics-certs" pod "4304f6cd-6a4b-40cf-985b-d42cb96d0ac9" (UID: "4304f6cd-6a4b-40cf-985b-d42cb96d0ac9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 04:04:48.866307 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:48.866275    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4304f6cd-6a4b-40cf-985b-d42cb96d0ac9-maas-api-tls\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:04:48.866307 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:48.866304    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzfsw\" (UniqueName: \"kubernetes.io/projected/4304f6cd-6a4b-40cf-985b-d42cb96d0ac9-kube-api-access-bzfsw\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:04:48.866307 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:48.866316    4496 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4304f6cd-6a4b-40cf-985b-d42cb96d0ac9-metrics-certs\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:04:48.866562 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:48.866326    4496 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4304f6cd-6a4b-40cf-985b-d42cb96d0ac9-openshift-service-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:04:48.963677 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:48.963577    4496 generic.go:356] "Generic (PLEG): container finished" podID="4304f6cd-6a4b-40cf-985b-d42cb96d0ac9" containerID="e6501811c5d8fe951fe82478192c592c3eecd26c9b2fb941ec3f12b34a3a9b98" exitCode=0
Nov 02 04:04:48.963677 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:48.963649    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-ac7ab4-547f75d6d4-dn58x"
Nov 02 04:04:48.963904 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:48.963693    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-ac7ab4-547f75d6d4-dn58x" event={"ID":"4304f6cd-6a4b-40cf-985b-d42cb96d0ac9","Type":"ContainerDied","Data":"e6501811c5d8fe951fe82478192c592c3eecd26c9b2fb941ec3f12b34a3a9b98"}
Nov 02 04:04:48.963904 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:48.963777    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-ac7ab4-547f75d6d4-dn58x" event={"ID":"4304f6cd-6a4b-40cf-985b-d42cb96d0ac9","Type":"ContainerDied","Data":"d11fa82fb3055f3e78ae2c43ce3f6b39f5aaa5268fedf761e6e96ff7552981b1"}
Nov 02 04:04:48.963904 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:48.963812    4496 scope.go:122] "RemoveContainer" containerID="e6501811c5d8fe951fe82478192c592c3eecd26c9b2fb941ec3f12b34a3a9b98"
Nov 02 04:04:48.975930 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:48.975911    4496 scope.go:122] "RemoveContainer" containerID="e6501811c5d8fe951fe82478192c592c3eecd26c9b2fb941ec3f12b34a3a9b98"
Nov 02 04:04:48.976250 ip-10-0-133-163 kubenswrapper[4496]: E1102 04:04:48.976230    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6501811c5d8fe951fe82478192c592c3eecd26c9b2fb941ec3f12b34a3a9b98\": container with ID starting with e6501811c5d8fe951fe82478192c592c3eecd26c9b2fb941ec3f12b34a3a9b98 not found: ID does not exist" containerID="e6501811c5d8fe951fe82478192c592c3eecd26c9b2fb941ec3f12b34a3a9b98"
Nov 02 04:04:48.976368 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:48.976255    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6501811c5d8fe951fe82478192c592c3eecd26c9b2fb941ec3f12b34a3a9b98"} err="failed to get container status \"e6501811c5d8fe951fe82478192c592c3eecd26c9b2fb941ec3f12b34a3a9b98\": rpc error: code = NotFound desc = could not find container \"e6501811c5d8fe951fe82478192c592c3eecd26c9b2fb941ec3f12b34a3a9b98\": container with ID starting with e6501811c5d8fe951fe82478192c592c3eecd26c9b2fb941ec3f12b34a3a9b98 not found: ID does not exist"
Nov 02 04:04:49.131635 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:49.131596    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-ac7ab4-547f75d6d4-dn58x"]
Nov 02 04:04:49.139060 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:49.137643    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-ac7ab4-547f75d6d4-dn58x"]
Nov 02 04:04:49.573646 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:49.573609    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4304f6cd-6a4b-40cf-985b-d42cb96d0ac9" path="/var/lib/kubelet/pods/4304f6cd-6a4b-40cf-985b-d42cb96d0ac9/volumes"
Nov 02 04:04:50.421361 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:50.421329    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-3f24d0-5d7d5dd778-xgccr"]
Nov 02 04:04:50.421970 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:50.421952    4496 cpu_manager.go:414] "containerMap: removing container" podUID="3ce4cf66-9b27-4031-aa2b-b2430b012b97" containerName="maas-api"
Nov 02 04:04:50.421970 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:50.421969    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ce4cf66-9b27-4031-aa2b-b2430b012b97" containerName="maas-api"
Nov 02 04:04:50.422179 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:50.421981    4496 cpu_manager.go:414] "containerMap: removing container" podUID="4304f6cd-6a4b-40cf-985b-d42cb96d0ac9" containerName="maas-api"
Nov 02 04:04:50.422179 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:50.421987    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4304f6cd-6a4b-40cf-985b-d42cb96d0ac9" containerName="maas-api"
Nov 02 04:04:50.422179 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:50.421998    4496 cpu_manager.go:414] "containerMap: removing container" podUID="85c3ba82-04d0-48f7-9da2-849ee385edc7" containerName="cleanup"
Nov 02 04:04:50.422179 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:50.422005    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85c3ba82-04d0-48f7-9da2-849ee385edc7" containerName="cleanup"
Nov 02 04:04:50.422179 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:50.422035    4496 cpu_manager.go:414] "containerMap: removing container" podUID="5215d837-30bb-43d6-81fa-6d50df5e45bf" containerName="payload-processing"
Nov 02 04:04:50.422179 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:50.422041    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5215d837-30bb-43d6-81fa-6d50df5e45bf" containerName="payload-processing"
Nov 02 04:04:50.422179 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:50.422051    4496 cpu_manager.go:414] "containerMap: removing container" podUID="1249045d-1bca-4013-a9f7-f3c139923ded" containerName="maas-api"
Nov 02 04:04:50.422179 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:50.422055    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1249045d-1bca-4013-a9f7-f3c139923ded" containerName="maas-api"
Nov 02 04:04:50.422179 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:50.422064    4496 cpu_manager.go:414] "containerMap: removing container" podUID="2046cb9f-964f-45f4-9f8a-e427b5818ffc" containerName="payload-pre-processing"
Nov 02 04:04:50.422179 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:50.422069    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2046cb9f-964f-45f4-9f8a-e427b5818ffc" containerName="payload-pre-processing"
Nov 02 04:04:50.422179 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:50.422082    4496 cpu_manager.go:414] "containerMap: removing container" podUID="725d05a7-caa3-4e07-be47-837ea9534075" containerName="payload-pre-processing"
Nov 02 04:04:50.422179 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:50.422090    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="725d05a7-caa3-4e07-be47-837ea9534075" containerName="payload-pre-processing"
Nov 02 04:04:50.422179 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:50.422101    4496 cpu_manager.go:414] "containerMap: removing container" podUID="9c4dbc8c-dcaf-4864-bcd7-0a8346d04d5b" containerName="maas-api"
Nov 02 04:04:50.422179 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:50.422105    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c4dbc8c-dcaf-4864-bcd7-0a8346d04d5b" containerName="maas-api"
Nov 02 04:04:50.422179 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:50.422117    4496 cpu_manager.go:414] "containerMap: removing container" podUID="1955f7a8-3b12-45cd-8059-7d701db7ac37" containerName="payload-pre-processing"
Nov 02 04:04:50.422179 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:50.422122    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1955f7a8-3b12-45cd-8059-7d701db7ac37" containerName="payload-pre-processing"
Nov 02 04:04:50.422179 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:50.422128    4496 cpu_manager.go:414] "containerMap: removing container" podUID="ab5ecfc3-592a-47ca-8acd-000d982544d5" containerName="payload-processing"
Nov 02 04:04:50.422179 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:50.422133    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab5ecfc3-592a-47ca-8acd-000d982544d5" containerName="payload-processing"
Nov 02 04:04:50.422179 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:50.422141    4496 cpu_manager.go:414] "containerMap: removing container" podUID="2d0cd262-c6a0-4b21-af3c-393ee8051f81" containerName="maas-api"
Nov 02 04:04:50.422179 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:50.422145    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d0cd262-c6a0-4b21-af3c-393ee8051f81" containerName="maas-api"
Nov 02 04:04:50.422179 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:50.422163    4496 cpu_manager.go:414] "containerMap: removing container" podUID="d147ea36-e9b2-42de-ac31-de7c5e03fe51" containerName="payload-processing"
Nov 02 04:04:50.422179 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:50.422170    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d147ea36-e9b2-42de-ac31-de7c5e03fe51" containerName="payload-processing"
Nov 02 04:04:50.422179 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:50.422182    4496 cpu_manager.go:414] "containerMap: removing container" podUID="7f80f210-8056-4a78-a27f-5166ab50c9d0" containerName="maas-api"
Nov 02 04:04:50.422179 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:50.422187    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f80f210-8056-4a78-a27f-5166ab50c9d0" containerName="maas-api"
Nov 02 04:04:50.423561 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:50.422198    4496 cpu_manager.go:414] "containerMap: removing container" podUID="5649b53a-018e-41be-8592-7d828b6ef19f" containerName="maas-api"
Nov 02 04:04:50.423561 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:50.422204    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5649b53a-018e-41be-8592-7d828b6ef19f" containerName="maas-api"
Nov 02 04:04:50.423561 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:50.422209    4496 cpu_manager.go:414] "containerMap: removing container" podUID="fc4e9a96-bcbe-4993-9509-3987dcedff81" containerName="maas-api"
Nov 02 04:04:50.423561 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:50.422215    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc4e9a96-bcbe-4993-9509-3987dcedff81" containerName="maas-api"
Nov 02 04:04:50.423561 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:50.422225    4496 cpu_manager.go:414] "containerMap: removing container" podUID="8ea63372-147c-486f-99d4-5d2a5390db50" containerName="payload-pre-processing"
Nov 02 04:04:50.423561 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:50.422230    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ea63372-147c-486f-99d4-5d2a5390db50" containerName="payload-pre-processing"
Nov 02 04:04:50.423561 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:50.422240    4496 cpu_manager.go:414] "containerMap: removing container" podUID="44122d78-2a1a-43bb-a123-4494b445d939" containerName="maas-api"
Nov 02 04:04:50.423561 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:50.422244    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44122d78-2a1a-43bb-a123-4494b445d939" containerName="maas-api"
Nov 02 04:04:50.423561 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:50.422251    4496 cpu_manager.go:414] "containerMap: removing container" podUID="a7fcc039-906f-40fd-8ffa-c244e4c2494c" containerName="maas-api"
Nov 02 04:04:50.423561 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:50.422255    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7fcc039-906f-40fd-8ffa-c244e4c2494c" containerName="maas-api"
Nov 02 04:04:50.423561 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:50.422262    4496 cpu_manager.go:414] "containerMap: removing container" podUID="ccb123ce-4b29-4f58-8201-34433a8bd0b0" containerName="payload-processing"
Nov 02 04:04:50.423561 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:50.422267    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccb123ce-4b29-4f58-8201-34433a8bd0b0" containerName="payload-processing"
Nov 02 04:04:50.423561 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:50.422272    4496 cpu_manager.go:414] "containerMap: removing container" podUID="9887dae0-cba1-4148-bce6-936ff305a1a9" containerName="payload-processing"
Nov 02 04:04:50.423561 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:50.422276    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9887dae0-cba1-4148-bce6-936ff305a1a9" containerName="payload-processing"
Nov 02 04:04:50.423561 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:50.422284    4496 cpu_manager.go:414] "containerMap: removing container" podUID="85c3ba82-04d0-48f7-9da2-849ee385edc7" containerName="cleanup"
Nov 02 04:04:50.423561 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:50.422288    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85c3ba82-04d0-48f7-9da2-849ee385edc7" containerName="cleanup"
Nov 02 04:04:50.423561 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:50.422392    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccb123ce-4b29-4f58-8201-34433a8bd0b0" containerName="payload-processing"
Nov 02 04:04:50.423561 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:50.422398    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="9887dae0-cba1-4148-bce6-936ff305a1a9" containerName="payload-processing"
Nov 02 04:04:50.423561 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:50.422403    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="85c3ba82-04d0-48f7-9da2-849ee385edc7" containerName="cleanup"
Nov 02 04:04:50.423561 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:50.422412    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ce4cf66-9b27-4031-aa2b-b2430b012b97" containerName="maas-api"
Nov 02 04:04:50.423561 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:50.422418    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="4304f6cd-6a4b-40cf-985b-d42cb96d0ac9" containerName="maas-api"
Nov 02 04:04:50.423561 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:50.422424    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="85c3ba82-04d0-48f7-9da2-849ee385edc7" containerName="cleanup"
Nov 02 04:04:50.423561 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:50.422429    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="5215d837-30bb-43d6-81fa-6d50df5e45bf" containerName="payload-processing"
Nov 02 04:04:50.423561 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:50.422434    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="1249045d-1bca-4013-a9f7-f3c139923ded" containerName="maas-api"
Nov 02 04:04:50.423561 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:50.422439    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="2046cb9f-964f-45f4-9f8a-e427b5818ffc" containerName="payload-pre-processing"
Nov 02 04:04:50.423561 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:50.422444    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="725d05a7-caa3-4e07-be47-837ea9534075" containerName="payload-pre-processing"
Nov 02 04:04:50.423561 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:50.422451    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c4dbc8c-dcaf-4864-bcd7-0a8346d04d5b" containerName="maas-api"
Nov 02 04:04:50.423561 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:50.422456    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="1955f7a8-3b12-45cd-8059-7d701db7ac37" containerName="payload-pre-processing"
Nov 02 04:04:50.423561 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:50.422461    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab5ecfc3-592a-47ca-8acd-000d982544d5" containerName="payload-processing"
Nov 02 04:04:50.423561 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:50.422466    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d0cd262-c6a0-4b21-af3c-393ee8051f81" containerName="maas-api"
Nov 02 04:04:50.423561 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:50.422472    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="d147ea36-e9b2-42de-ac31-de7c5e03fe51" containerName="payload-processing"
Nov 02 04:04:50.423561 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:50.422479    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f80f210-8056-4a78-a27f-5166ab50c9d0" containerName="maas-api"
Nov 02 04:04:50.423561 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:50.422489    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="5649b53a-018e-41be-8592-7d828b6ef19f" containerName="maas-api"
Nov 02 04:04:50.423561 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:50.422496    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc4e9a96-bcbe-4993-9509-3987dcedff81" containerName="maas-api"
Nov 02 04:04:50.423561 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:50.422503    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ea63372-147c-486f-99d4-5d2a5390db50" containerName="payload-pre-processing"
Nov 02 04:04:50.423561 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:50.422509    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="44122d78-2a1a-43bb-a123-4494b445d939" containerName="maas-api"
Nov 02 04:04:50.423561 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:50.422514    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7fcc039-906f-40fd-8ffa-c244e4c2494c" containerName="maas-api"
Nov 02 04:04:50.427263 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:50.427241    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3f24d0-5d7d5dd778-xgccr"
Nov 02 04:04:50.484650 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:50.484618    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-3f24d0-5d7d5dd778-xgccr"]
Nov 02 04:04:50.486952 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:50.486922    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dndjp\" (UniqueName: \"kubernetes.io/projected/7c022b24-eb29-42be-a94f-7c1df4216688-kube-api-access-dndjp\") pod \"maas-api-e2e-extmod-a-3f24d0-5d7d5dd778-xgccr\" (UID: \"7c022b24-eb29-42be-a94f-7c1df4216688\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3f24d0-5d7d5dd778-xgccr"
Nov 02 04:04:50.487145 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:50.487126    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c022b24-eb29-42be-a94f-7c1df4216688-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-3f24d0-5d7d5dd778-xgccr\" (UID: \"7c022b24-eb29-42be-a94f-7c1df4216688\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3f24d0-5d7d5dd778-xgccr"
Nov 02 04:04:50.487223 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:50.487201    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c022b24-eb29-42be-a94f-7c1df4216688-metrics-certs\") pod \"maas-api-e2e-extmod-a-3f24d0-5d7d5dd778-xgccr\" (UID: \"7c022b24-eb29-42be-a94f-7c1df4216688\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3f24d0-5d7d5dd778-xgccr"
Nov 02 04:04:50.487279 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:50.487244    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c022b24-eb29-42be-a94f-7c1df4216688-maas-api-tls\") pod \"maas-api-e2e-extmod-a-3f24d0-5d7d5dd778-xgccr\" (UID: \"7c022b24-eb29-42be-a94f-7c1df4216688\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3f24d0-5d7d5dd778-xgccr"
Nov 02 04:04:50.530755 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:50.530729    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-3f24d0-serving-cert\""
Nov 02 04:04:50.588213 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:50.588183    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c022b24-eb29-42be-a94f-7c1df4216688-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-3f24d0-5d7d5dd778-xgccr\" (UID: \"7c022b24-eb29-42be-a94f-7c1df4216688\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3f24d0-5d7d5dd778-xgccr"
Nov 02 04:04:50.588627 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:50.588228    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c022b24-eb29-42be-a94f-7c1df4216688-metrics-certs\") pod \"maas-api-e2e-extmod-a-3f24d0-5d7d5dd778-xgccr\" (UID: \"7c022b24-eb29-42be-a94f-7c1df4216688\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3f24d0-5d7d5dd778-xgccr"
Nov 02 04:04:50.588627 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:50.588252    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c022b24-eb29-42be-a94f-7c1df4216688-maas-api-tls\") pod \"maas-api-e2e-extmod-a-3f24d0-5d7d5dd778-xgccr\" (UID: \"7c022b24-eb29-42be-a94f-7c1df4216688\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3f24d0-5d7d5dd778-xgccr"
Nov 02 04:04:50.588627 ip-10-0-133-163 kubenswrapper[4496]: E1102 04:04:50.588355    4496 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-3f24d0-serving-cert: secret "maas-api-e2e-extmod-a-3f24d0-serving-cert" not found
Nov 02 04:04:50.588627 ip-10-0-133-163 kubenswrapper[4496]: E1102 04:04:50.588405    4496 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c022b24-eb29-42be-a94f-7c1df4216688-maas-api-tls podName:7c022b24-eb29-42be-a94f-7c1df4216688 nodeName:}" failed. No retries permitted until 2026-11-02 04:04:51.088386743 +0000 UTC m=+4631.666994114 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7c022b24-eb29-42be-a94f-7c1df4216688-maas-api-tls") pod "maas-api-e2e-extmod-a-3f24d0-5d7d5dd778-xgccr" (UID: "7c022b24-eb29-42be-a94f-7c1df4216688") : secret "maas-api-e2e-extmod-a-3f24d0-serving-cert" not found
Nov 02 04:04:50.588627 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:50.588423    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dndjp\" (UniqueName: \"kubernetes.io/projected/7c022b24-eb29-42be-a94f-7c1df4216688-kube-api-access-dndjp\") pod \"maas-api-e2e-extmod-a-3f24d0-5d7d5dd778-xgccr\" (UID: \"7c022b24-eb29-42be-a94f-7c1df4216688\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3f24d0-5d7d5dd778-xgccr"
Nov 02 04:04:50.588951 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:50.588929    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c022b24-eb29-42be-a94f-7c1df4216688-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-3f24d0-5d7d5dd778-xgccr\" (UID: \"7c022b24-eb29-42be-a94f-7c1df4216688\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3f24d0-5d7d5dd778-xgccr"
Nov 02 04:04:50.590621 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:50.590601    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c022b24-eb29-42be-a94f-7c1df4216688-metrics-certs\") pod \"maas-api-e2e-extmod-a-3f24d0-5d7d5dd778-xgccr\" (UID: \"7c022b24-eb29-42be-a94f-7c1df4216688\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3f24d0-5d7d5dd778-xgccr"
Nov 02 04:04:50.646644 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:50.646615    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dndjp\" (UniqueName: \"kubernetes.io/projected/7c022b24-eb29-42be-a94f-7c1df4216688-kube-api-access-dndjp\") pod \"maas-api-e2e-extmod-a-3f24d0-5d7d5dd778-xgccr\" (UID: \"7c022b24-eb29-42be-a94f-7c1df4216688\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3f24d0-5d7d5dd778-xgccr"
Nov 02 04:04:51.094380 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:51.094346    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c022b24-eb29-42be-a94f-7c1df4216688-maas-api-tls\") pod \"maas-api-e2e-extmod-a-3f24d0-5d7d5dd778-xgccr\" (UID: \"7c022b24-eb29-42be-a94f-7c1df4216688\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3f24d0-5d7d5dd778-xgccr"
Nov 02 04:04:51.096695 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:51.096669    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c022b24-eb29-42be-a94f-7c1df4216688-maas-api-tls\") pod \"maas-api-e2e-extmod-a-3f24d0-5d7d5dd778-xgccr\" (UID: \"7c022b24-eb29-42be-a94f-7c1df4216688\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3f24d0-5d7d5dd778-xgccr"
Nov 02 04:04:51.342147 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:51.342114    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3f24d0-5d7d5dd778-xgccr"
Nov 02 04:04:51.734177 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:51.734144    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-3f24d0-5d7d5dd778-xgccr"]
Nov 02 04:04:51.739361 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:51.738343    4496 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 04:04:52.000698 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:52.000611    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3f24d0-5d7d5dd778-xgccr" event={"ID":"7c022b24-eb29-42be-a94f-7c1df4216688","Type":"ContainerStarted","Data":"a5505046f68bcef6a8ff35b5817ec10b18efb277325c0a57a53ae7fbf5a1b848"}
Nov 02 04:04:53.009457 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:53.009418    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3f24d0-5d7d5dd778-xgccr" event={"ID":"7c022b24-eb29-42be-a94f-7c1df4216688","Type":"ContainerStarted","Data":"28f4d798bd970d538655a5c90b1e5bd4dea9c17529e569a71ca618c748e62ed3"}
Nov 02 04:04:53.009848 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:53.009497    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3f24d0-5d7d5dd778-xgccr"
Nov 02 04:04:53.127688 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:53.127635    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3f24d0-5d7d5dd778-xgccr" podStartSLOduration=2.220066452 podStartE2EDuration="3.1276226s" totalImagesPullingTime="907.556148ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 04:04:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 04:04:53.125942446 +0000 UTC m=+4633.704549827" watchObservedRunningTime="2026-11-02 04:04:53.1276226 +0000 UTC m=+4633.706229980"
Nov 02 04:04:57.210207 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:57.210171    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-ca6546/simulator-w2-ca6546-kserve-5959bb649b-5x6tf"]
Nov 02 04:04:59.030130 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:04:59.030096    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3f24d0-5d7d5dd778-xgccr"
Nov 02 04:05:00.794239 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:00.794200    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3b27a5-7d4d48687f-hq6zz"]
Nov 02 04:05:00.794648 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:00.794415    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3b27a5-7d4d48687f-hq6zz" podUID="51249edc-f28f-4f57-94cd-eca441b6c5c6" containerName="maas-api" containerID="cri-o://fb7547cd4d3400c22dc944ea4d18afb65d09d8dfce34e3fc4e43b13716192d92" gracePeriod=30
Nov 02 04:05:01.065416 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:01.065385    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3b27a5-7d4d48687f-hq6zz"
Nov 02 04:05:01.069089 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:01.069063    4496 generic.go:356] "Generic (PLEG): container finished" podID="51249edc-f28f-4f57-94cd-eca441b6c5c6" containerID="fb7547cd4d3400c22dc944ea4d18afb65d09d8dfce34e3fc4e43b13716192d92" exitCode=0
Nov 02 04:05:01.069245 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:01.069104    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3b27a5-7d4d48687f-hq6zz"
Nov 02 04:05:01.069245 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:01.069138    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3b27a5-7d4d48687f-hq6zz" event={"ID":"51249edc-f28f-4f57-94cd-eca441b6c5c6","Type":"ContainerDied","Data":"fb7547cd4d3400c22dc944ea4d18afb65d09d8dfce34e3fc4e43b13716192d92"}
Nov 02 04:05:01.069245 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:01.069192    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3b27a5-7d4d48687f-hq6zz" event={"ID":"51249edc-f28f-4f57-94cd-eca441b6c5c6","Type":"ContainerDied","Data":"349eda8df30c5496d9f5cf96f58813e09f606016f70e9c3f5b45c23fdf4b0689"}
Nov 02 04:05:01.069245 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:01.069214    4496 scope.go:122] "RemoveContainer" containerID="fb7547cd4d3400c22dc944ea4d18afb65d09d8dfce34e3fc4e43b13716192d92"
Nov 02 04:05:01.081373 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:01.081348    4496 scope.go:122] "RemoveContainer" containerID="fb7547cd4d3400c22dc944ea4d18afb65d09d8dfce34e3fc4e43b13716192d92"
Nov 02 04:05:01.081671 ip-10-0-133-163 kubenswrapper[4496]: E1102 04:05:01.081649    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb7547cd4d3400c22dc944ea4d18afb65d09d8dfce34e3fc4e43b13716192d92\": container with ID starting with fb7547cd4d3400c22dc944ea4d18afb65d09d8dfce34e3fc4e43b13716192d92 not found: ID does not exist" containerID="fb7547cd4d3400c22dc944ea4d18afb65d09d8dfce34e3fc4e43b13716192d92"
Nov 02 04:05:01.081718 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:01.081684    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb7547cd4d3400c22dc944ea4d18afb65d09d8dfce34e3fc4e43b13716192d92"} err="failed to get container status \"fb7547cd4d3400c22dc944ea4d18afb65d09d8dfce34e3fc4e43b13716192d92\": rpc error: code = NotFound desc = could not find container \"fb7547cd4d3400c22dc944ea4d18afb65d09d8dfce34e3fc4e43b13716192d92\": container with ID starting with fb7547cd4d3400c22dc944ea4d18afb65d09d8dfce34e3fc4e43b13716192d92 not found: ID does not exist"
Nov 02 04:05:01.206447 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:01.206409    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51249edc-f28f-4f57-94cd-eca441b6c5c6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51249edc-f28f-4f57-94cd-eca441b6c5c6-maas-api-tls\") pod \"51249edc-f28f-4f57-94cd-eca441b6c5c6\" (UID: \"51249edc-f28f-4f57-94cd-eca441b6c5c6\") "
Nov 02 04:05:01.206603 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:01.206575    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51249edc-f28f-4f57-94cd-eca441b6c5c6-kube-api-access-vx794\" (UniqueName: \"kubernetes.io/projected/51249edc-f28f-4f57-94cd-eca441b6c5c6-kube-api-access-vx794\") pod \"51249edc-f28f-4f57-94cd-eca441b6c5c6\" (UID: \"51249edc-f28f-4f57-94cd-eca441b6c5c6\") "
Nov 02 04:05:01.206671 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:01.206642    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51249edc-f28f-4f57-94cd-eca441b6c5c6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/51249edc-f28f-4f57-94cd-eca441b6c5c6-metrics-certs\") pod \"51249edc-f28f-4f57-94cd-eca441b6c5c6\" (UID: \"51249edc-f28f-4f57-94cd-eca441b6c5c6\") "
Nov 02 04:05:01.206737 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:01.206721    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51249edc-f28f-4f57-94cd-eca441b6c5c6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51249edc-f28f-4f57-94cd-eca441b6c5c6-openshift-service-ca\") pod \"51249edc-f28f-4f57-94cd-eca441b6c5c6\" (UID: \"51249edc-f28f-4f57-94cd-eca441b6c5c6\") "
Nov 02 04:05:01.207323 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:01.207293    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51249edc-f28f-4f57-94cd-eca441b6c5c6-openshift-service-ca" pod "51249edc-f28f-4f57-94cd-eca441b6c5c6" (UID: "51249edc-f28f-4f57-94cd-eca441b6c5c6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 04:05:01.208691 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:01.208662    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51249edc-f28f-4f57-94cd-eca441b6c5c6-maas-api-tls" pod "51249edc-f28f-4f57-94cd-eca441b6c5c6" (UID: "51249edc-f28f-4f57-94cd-eca441b6c5c6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 04:05:01.208787 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:01.208728    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51249edc-f28f-4f57-94cd-eca441b6c5c6-kube-api-access-vx794" pod "51249edc-f28f-4f57-94cd-eca441b6c5c6" (UID: "51249edc-f28f-4f57-94cd-eca441b6c5c6"). InnerVolumeSpecName "kube-api-access-vx794". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 04:05:01.209238 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:01.209212    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51249edc-f28f-4f57-94cd-eca441b6c5c6-metrics-certs" pod "51249edc-f28f-4f57-94cd-eca441b6c5c6" (UID: "51249edc-f28f-4f57-94cd-eca441b6c5c6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 04:05:01.308846 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:01.308758    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51249edc-f28f-4f57-94cd-eca441b6c5c6-maas-api-tls\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:05:01.308846 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:01.308789    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vx794\" (UniqueName: \"kubernetes.io/projected/51249edc-f28f-4f57-94cd-eca441b6c5c6-kube-api-access-vx794\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:05:01.308846 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:01.308801    4496 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51249edc-f28f-4f57-94cd-eca441b6c5c6-metrics-certs\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:05:01.308846 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:01.308810    4496 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51249edc-f28f-4f57-94cd-eca441b6c5c6-openshift-service-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:05:01.498173 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:01.498142    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3b27a5-7d4d48687f-hq6zz"]
Nov 02 04:05:01.502177 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:01.502150    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3b27a5-7d4d48687f-hq6zz"]
Nov 02 04:05:01.574177 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:01.574089    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51249edc-f28f-4f57-94cd-eca441b6c5c6" path="/var/lib/kubelet/pods/51249edc-f28f-4f57-94cd-eca441b6c5c6/volumes"
Nov 02 04:05:03.449484 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:03.449445    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-398828/distinct-2-w4-398828-kserve-545449d49b-6mg4b"]
Nov 02 04:05:03.500311 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:03.500278    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-398828/simulator-w4-398828-kserve-6d5bd49c6f-9tw2c"]
Nov 02 04:05:03.528762 ip-10-0-133-163 kubenswrapper[4496]: E1102 04:05:03.528730    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 04:05:04.574511 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:04.574475    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-ca6546-8d886cbbf-qx4dh"]
Nov 02 04:05:04.682694 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:04.682664    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-ca6546-76c6c456c9-vxsgp"]
Nov 02 04:05:09.575145 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:09.575109    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-ca6546-8d886cbbf-qx4dh" podUID="0fabbfdc-768a-49d7-a62d-2bf8680a5cc8" containerName="payload-processing" containerID="cri-o://541e7fab7b4167b8ccec28cd967e41b7bb3ae7c1e42a7e04d4b97d385f1b8fa4" gracePeriod=25
Nov 02 04:05:09.683785 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:09.683744    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ca6546-76c6c456c9-vxsgp" podUID="57fc7cac-886f-4b70-86ee-6f97a711531a" containerName="payload-pre-processing" containerID="cri-o://8bb2eb3c9f6d6d1bd9c14597c4210682e67a1cf29c335b66ca3fede9c3561db2" gracePeriod=25
Nov 02 04:05:09.904582 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:09.904553    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-ca6546-8d886cbbf-qx4dh"
Nov 02 04:05:09.987971 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:09.987948    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ca6546-76c6c456c9-vxsgp"
Nov 02 04:05:10.009389 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:10.009366    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fabbfdc-768a-49d7-a62d-2bf8680a5cc8-kube-api-access-q8kdw\" (UniqueName: \"kubernetes.io/projected/0fabbfdc-768a-49d7-a62d-2bf8680a5cc8-kube-api-access-q8kdw\") pod \"0fabbfdc-768a-49d7-a62d-2bf8680a5cc8\" (UID: \"0fabbfdc-768a-49d7-a62d-2bf8680a5cc8\") "
Nov 02 04:05:10.009574 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:10.009556    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fabbfdc-768a-49d7-a62d-2bf8680a5cc8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fabbfdc-768a-49d7-a62d-2bf8680a5cc8-plugins-config-volume\") pod \"0fabbfdc-768a-49d7-a62d-2bf8680a5cc8\" (UID: \"0fabbfdc-768a-49d7-a62d-2bf8680a5cc8\") "
Nov 02 04:05:10.009969 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:10.009945    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fabbfdc-768a-49d7-a62d-2bf8680a5cc8-plugins-config-volume" pod "0fabbfdc-768a-49d7-a62d-2bf8680a5cc8" (UID: "0fabbfdc-768a-49d7-a62d-2bf8680a5cc8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 04:05:10.011379 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:10.011357    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fabbfdc-768a-49d7-a62d-2bf8680a5cc8-kube-api-access-q8kdw" pod "0fabbfdc-768a-49d7-a62d-2bf8680a5cc8" (UID: "0fabbfdc-768a-49d7-a62d-2bf8680a5cc8"). InnerVolumeSpecName "kube-api-access-q8kdw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 04:05:10.110879 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:10.110847    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57fc7cac-886f-4b70-86ee-6f97a711531a-kube-api-access-kzrgb\" (UniqueName: \"kubernetes.io/projected/57fc7cac-886f-4b70-86ee-6f97a711531a-kube-api-access-kzrgb\") pod \"57fc7cac-886f-4b70-86ee-6f97a711531a\" (UID: \"57fc7cac-886f-4b70-86ee-6f97a711531a\") "
Nov 02 04:05:10.111080 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:10.111005    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57fc7cac-886f-4b70-86ee-6f97a711531a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57fc7cac-886f-4b70-86ee-6f97a711531a-plugins-config-volume\") pod \"57fc7cac-886f-4b70-86ee-6f97a711531a\" (UID: \"57fc7cac-886f-4b70-86ee-6f97a711531a\") "
Nov 02 04:05:10.111299 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:10.111283    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8kdw\" (UniqueName: \"kubernetes.io/projected/0fabbfdc-768a-49d7-a62d-2bf8680a5cc8-kube-api-access-q8kdw\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:05:10.111343 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:10.111305    4496 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fabbfdc-768a-49d7-a62d-2bf8680a5cc8-plugins-config-volume\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:05:10.111343 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:10.111323    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57fc7cac-886f-4b70-86ee-6f97a711531a-plugins-config-volume" pod "57fc7cac-886f-4b70-86ee-6f97a711531a" (UID: "57fc7cac-886f-4b70-86ee-6f97a711531a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 04:05:10.113045 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:10.113010    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57fc7cac-886f-4b70-86ee-6f97a711531a-kube-api-access-kzrgb" pod "57fc7cac-886f-4b70-86ee-6f97a711531a" (UID: "57fc7cac-886f-4b70-86ee-6f97a711531a"). InnerVolumeSpecName "kube-api-access-kzrgb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 04:05:10.135397 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:10.135324    4496 generic.go:356] "Generic (PLEG): container finished" podID="0fabbfdc-768a-49d7-a62d-2bf8680a5cc8" containerID="541e7fab7b4167b8ccec28cd967e41b7bb3ae7c1e42a7e04d4b97d385f1b8fa4" exitCode=0
Nov 02 04:05:10.135397 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:10.135380    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-ca6546-8d886cbbf-qx4dh"
Nov 02 04:05:10.135574 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:10.135399    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-ca6546-8d886cbbf-qx4dh" event={"ID":"0fabbfdc-768a-49d7-a62d-2bf8680a5cc8","Type":"ContainerDied","Data":"541e7fab7b4167b8ccec28cd967e41b7bb3ae7c1e42a7e04d4b97d385f1b8fa4"}
Nov 02 04:05:10.135574 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:10.135459    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-ca6546-8d886cbbf-qx4dh" event={"ID":"0fabbfdc-768a-49d7-a62d-2bf8680a5cc8","Type":"ContainerDied","Data":"a2b82b13393601331dbf1bba50e8af82c80b6526f2ca90a3ad556cf949b46190"}
Nov 02 04:05:10.135574 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:10.135483    4496 scope.go:122] "RemoveContainer" containerID="541e7fab7b4167b8ccec28cd967e41b7bb3ae7c1e42a7e04d4b97d385f1b8fa4"
Nov 02 04:05:10.136995 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:10.136974    4496 generic.go:356] "Generic (PLEG): container finished" podID="57fc7cac-886f-4b70-86ee-6f97a711531a" containerID="8bb2eb3c9f6d6d1bd9c14597c4210682e67a1cf29c335b66ca3fede9c3561db2" exitCode=0
Nov 02 04:05:10.137131 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:10.137052    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ca6546-76c6c456c9-vxsgp"
Nov 02 04:05:10.137131 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:10.137065    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ca6546-76c6c456c9-vxsgp" event={"ID":"57fc7cac-886f-4b70-86ee-6f97a711531a","Type":"ContainerDied","Data":"8bb2eb3c9f6d6d1bd9c14597c4210682e67a1cf29c335b66ca3fede9c3561db2"}
Nov 02 04:05:10.137131 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:10.137126    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ca6546-76c6c456c9-vxsgp" event={"ID":"57fc7cac-886f-4b70-86ee-6f97a711531a","Type":"ContainerDied","Data":"355642174b8498c4b63bfe08b5611c64ba1bfe6e8a84318d97eace49b90eda4b"}
Nov 02 04:05:10.158010 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:10.148285    4496 scope.go:122] "RemoveContainer" containerID="541e7fab7b4167b8ccec28cd967e41b7bb3ae7c1e42a7e04d4b97d385f1b8fa4"
Nov 02 04:05:10.158010 ip-10-0-133-163 kubenswrapper[4496]: E1102 04:05:10.148672    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"541e7fab7b4167b8ccec28cd967e41b7bb3ae7c1e42a7e04d4b97d385f1b8fa4\": container with ID starting with 541e7fab7b4167b8ccec28cd967e41b7bb3ae7c1e42a7e04d4b97d385f1b8fa4 not found: ID does not exist" containerID="541e7fab7b4167b8ccec28cd967e41b7bb3ae7c1e42a7e04d4b97d385f1b8fa4"
Nov 02 04:05:10.158010 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:10.148713    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"541e7fab7b4167b8ccec28cd967e41b7bb3ae7c1e42a7e04d4b97d385f1b8fa4"} err="failed to get container status \"541e7fab7b4167b8ccec28cd967e41b7bb3ae7c1e42a7e04d4b97d385f1b8fa4\": rpc error: code = NotFound desc = could not find container \"541e7fab7b4167b8ccec28cd967e41b7bb3ae7c1e42a7e04d4b97d385f1b8fa4\": container with ID starting with 541e7fab7b4167b8ccec28cd967e41b7bb3ae7c1e42a7e04d4b97d385f1b8fa4 not found: ID does not exist"
Nov 02 04:05:10.158010 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:10.148732    4496 scope.go:122] "RemoveContainer" containerID="8bb2eb3c9f6d6d1bd9c14597c4210682e67a1cf29c335b66ca3fede9c3561db2"
Nov 02 04:05:10.159193 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:10.159164    4496 scope.go:122] "RemoveContainer" containerID="8bb2eb3c9f6d6d1bd9c14597c4210682e67a1cf29c335b66ca3fede9c3561db2"
Nov 02 04:05:10.159467 ip-10-0-133-163 kubenswrapper[4496]: E1102 04:05:10.159449    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bb2eb3c9f6d6d1bd9c14597c4210682e67a1cf29c335b66ca3fede9c3561db2\": container with ID starting with 8bb2eb3c9f6d6d1bd9c14597c4210682e67a1cf29c335b66ca3fede9c3561db2 not found: ID does not exist" containerID="8bb2eb3c9f6d6d1bd9c14597c4210682e67a1cf29c335b66ca3fede9c3561db2"
Nov 02 04:05:10.159554 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:10.159472    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bb2eb3c9f6d6d1bd9c14597c4210682e67a1cf29c335b66ca3fede9c3561db2"} err="failed to get container status \"8bb2eb3c9f6d6d1bd9c14597c4210682e67a1cf29c335b66ca3fede9c3561db2\": rpc error: code = NotFound desc = could not find container \"8bb2eb3c9f6d6d1bd9c14597c4210682e67a1cf29c335b66ca3fede9c3561db2\": container with ID starting with 8bb2eb3c9f6d6d1bd9c14597c4210682e67a1cf29c335b66ca3fede9c3561db2 not found: ID does not exist"
Nov 02 04:05:10.212472 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:10.212438    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzrgb\" (UniqueName: \"kubernetes.io/projected/57fc7cac-886f-4b70-86ee-6f97a711531a-kube-api-access-kzrgb\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:05:10.212472 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:10.212466    4496 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57fc7cac-886f-4b70-86ee-6f97a711531a-plugins-config-volume\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:05:10.368107 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:10.368066    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-ca6546-8d886cbbf-qx4dh"]
Nov 02 04:05:10.371782 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:10.371754    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-ca6546-8d886cbbf-qx4dh"]
Nov 02 04:05:10.461936 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:10.461855    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-398828-54955d6b75-czhwj"]
Nov 02 04:05:10.482278 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:10.482250    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-ca6546-76c6c456c9-vxsgp"]
Nov 02 04:05:10.484928 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:10.484904    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-ca6546-76c6c456c9-vxsgp"]
Nov 02 04:05:10.566121 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:10.566094    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-398828-75678b79b7-bbqqw"]
Nov 02 04:05:11.574307 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:11.574273    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fabbfdc-768a-49d7-a62d-2bf8680a5cc8" path="/var/lib/kubelet/pods/0fabbfdc-768a-49d7-a62d-2bf8680a5cc8/volumes"
Nov 02 04:05:11.574698 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:11.574659    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57fc7cac-886f-4b70-86ee-6f97a711531a" path="/var/lib/kubelet/pods/57fc7cac-886f-4b70-86ee-6f97a711531a/volumes"
Nov 02 04:05:12.231826 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:12.231780    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-ca6546/simulator-w2-ca6546-kserve-5959bb649b-5x6tf" podUID="4fd6a8f7-0b10-4009-a6db-de0a01aca2b8" containerName="main" containerID="cri-o://b0946945efb6dcfac7ea268630e2fdc0154b942c5f158bb1a50e0810520623b2" gracePeriod=45
Nov 02 04:05:12.486127 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:12.486062    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-ca6546/simulator-w2-ca6546-kserve-5959bb649b-5x6tf"
Nov 02 04:05:12.538147 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:12.538114    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fd6a8f7-0b10-4009-a6db-de0a01aca2b8-kube-api-access-n7zxp\" (UniqueName: \"kubernetes.io/projected/4fd6a8f7-0b10-4009-a6db-de0a01aca2b8-kube-api-access-n7zxp\") pod \"4fd6a8f7-0b10-4009-a6db-de0a01aca2b8\" (UID: \"4fd6a8f7-0b10-4009-a6db-de0a01aca2b8\") "
Nov 02 04:05:12.538300 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:12.538191    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fd6a8f7-0b10-4009-a6db-de0a01aca2b8-home\" (UniqueName: \"kubernetes.io/empty-dir/4fd6a8f7-0b10-4009-a6db-de0a01aca2b8-home\") pod \"4fd6a8f7-0b10-4009-a6db-de0a01aca2b8\" (UID: \"4fd6a8f7-0b10-4009-a6db-de0a01aca2b8\") "
Nov 02 04:05:12.538300 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:12.538231    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fd6a8f7-0b10-4009-a6db-de0a01aca2b8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fd6a8f7-0b10-4009-a6db-de0a01aca2b8-tmp-dir\") pod \"4fd6a8f7-0b10-4009-a6db-de0a01aca2b8\" (UID: \"4fd6a8f7-0b10-4009-a6db-de0a01aca2b8\") "
Nov 02 04:05:12.538434 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:12.538307    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fd6a8f7-0b10-4009-a6db-de0a01aca2b8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fd6a8f7-0b10-4009-a6db-de0a01aca2b8-model-cache\") pod \"4fd6a8f7-0b10-4009-a6db-de0a01aca2b8\" (UID: \"4fd6a8f7-0b10-4009-a6db-de0a01aca2b8\") "
Nov 02 04:05:12.538434 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:12.538354    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fd6a8f7-0b10-4009-a6db-de0a01aca2b8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fd6a8f7-0b10-4009-a6db-de0a01aca2b8-dshm\") pod \"4fd6a8f7-0b10-4009-a6db-de0a01aca2b8\" (UID: \"4fd6a8f7-0b10-4009-a6db-de0a01aca2b8\") "
Nov 02 04:05:12.538434 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:12.538419    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4fd6a8f7-0b10-4009-a6db-de0a01aca2b8-tls-certs\" (UniqueName: \"kubernetes.io/secret/4fd6a8f7-0b10-4009-a6db-de0a01aca2b8-tls-certs\") pod \"4fd6a8f7-0b10-4009-a6db-de0a01aca2b8\" (UID: \"4fd6a8f7-0b10-4009-a6db-de0a01aca2b8\") "
Nov 02 04:05:12.538649 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:12.538497    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fd6a8f7-0b10-4009-a6db-de0a01aca2b8-tmp-dir" pod "4fd6a8f7-0b10-4009-a6db-de0a01aca2b8" (UID: "4fd6a8f7-0b10-4009-a6db-de0a01aca2b8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 04:05:12.538649 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:12.538486    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fd6a8f7-0b10-4009-a6db-de0a01aca2b8-home" pod "4fd6a8f7-0b10-4009-a6db-de0a01aca2b8" (UID: "4fd6a8f7-0b10-4009-a6db-de0a01aca2b8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 04:05:12.538796 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:12.538695    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fd6a8f7-0b10-4009-a6db-de0a01aca2b8-model-cache" pod "4fd6a8f7-0b10-4009-a6db-de0a01aca2b8" (UID: "4fd6a8f7-0b10-4009-a6db-de0a01aca2b8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 04:05:12.538907 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:12.538872    4496 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fd6a8f7-0b10-4009-a6db-de0a01aca2b8-home\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:05:12.538907 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:12.538895    4496 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fd6a8f7-0b10-4009-a6db-de0a01aca2b8-tmp-dir\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:05:12.539053 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:12.538908    4496 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fd6a8f7-0b10-4009-a6db-de0a01aca2b8-model-cache\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:05:12.540705 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:12.540674    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fd6a8f7-0b10-4009-a6db-de0a01aca2b8-dshm" pod "4fd6a8f7-0b10-4009-a6db-de0a01aca2b8" (UID: "4fd6a8f7-0b10-4009-a6db-de0a01aca2b8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 04:05:12.541086 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:12.541063    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fd6a8f7-0b10-4009-a6db-de0a01aca2b8-kube-api-access-n7zxp" pod "4fd6a8f7-0b10-4009-a6db-de0a01aca2b8" (UID: "4fd6a8f7-0b10-4009-a6db-de0a01aca2b8"). InnerVolumeSpecName "kube-api-access-n7zxp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 04:05:12.541185 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:12.541095    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fd6a8f7-0b10-4009-a6db-de0a01aca2b8-tls-certs" pod "4fd6a8f7-0b10-4009-a6db-de0a01aca2b8" (UID: "4fd6a8f7-0b10-4009-a6db-de0a01aca2b8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 04:05:12.640411 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:12.640368    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7zxp\" (UniqueName: \"kubernetes.io/projected/4fd6a8f7-0b10-4009-a6db-de0a01aca2b8-kube-api-access-n7zxp\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:05:12.640411 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:12.640403    4496 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fd6a8f7-0b10-4009-a6db-de0a01aca2b8-dshm\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:05:12.641821 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:12.640467    4496 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fd6a8f7-0b10-4009-a6db-de0a01aca2b8-tls-certs\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:05:13.160736 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:13.160699    4496 generic.go:356] "Generic (PLEG): container finished" podID="4fd6a8f7-0b10-4009-a6db-de0a01aca2b8" containerID="b0946945efb6dcfac7ea268630e2fdc0154b942c5f158bb1a50e0810520623b2" exitCode=0
Nov 02 04:05:13.160959 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:13.160745    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-ca6546/simulator-w2-ca6546-kserve-5959bb649b-5x6tf"
Nov 02 04:05:13.160959 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:13.160791    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-ca6546/simulator-w2-ca6546-kserve-5959bb649b-5x6tf" event={"ID":"4fd6a8f7-0b10-4009-a6db-de0a01aca2b8","Type":"ContainerDied","Data":"b0946945efb6dcfac7ea268630e2fdc0154b942c5f158bb1a50e0810520623b2"}
Nov 02 04:05:13.160959 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:13.160846    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-ca6546/simulator-w2-ca6546-kserve-5959bb649b-5x6tf" event={"ID":"4fd6a8f7-0b10-4009-a6db-de0a01aca2b8","Type":"ContainerDied","Data":"95bf30c0c02bd5edbfa86047913f2a33dbf65d31775826e5b77ffdba3a035500"}
Nov 02 04:05:13.160959 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:13.160871    4496 scope.go:122] "RemoveContainer" containerID="b0946945efb6dcfac7ea268630e2fdc0154b942c5f158bb1a50e0810520623b2"
Nov 02 04:05:13.171523 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:13.171506    4496 scope.go:122] "RemoveContainer" containerID="b0946945efb6dcfac7ea268630e2fdc0154b942c5f158bb1a50e0810520623b2"
Nov 02 04:05:13.171797 ip-10-0-133-163 kubenswrapper[4496]: E1102 04:05:13.171778    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0946945efb6dcfac7ea268630e2fdc0154b942c5f158bb1a50e0810520623b2\": container with ID starting with b0946945efb6dcfac7ea268630e2fdc0154b942c5f158bb1a50e0810520623b2 not found: ID does not exist" containerID="b0946945efb6dcfac7ea268630e2fdc0154b942c5f158bb1a50e0810520623b2"
Nov 02 04:05:13.171847 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:13.171803    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0946945efb6dcfac7ea268630e2fdc0154b942c5f158bb1a50e0810520623b2"} err="failed to get container status \"b0946945efb6dcfac7ea268630e2fdc0154b942c5f158bb1a50e0810520623b2\": rpc error: code = NotFound desc = could not find container \"b0946945efb6dcfac7ea268630e2fdc0154b942c5f158bb1a50e0810520623b2\": container with ID starting with b0946945efb6dcfac7ea268630e2fdc0154b942c5f158bb1a50e0810520623b2 not found: ID does not exist"
Nov 02 04:05:13.328540 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:13.328506    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-ca6546/simulator-w2-ca6546-kserve-5959bb649b-5x6tf"]
Nov 02 04:05:13.379072 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:13.379040    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-ca6546/simulator-w2-ca6546-kserve-5959bb649b-5x6tf"]
Nov 02 04:05:13.417150 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:13.417068    4496 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ca6546/simulator-w2-ca6546-kserve-5959bb649b-5x6tf" podUID="4fd6a8f7-0b10-4009-a6db-de0a01aca2b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": context deadline exceeded"
Nov 02 04:05:13.574516 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:13.574480    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fd6a8f7-0b10-4009-a6db-de0a01aca2b8" path="/var/lib/kubelet/pods/4fd6a8f7-0b10-4009-a6db-de0a01aca2b8/volumes"
Nov 02 04:05:15.145552 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:15.145510    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-5ac93f/distinct-w1-5ac93f-kserve-64697cb8c9-vj9sk"]
Nov 02 04:05:15.462890 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:15.462787    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-398828-54955d6b75-czhwj" podUID="ba5a7a35-1ea6-48e9-8817-dd7a7f6f08fc" containerName="payload-processing" containerID="cri-o://2cb2c85fa4ee35903735c777ca2c32e2a41edb03c856637dcc5b1398897639ba" gracePeriod=25
Nov 02 04:05:15.567250 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:15.567208    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-398828-75678b79b7-bbqqw" podUID="ce91e365-c823-4791-900a-c3bb3ccd4d36" containerName="payload-pre-processing" containerID="cri-o://cfbc8cee8ca607ed8ec1ba68b6374a79b6831788b1e73d2fff895294fa0cbfc5" gracePeriod=25
Nov 02 04:05:18.472962 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:18.472910    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-398828/distinct-2-w4-398828-kserve-545449d49b-6mg4b" podUID="36c1912c-2ab1-40c8-a09d-2425ab08ec8d" containerName="main" containerID="cri-o://c415850ad33f904fe4b236a1dff920b58a59368ea86dbd194172fde21cc3214f" gracePeriod=45
Nov 02 04:05:18.520634 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:18.520599    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-398828/simulator-w4-398828-kserve-6d5bd49c6f-9tw2c" podUID="5b6aecfa-3e3b-4fb3-a80c-35b77de60ce2" containerName="main" containerID="cri-o://3931bdaf7ea78a43290417fc714792f90b1af645570ea3a027e5faa7ccee9639" gracePeriod=45
Nov 02 04:05:18.795463 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:18.795434    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-398828/distinct-2-w4-398828-kserve-545449d49b-6mg4b"
Nov 02 04:05:18.813346 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:18.813321    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-398828/simulator-w4-398828-kserve-6d5bd49c6f-9tw2c"
Nov 02 04:05:18.907496 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:18.907463    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36c1912c-2ab1-40c8-a09d-2425ab08ec8d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/36c1912c-2ab1-40c8-a09d-2425ab08ec8d-dshm\") pod \"36c1912c-2ab1-40c8-a09d-2425ab08ec8d\" (UID: \"36c1912c-2ab1-40c8-a09d-2425ab08ec8d\") "
Nov 02 04:05:18.907496 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:18.907504    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36c1912c-2ab1-40c8-a09d-2425ab08ec8d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36c1912c-2ab1-40c8-a09d-2425ab08ec8d-tmp-dir\") pod \"36c1912c-2ab1-40c8-a09d-2425ab08ec8d\" (UID: \"36c1912c-2ab1-40c8-a09d-2425ab08ec8d\") "
Nov 02 04:05:18.907753 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:18.907531    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b6aecfa-3e3b-4fb3-a80c-35b77de60ce2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b6aecfa-3e3b-4fb3-a80c-35b77de60ce2-tmp-dir\") pod \"5b6aecfa-3e3b-4fb3-a80c-35b77de60ce2\" (UID: \"5b6aecfa-3e3b-4fb3-a80c-35b77de60ce2\") "
Nov 02 04:05:18.907753 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:18.907556    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36c1912c-2ab1-40c8-a09d-2425ab08ec8d-kube-api-access-fn527\" (UniqueName: \"kubernetes.io/projected/36c1912c-2ab1-40c8-a09d-2425ab08ec8d-kube-api-access-fn527\") pod \"36c1912c-2ab1-40c8-a09d-2425ab08ec8d\" (UID: \"36c1912c-2ab1-40c8-a09d-2425ab08ec8d\") "
Nov 02 04:05:18.907753 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:18.907675    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b6aecfa-3e3b-4fb3-a80c-35b77de60ce2-home\" (UniqueName: \"kubernetes.io/empty-dir/5b6aecfa-3e3b-4fb3-a80c-35b77de60ce2-home\") pod \"5b6aecfa-3e3b-4fb3-a80c-35b77de60ce2\" (UID: \"5b6aecfa-3e3b-4fb3-a80c-35b77de60ce2\") "
Nov 02 04:05:18.907753 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:18.907742    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b6aecfa-3e3b-4fb3-a80c-35b77de60ce2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b6aecfa-3e3b-4fb3-a80c-35b77de60ce2-model-cache\") pod \"5b6aecfa-3e3b-4fb3-a80c-35b77de60ce2\" (UID: \"5b6aecfa-3e3b-4fb3-a80c-35b77de60ce2\") "
Nov 02 04:05:18.907969 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:18.907783    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b6aecfa-3e3b-4fb3-a80c-35b77de60ce2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b6aecfa-3e3b-4fb3-a80c-35b77de60ce2-dshm\") pod \"5b6aecfa-3e3b-4fb3-a80c-35b77de60ce2\" (UID: \"5b6aecfa-3e3b-4fb3-a80c-35b77de60ce2\") "
Nov 02 04:05:18.907969 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:18.907820    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b6aecfa-3e3b-4fb3-a80c-35b77de60ce2-kube-api-access-rh2f2\" (UniqueName: \"kubernetes.io/projected/5b6aecfa-3e3b-4fb3-a80c-35b77de60ce2-kube-api-access-rh2f2\") pod \"5b6aecfa-3e3b-4fb3-a80c-35b77de60ce2\" (UID: \"5b6aecfa-3e3b-4fb3-a80c-35b77de60ce2\") "
Nov 02 04:05:18.907969 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:18.907885    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b6aecfa-3e3b-4fb3-a80c-35b77de60ce2-tls-certs\" (UniqueName: \"kubernetes.io/secret/5b6aecfa-3e3b-4fb3-a80c-35b77de60ce2-tls-certs\") pod \"5b6aecfa-3e3b-4fb3-a80c-35b77de60ce2\" (UID: \"5b6aecfa-3e3b-4fb3-a80c-35b77de60ce2\") "
Nov 02 04:05:18.907969 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:18.907952    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36c1912c-2ab1-40c8-a09d-2425ab08ec8d-tls-certs\" (UniqueName: \"kubernetes.io/secret/36c1912c-2ab1-40c8-a09d-2425ab08ec8d-tls-certs\") pod \"36c1912c-2ab1-40c8-a09d-2425ab08ec8d\" (UID: \"36c1912c-2ab1-40c8-a09d-2425ab08ec8d\") "
Nov 02 04:05:18.908218 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:18.907979    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36c1912c-2ab1-40c8-a09d-2425ab08ec8d-home\" (UniqueName: \"kubernetes.io/empty-dir/36c1912c-2ab1-40c8-a09d-2425ab08ec8d-home\") pod \"36c1912c-2ab1-40c8-a09d-2425ab08ec8d\" (UID: \"36c1912c-2ab1-40c8-a09d-2425ab08ec8d\") "
Nov 02 04:05:18.908218 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:18.908083    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36c1912c-2ab1-40c8-a09d-2425ab08ec8d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36c1912c-2ab1-40c8-a09d-2425ab08ec8d-model-cache\") pod \"36c1912c-2ab1-40c8-a09d-2425ab08ec8d\" (UID: \"36c1912c-2ab1-40c8-a09d-2425ab08ec8d\") "
Nov 02 04:05:18.908360 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:18.908330    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b6aecfa-3e3b-4fb3-a80c-35b77de60ce2-tmp-dir" pod "5b6aecfa-3e3b-4fb3-a80c-35b77de60ce2" (UID: "5b6aecfa-3e3b-4fb3-a80c-35b77de60ce2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 04:05:18.908503 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:18.908448    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b6aecfa-3e3b-4fb3-a80c-35b77de60ce2-home" pod "5b6aecfa-3e3b-4fb3-a80c-35b77de60ce2" (UID: "5b6aecfa-3e3b-4fb3-a80c-35b77de60ce2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 04:05:18.908503 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:18.908470    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b6aecfa-3e3b-4fb3-a80c-35b77de60ce2-model-cache" pod "5b6aecfa-3e3b-4fb3-a80c-35b77de60ce2" (UID: "5b6aecfa-3e3b-4fb3-a80c-35b77de60ce2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 04:05:18.908667 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:18.908628    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36c1912c-2ab1-40c8-a09d-2425ab08ec8d-tmp-dir" pod "36c1912c-2ab1-40c8-a09d-2425ab08ec8d" (UID: "36c1912c-2ab1-40c8-a09d-2425ab08ec8d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 04:05:18.908667 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:18.908652    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36c1912c-2ab1-40c8-a09d-2425ab08ec8d-model-cache" pod "36c1912c-2ab1-40c8-a09d-2425ab08ec8d" (UID: "36c1912c-2ab1-40c8-a09d-2425ab08ec8d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 04:05:18.908826 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:18.908809    4496 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b6aecfa-3e3b-4fb3-a80c-35b77de60ce2-home\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:05:18.908889 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:18.908834    4496 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b6aecfa-3e3b-4fb3-a80c-35b77de60ce2-model-cache\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:05:18.908889 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:18.908840    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36c1912c-2ab1-40c8-a09d-2425ab08ec8d-home" pod "36c1912c-2ab1-40c8-a09d-2425ab08ec8d" (UID: "36c1912c-2ab1-40c8-a09d-2425ab08ec8d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 04:05:18.908889 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:18.908849    4496 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36c1912c-2ab1-40c8-a09d-2425ab08ec8d-model-cache\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:05:18.909167 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:18.908886    4496 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36c1912c-2ab1-40c8-a09d-2425ab08ec8d-tmp-dir\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:05:18.909167 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:18.908904    4496 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b6aecfa-3e3b-4fb3-a80c-35b77de60ce2-tmp-dir\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:05:18.913484 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:18.913455    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36c1912c-2ab1-40c8-a09d-2425ab08ec8d-dshm" pod "36c1912c-2ab1-40c8-a09d-2425ab08ec8d" (UID: "36c1912c-2ab1-40c8-a09d-2425ab08ec8d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 04:05:18.913484 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:18.913477    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b6aecfa-3e3b-4fb3-a80c-35b77de60ce2-tls-certs" pod "5b6aecfa-3e3b-4fb3-a80c-35b77de60ce2" (UID: "5b6aecfa-3e3b-4fb3-a80c-35b77de60ce2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 04:05:18.913685 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:18.913458    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b6aecfa-3e3b-4fb3-a80c-35b77de60ce2-dshm" pod "5b6aecfa-3e3b-4fb3-a80c-35b77de60ce2" (UID: "5b6aecfa-3e3b-4fb3-a80c-35b77de60ce2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 04:05:18.913685 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:18.913510    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b6aecfa-3e3b-4fb3-a80c-35b77de60ce2-kube-api-access-rh2f2" pod "5b6aecfa-3e3b-4fb3-a80c-35b77de60ce2" (UID: "5b6aecfa-3e3b-4fb3-a80c-35b77de60ce2"). InnerVolumeSpecName "kube-api-access-rh2f2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 04:05:18.913685 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:18.913580    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36c1912c-2ab1-40c8-a09d-2425ab08ec8d-tls-certs" pod "36c1912c-2ab1-40c8-a09d-2425ab08ec8d" (UID: "36c1912c-2ab1-40c8-a09d-2425ab08ec8d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 04:05:18.913881 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:18.913842    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36c1912c-2ab1-40c8-a09d-2425ab08ec8d-kube-api-access-fn527" pod "36c1912c-2ab1-40c8-a09d-2425ab08ec8d" (UID: "36c1912c-2ab1-40c8-a09d-2425ab08ec8d"). InnerVolumeSpecName "kube-api-access-fn527". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 04:05:19.010297 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:19.010217    4496 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b6aecfa-3e3b-4fb3-a80c-35b77de60ce2-dshm\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:05:19.010297 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:19.010247    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rh2f2\" (UniqueName: \"kubernetes.io/projected/5b6aecfa-3e3b-4fb3-a80c-35b77de60ce2-kube-api-access-rh2f2\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:05:19.010297 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:19.010258    4496 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b6aecfa-3e3b-4fb3-a80c-35b77de60ce2-tls-certs\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:05:19.010297 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:19.010267    4496 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36c1912c-2ab1-40c8-a09d-2425ab08ec8d-tls-certs\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:05:19.010297 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:19.010276    4496 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36c1912c-2ab1-40c8-a09d-2425ab08ec8d-home\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:05:19.010297 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:19.010284    4496 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36c1912c-2ab1-40c8-a09d-2425ab08ec8d-dshm\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:05:19.010297 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:19.010293    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fn527\" (UniqueName: \"kubernetes.io/projected/36c1912c-2ab1-40c8-a09d-2425ab08ec8d-kube-api-access-fn527\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:05:19.202057 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:19.202005    4496 generic.go:356] "Generic (PLEG): container finished" podID="36c1912c-2ab1-40c8-a09d-2425ab08ec8d" containerID="c415850ad33f904fe4b236a1dff920b58a59368ea86dbd194172fde21cc3214f" exitCode=0
Nov 02 04:05:19.202057 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:19.202058    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-398828/distinct-2-w4-398828-kserve-545449d49b-6mg4b"
Nov 02 04:05:19.202318 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:19.202089    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-398828/distinct-2-w4-398828-kserve-545449d49b-6mg4b" event={"ID":"36c1912c-2ab1-40c8-a09d-2425ab08ec8d","Type":"ContainerDied","Data":"c415850ad33f904fe4b236a1dff920b58a59368ea86dbd194172fde21cc3214f"}
Nov 02 04:05:19.202318 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:19.202136    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-398828/distinct-2-w4-398828-kserve-545449d49b-6mg4b" event={"ID":"36c1912c-2ab1-40c8-a09d-2425ab08ec8d","Type":"ContainerDied","Data":"934ef081fedfdc5a0fa622ff6f3a69696e3dc0de9b850088633abaa8578471f1"}
Nov 02 04:05:19.202318 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:19.202153    4496 scope.go:122] "RemoveContainer" containerID="c415850ad33f904fe4b236a1dff920b58a59368ea86dbd194172fde21cc3214f"
Nov 02 04:05:19.203639 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:19.203616    4496 generic.go:356] "Generic (PLEG): container finished" podID="5b6aecfa-3e3b-4fb3-a80c-35b77de60ce2" containerID="3931bdaf7ea78a43290417fc714792f90b1af645570ea3a027e5faa7ccee9639" exitCode=0
Nov 02 04:05:19.203788 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:19.203657    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-398828/simulator-w4-398828-kserve-6d5bd49c6f-9tw2c"
Nov 02 04:05:19.203788 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:19.203704    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-398828/simulator-w4-398828-kserve-6d5bd49c6f-9tw2c" event={"ID":"5b6aecfa-3e3b-4fb3-a80c-35b77de60ce2","Type":"ContainerDied","Data":"3931bdaf7ea78a43290417fc714792f90b1af645570ea3a027e5faa7ccee9639"}
Nov 02 04:05:19.203788 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:19.203759    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-398828/simulator-w4-398828-kserve-6d5bd49c6f-9tw2c" event={"ID":"5b6aecfa-3e3b-4fb3-a80c-35b77de60ce2","Type":"ContainerDied","Data":"eb5bc5b6e8df486632e56587f8ae2368638b08630c2403aafc614dd77c7d2510"}
Nov 02 04:05:19.215427 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:19.215409    4496 scope.go:122] "RemoveContainer" containerID="c415850ad33f904fe4b236a1dff920b58a59368ea86dbd194172fde21cc3214f"
Nov 02 04:05:19.215851 ip-10-0-133-163 kubenswrapper[4496]: E1102 04:05:19.215710    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c415850ad33f904fe4b236a1dff920b58a59368ea86dbd194172fde21cc3214f\": container with ID starting with c415850ad33f904fe4b236a1dff920b58a59368ea86dbd194172fde21cc3214f not found: ID does not exist" containerID="c415850ad33f904fe4b236a1dff920b58a59368ea86dbd194172fde21cc3214f"
Nov 02 04:05:19.215851 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:19.215747    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c415850ad33f904fe4b236a1dff920b58a59368ea86dbd194172fde21cc3214f"} err="failed to get container status \"c415850ad33f904fe4b236a1dff920b58a59368ea86dbd194172fde21cc3214f\": rpc error: code = NotFound desc = could not find container \"c415850ad33f904fe4b236a1dff920b58a59368ea86dbd194172fde21cc3214f\": container with ID starting with c415850ad33f904fe4b236a1dff920b58a59368ea86dbd194172fde21cc3214f not found: ID does not exist"
Nov 02 04:05:19.215851 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:19.215770    4496 scope.go:122] "RemoveContainer" containerID="3931bdaf7ea78a43290417fc714792f90b1af645570ea3a027e5faa7ccee9639"
Nov 02 04:05:19.224832 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:19.224812    4496 scope.go:122] "RemoveContainer" containerID="3931bdaf7ea78a43290417fc714792f90b1af645570ea3a027e5faa7ccee9639"
Nov 02 04:05:19.225142 ip-10-0-133-163 kubenswrapper[4496]: E1102 04:05:19.225124    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3931bdaf7ea78a43290417fc714792f90b1af645570ea3a027e5faa7ccee9639\": container with ID starting with 3931bdaf7ea78a43290417fc714792f90b1af645570ea3a027e5faa7ccee9639 not found: ID does not exist" containerID="3931bdaf7ea78a43290417fc714792f90b1af645570ea3a027e5faa7ccee9639"
Nov 02 04:05:19.225222 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:19.225147    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3931bdaf7ea78a43290417fc714792f90b1af645570ea3a027e5faa7ccee9639"} err="failed to get container status \"3931bdaf7ea78a43290417fc714792f90b1af645570ea3a027e5faa7ccee9639\": rpc error: code = NotFound desc = could not find container \"3931bdaf7ea78a43290417fc714792f90b1af645570ea3a027e5faa7ccee9639\": container with ID starting with 3931bdaf7ea78a43290417fc714792f90b1af645570ea3a027e5faa7ccee9639 not found: ID does not exist"
Nov 02 04:05:19.370093 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:19.369980    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-398828/distinct-2-w4-398828-kserve-545449d49b-6mg4b"]
Nov 02 04:05:19.372621 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:19.372590    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-398828/distinct-2-w4-398828-kserve-545449d49b-6mg4b"]
Nov 02 04:05:19.484059 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:19.484005    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-398828/simulator-w4-398828-kserve-6d5bd49c6f-9tw2c"]
Nov 02 04:05:19.486803 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:19.486775    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-398828/simulator-w4-398828-kserve-6d5bd49c6f-9tw2c"]
Nov 02 04:05:19.574375 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:19.574342    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36c1912c-2ab1-40c8-a09d-2425ab08ec8d" path="/var/lib/kubelet/pods/36c1912c-2ab1-40c8-a09d-2425ab08ec8d/volumes"
Nov 02 04:05:19.574833 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:19.574804    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b6aecfa-3e3b-4fb3-a80c-35b77de60ce2" path="/var/lib/kubelet/pods/5b6aecfa-3e3b-4fb3-a80c-35b77de60ce2/volumes"
Nov 02 04:05:21.593302 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:21.593266    4496 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-398828-75678b79b7-bbqqw" podUID="ce91e365-c823-4791-900a-c3bb3ccd4d36" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 04:05:21.593817 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:21.593698    4496 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-398828-54955d6b75-czhwj" podUID="ba5a7a35-1ea6-48e9-8817-dd7a7f6f08fc" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 04:05:22.246161 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:22.246124    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-5ac93f-6f67ddfdc5-bwhr4"]
Nov 02 04:05:22.354009 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:22.353972    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-5ac93f-6578c5846c-lxsjl"]
Nov 02 04:05:27.246471 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:27.246403    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-5ac93f-6f67ddfdc5-bwhr4" podUID="23af24b5-7173-4169-9169-ea0c14916a53" containerName="payload-processing" containerID="cri-o://e7459e9ca917861b5598286d1018fedc02b0f2c4f3ecfc704b0250722453072a" gracePeriod=25
Nov 02 04:05:27.354798 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:27.354759    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5ac93f-6578c5846c-lxsjl" podUID="deeeaa8e-ec07-4416-8f1c-c032ffe0e55e" containerName="payload-pre-processing" containerID="cri-o://397524e61b69d74734d85c3de412784475eb5cebeb98767cf3543f18707240a3" gracePeriod=25
Nov 02 04:05:30.166219 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:30.166164    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-5ac93f/distinct-w1-5ac93f-kserve-64697cb8c9-vj9sk" podUID="75a1c221-38e8-47cf-a8b5-305396cee703" containerName="main" containerID="cri-o://dcf0712d1c3794925117aea7b6bdd0f9d526e48868becfc5d506b9eedc61a1ae" gracePeriod=45
Nov 02 04:05:30.280958 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:30.280933    4496 generic.go:356] "Generic (PLEG): container finished" podID="75a1c221-38e8-47cf-a8b5-305396cee703" containerID="dcf0712d1c3794925117aea7b6bdd0f9d526e48868becfc5d506b9eedc61a1ae" exitCode=0
Nov 02 04:05:30.281077 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:30.281040    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5ac93f/distinct-w1-5ac93f-kserve-64697cb8c9-vj9sk" event={"ID":"75a1c221-38e8-47cf-a8b5-305396cee703","Type":"ContainerDied","Data":"dcf0712d1c3794925117aea7b6bdd0f9d526e48868becfc5d506b9eedc61a1ae"}
Nov 02 04:05:30.304332 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:30.304296    4496 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5ac93f/distinct-w1-5ac93f-kserve-64697cb8c9-vj9sk" podUID="75a1c221-38e8-47cf-a8b5-305396cee703" containerName="main" probeResult="failure" output="Get \"https://10.134.0.102:8000/ready\": dial tcp 10.134.0.102:8000: connect: connection refused"
Nov 02 04:05:30.432342 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:30.432284    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5ac93f/distinct-w1-5ac93f-kserve-64697cb8c9-vj9sk"
Nov 02 04:05:30.528980 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:30.528948    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75a1c221-38e8-47cf-a8b5-305396cee703-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75a1c221-38e8-47cf-a8b5-305396cee703-model-cache\") pod \"75a1c221-38e8-47cf-a8b5-305396cee703\" (UID: \"75a1c221-38e8-47cf-a8b5-305396cee703\") "
Nov 02 04:05:30.529154 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:30.529057    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75a1c221-38e8-47cf-a8b5-305396cee703-dshm\" (UniqueName: \"kubernetes.io/empty-dir/75a1c221-38e8-47cf-a8b5-305396cee703-dshm\") pod \"75a1c221-38e8-47cf-a8b5-305396cee703\" (UID: \"75a1c221-38e8-47cf-a8b5-305396cee703\") "
Nov 02 04:05:30.529154 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:30.529091    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75a1c221-38e8-47cf-a8b5-305396cee703-tls-certs\" (UniqueName: \"kubernetes.io/secret/75a1c221-38e8-47cf-a8b5-305396cee703-tls-certs\") pod \"75a1c221-38e8-47cf-a8b5-305396cee703\" (UID: \"75a1c221-38e8-47cf-a8b5-305396cee703\") "
Nov 02 04:05:30.529154 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:30.529116    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75a1c221-38e8-47cf-a8b5-305396cee703-kube-api-access-cjggc\" (UniqueName: \"kubernetes.io/projected/75a1c221-38e8-47cf-a8b5-305396cee703-kube-api-access-cjggc\") pod \"75a1c221-38e8-47cf-a8b5-305396cee703\" (UID: \"75a1c221-38e8-47cf-a8b5-305396cee703\") "
Nov 02 04:05:30.529154 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:30.529136    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75a1c221-38e8-47cf-a8b5-305396cee703-home\" (UniqueName: \"kubernetes.io/empty-dir/75a1c221-38e8-47cf-a8b5-305396cee703-home\") pod \"75a1c221-38e8-47cf-a8b5-305396cee703\" (UID: \"75a1c221-38e8-47cf-a8b5-305396cee703\") "
Nov 02 04:05:30.529396 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:30.529196    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75a1c221-38e8-47cf-a8b5-305396cee703-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75a1c221-38e8-47cf-a8b5-305396cee703-tmp-dir\") pod \"75a1c221-38e8-47cf-a8b5-305396cee703\" (UID: \"75a1c221-38e8-47cf-a8b5-305396cee703\") "
Nov 02 04:05:30.529396 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:30.529260    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75a1c221-38e8-47cf-a8b5-305396cee703-model-cache" pod "75a1c221-38e8-47cf-a8b5-305396cee703" (UID: "75a1c221-38e8-47cf-a8b5-305396cee703"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 04:05:30.529515 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:30.529451    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75a1c221-38e8-47cf-a8b5-305396cee703-home" pod "75a1c221-38e8-47cf-a8b5-305396cee703" (UID: "75a1c221-38e8-47cf-a8b5-305396cee703"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 04:05:30.529515 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:30.529465    4496 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75a1c221-38e8-47cf-a8b5-305396cee703-model-cache\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:05:30.529515 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:30.529472    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75a1c221-38e8-47cf-a8b5-305396cee703-tmp-dir" pod "75a1c221-38e8-47cf-a8b5-305396cee703" (UID: "75a1c221-38e8-47cf-a8b5-305396cee703"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 04:05:30.531291 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:30.531266    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75a1c221-38e8-47cf-a8b5-305396cee703-dshm" pod "75a1c221-38e8-47cf-a8b5-305396cee703" (UID: "75a1c221-38e8-47cf-a8b5-305396cee703"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 04:05:30.531411 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:30.531371    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75a1c221-38e8-47cf-a8b5-305396cee703-kube-api-access-cjggc" pod "75a1c221-38e8-47cf-a8b5-305396cee703" (UID: "75a1c221-38e8-47cf-a8b5-305396cee703"). InnerVolumeSpecName "kube-api-access-cjggc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 04:05:30.531411 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:30.531383    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75a1c221-38e8-47cf-a8b5-305396cee703-tls-certs" pod "75a1c221-38e8-47cf-a8b5-305396cee703" (UID: "75a1c221-38e8-47cf-a8b5-305396cee703"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 04:05:30.630928 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:30.630572    4496 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75a1c221-38e8-47cf-a8b5-305396cee703-tmp-dir\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:05:30.630928 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:30.630608    4496 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75a1c221-38e8-47cf-a8b5-305396cee703-dshm\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:05:30.630928 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:30.630624    4496 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75a1c221-38e8-47cf-a8b5-305396cee703-tls-certs\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:05:30.630928 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:30.630638    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjggc\" (UniqueName: \"kubernetes.io/projected/75a1c221-38e8-47cf-a8b5-305396cee703-kube-api-access-cjggc\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:05:30.630928 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:30.630706    4496 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75a1c221-38e8-47cf-a8b5-305396cee703-home\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:05:31.288709 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:31.288669    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5ac93f/distinct-w1-5ac93f-kserve-64697cb8c9-vj9sk" event={"ID":"75a1c221-38e8-47cf-a8b5-305396cee703","Type":"ContainerDied","Data":"f914b8941a44be80c4b6d985232f9ff1a86ac8d66f6dd6d4311c2e2be80e21de"}
Nov 02 04:05:31.289201 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:31.288723    4496 scope.go:122] "RemoveContainer" containerID="dcf0712d1c3794925117aea7b6bdd0f9d526e48868becfc5d506b9eedc61a1ae"
Nov 02 04:05:31.289201 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:31.288691    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5ac93f/distinct-w1-5ac93f-kserve-64697cb8c9-vj9sk"
Nov 02 04:05:31.457242 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:31.457208    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-5ac93f/distinct-w1-5ac93f-kserve-64697cb8c9-vj9sk"]
Nov 02 04:05:31.507927 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:31.507898    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-5ac93f/distinct-w1-5ac93f-kserve-64697cb8c9-vj9sk"]
Nov 02 04:05:31.573800 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:31.573725    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75a1c221-38e8-47cf-a8b5-305396cee703" path="/var/lib/kubelet/pods/75a1c221-38e8-47cf-a8b5-305396cee703/volumes"
Nov 02 04:05:31.592793 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:31.592757    4496 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-398828-75678b79b7-bbqqw" podUID="ce91e365-c823-4791-900a-c3bb3ccd4d36" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 04:05:31.592793 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:31.592773    4496 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-398828-54955d6b75-czhwj" podUID="ba5a7a35-1ea6-48e9-8817-dd7a7f6f08fc" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 04:05:33.529457 ip-10-0-133-163 kubenswrapper[4496]: E1102 04:05:33.529416    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 04:05:33.735819 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:33.735783    4496 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5ac93f-6578c5846c-lxsjl" podUID="deeeaa8e-ec07-4416-8f1c-c032ffe0e55e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 04:05:33.735998 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:33.735845    4496 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-5ac93f-6f67ddfdc5-bwhr4" podUID="23af24b5-7173-4169-9169-ea0c14916a53" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 04:05:35.728462 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:35.728434    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-398828-54955d6b75-czhwj"
Nov 02 04:05:35.821661 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:35.821635    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-398828-75678b79b7-bbqqw"
Nov 02 04:05:35.890642 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:35.890612    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba5a7a35-1ea6-48e9-8817-dd7a7f6f08fc-kube-api-access-gsz5g\" (UniqueName: \"kubernetes.io/projected/ba5a7a35-1ea6-48e9-8817-dd7a7f6f08fc-kube-api-access-gsz5g\") pod \"ba5a7a35-1ea6-48e9-8817-dd7a7f6f08fc\" (UID: \"ba5a7a35-1ea6-48e9-8817-dd7a7f6f08fc\") "
Nov 02 04:05:35.890826 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:35.890654    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba5a7a35-1ea6-48e9-8817-dd7a7f6f08fc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba5a7a35-1ea6-48e9-8817-dd7a7f6f08fc-plugins-config-volume\") pod \"ba5a7a35-1ea6-48e9-8817-dd7a7f6f08fc\" (UID: \"ba5a7a35-1ea6-48e9-8817-dd7a7f6f08fc\") "
Nov 02 04:05:35.891109 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:35.891084    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba5a7a35-1ea6-48e9-8817-dd7a7f6f08fc-plugins-config-volume" pod "ba5a7a35-1ea6-48e9-8817-dd7a7f6f08fc" (UID: "ba5a7a35-1ea6-48e9-8817-dd7a7f6f08fc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 04:05:35.892758 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:35.892739    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba5a7a35-1ea6-48e9-8817-dd7a7f6f08fc-kube-api-access-gsz5g" pod "ba5a7a35-1ea6-48e9-8817-dd7a7f6f08fc" (UID: "ba5a7a35-1ea6-48e9-8817-dd7a7f6f08fc"). InnerVolumeSpecName "kube-api-access-gsz5g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 04:05:35.992028 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:35.991944    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce91e365-c823-4791-900a-c3bb3ccd4d36-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce91e365-c823-4791-900a-c3bb3ccd4d36-plugins-config-volume\") pod \"ce91e365-c823-4791-900a-c3bb3ccd4d36\" (UID: \"ce91e365-c823-4791-900a-c3bb3ccd4d36\") "
Nov 02 04:05:35.992028 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:35.991980    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce91e365-c823-4791-900a-c3bb3ccd4d36-kube-api-access-nr4kq\" (UniqueName: \"kubernetes.io/projected/ce91e365-c823-4791-900a-c3bb3ccd4d36-kube-api-access-nr4kq\") pod \"ce91e365-c823-4791-900a-c3bb3ccd4d36\" (UID: \"ce91e365-c823-4791-900a-c3bb3ccd4d36\") "
Nov 02 04:05:35.992406 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:35.992372    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce91e365-c823-4791-900a-c3bb3ccd4d36-plugins-config-volume" pod "ce91e365-c823-4791-900a-c3bb3ccd4d36" (UID: "ce91e365-c823-4791-900a-c3bb3ccd4d36"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 04:05:35.992556 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:35.992505    4496 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce91e365-c823-4791-900a-c3bb3ccd4d36-plugins-config-volume\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:05:35.992556 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:35.992528    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsz5g\" (UniqueName: \"kubernetes.io/projected/ba5a7a35-1ea6-48e9-8817-dd7a7f6f08fc-kube-api-access-gsz5g\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:05:35.992556 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:35.992542    4496 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba5a7a35-1ea6-48e9-8817-dd7a7f6f08fc-plugins-config-volume\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:05:35.994091 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:35.994071    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce91e365-c823-4791-900a-c3bb3ccd4d36-kube-api-access-nr4kq" pod "ce91e365-c823-4791-900a-c3bb3ccd4d36" (UID: "ce91e365-c823-4791-900a-c3bb3ccd4d36"). InnerVolumeSpecName "kube-api-access-nr4kq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 04:05:36.094111 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:36.094066    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nr4kq\" (UniqueName: \"kubernetes.io/projected/ce91e365-c823-4791-900a-c3bb3ccd4d36-kube-api-access-nr4kq\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:05:36.325070 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:36.324970    4496 generic.go:356] "Generic (PLEG): container finished" podID="ce91e365-c823-4791-900a-c3bb3ccd4d36" containerID="cfbc8cee8ca607ed8ec1ba68b6374a79b6831788b1e73d2fff895294fa0cbfc5" exitCode=0
Nov 02 04:05:36.325284 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:36.325083    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-398828-75678b79b7-bbqqw"
Nov 02 04:05:36.325284 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:36.325062    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-398828-75678b79b7-bbqqw" event={"ID":"ce91e365-c823-4791-900a-c3bb3ccd4d36","Type":"ContainerDied","Data":"cfbc8cee8ca607ed8ec1ba68b6374a79b6831788b1e73d2fff895294fa0cbfc5"}
Nov 02 04:05:36.325284 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:36.325184    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-398828-75678b79b7-bbqqw" event={"ID":"ce91e365-c823-4791-900a-c3bb3ccd4d36","Type":"ContainerDied","Data":"9386d0c28da999050e0fd491eedb424cd79950cf250ba3e0566630023360aeee"}
Nov 02 04:05:36.325284 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:36.325228    4496 scope.go:122] "RemoveContainer" containerID="cfbc8cee8ca607ed8ec1ba68b6374a79b6831788b1e73d2fff895294fa0cbfc5"
Nov 02 04:05:36.326645 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:36.326618    4496 generic.go:356] "Generic (PLEG): container finished" podID="ba5a7a35-1ea6-48e9-8817-dd7a7f6f08fc" containerID="2cb2c85fa4ee35903735c777ca2c32e2a41edb03c856637dcc5b1398897639ba" exitCode=0
Nov 02 04:05:36.326752 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:36.326652    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-398828-54955d6b75-czhwj"
Nov 02 04:05:36.326752 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:36.326699    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-398828-54955d6b75-czhwj" event={"ID":"ba5a7a35-1ea6-48e9-8817-dd7a7f6f08fc","Type":"ContainerDied","Data":"2cb2c85fa4ee35903735c777ca2c32e2a41edb03c856637dcc5b1398897639ba"}
Nov 02 04:05:36.326752 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:36.326749    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-398828-54955d6b75-czhwj" event={"ID":"ba5a7a35-1ea6-48e9-8817-dd7a7f6f08fc","Type":"ContainerDied","Data":"61724c17c9ec05ffa3493f241c9790b38aacbeba527300eb01ca1c9c696e03b5"}
Nov 02 04:05:36.336274 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:36.336252    4496 scope.go:122] "RemoveContainer" containerID="cfbc8cee8ca607ed8ec1ba68b6374a79b6831788b1e73d2fff895294fa0cbfc5"
Nov 02 04:05:36.336631 ip-10-0-133-163 kubenswrapper[4496]: E1102 04:05:36.336596    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfbc8cee8ca607ed8ec1ba68b6374a79b6831788b1e73d2fff895294fa0cbfc5\": container with ID starting with cfbc8cee8ca607ed8ec1ba68b6374a79b6831788b1e73d2fff895294fa0cbfc5 not found: ID does not exist" containerID="cfbc8cee8ca607ed8ec1ba68b6374a79b6831788b1e73d2fff895294fa0cbfc5"
Nov 02 04:05:36.336742 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:36.336637    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfbc8cee8ca607ed8ec1ba68b6374a79b6831788b1e73d2fff895294fa0cbfc5"} err="failed to get container status \"cfbc8cee8ca607ed8ec1ba68b6374a79b6831788b1e73d2fff895294fa0cbfc5\": rpc error: code = NotFound desc = could not find container \"cfbc8cee8ca607ed8ec1ba68b6374a79b6831788b1e73d2fff895294fa0cbfc5\": container with ID starting with cfbc8cee8ca607ed8ec1ba68b6374a79b6831788b1e73d2fff895294fa0cbfc5 not found: ID does not exist"
Nov 02 04:05:36.336742 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:36.336656    4496 scope.go:122] "RemoveContainer" containerID="2cb2c85fa4ee35903735c777ca2c32e2a41edb03c856637dcc5b1398897639ba"
Nov 02 04:05:36.347431 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:36.347411    4496 scope.go:122] "RemoveContainer" containerID="2cb2c85fa4ee35903735c777ca2c32e2a41edb03c856637dcc5b1398897639ba"
Nov 02 04:05:36.347701 ip-10-0-133-163 kubenswrapper[4496]: E1102 04:05:36.347683    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cb2c85fa4ee35903735c777ca2c32e2a41edb03c856637dcc5b1398897639ba\": container with ID starting with 2cb2c85fa4ee35903735c777ca2c32e2a41edb03c856637dcc5b1398897639ba not found: ID does not exist" containerID="2cb2c85fa4ee35903735c777ca2c32e2a41edb03c856637dcc5b1398897639ba"
Nov 02 04:05:36.347844 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:36.347712    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cb2c85fa4ee35903735c777ca2c32e2a41edb03c856637dcc5b1398897639ba"} err="failed to get container status \"2cb2c85fa4ee35903735c777ca2c32e2a41edb03c856637dcc5b1398897639ba\": rpc error: code = NotFound desc = could not find container \"2cb2c85fa4ee35903735c777ca2c32e2a41edb03c856637dcc5b1398897639ba\": container with ID starting with 2cb2c85fa4ee35903735c777ca2c32e2a41edb03c856637dcc5b1398897639ba not found: ID does not exist"
Nov 02 04:05:36.494537 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:36.494508    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-398828-75678b79b7-bbqqw"]
Nov 02 04:05:36.545793 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:36.545762    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-398828-75678b79b7-bbqqw"]
Nov 02 04:05:36.611665 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:36.611584    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-398828-54955d6b75-czhwj"]
Nov 02 04:05:36.622524 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:36.622494    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-398828-54955d6b75-czhwj"]
Nov 02 04:05:37.577803 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:37.577767    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba5a7a35-1ea6-48e9-8817-dd7a7f6f08fc" path="/var/lib/kubelet/pods/ba5a7a35-1ea6-48e9-8817-dd7a7f6f08fc/volumes"
Nov 02 04:05:37.578421 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:37.578399    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce91e365-c823-4791-900a-c3bb3ccd4d36" path="/var/lib/kubelet/pods/ce91e365-c823-4791-900a-c3bb3ccd4d36/volumes"
Nov 02 04:05:43.735620 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:43.735579    4496 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5ac93f-6578c5846c-lxsjl" podUID="deeeaa8e-ec07-4416-8f1c-c032ffe0e55e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 04:05:43.736049 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:43.735579    4496 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-5ac93f-6f67ddfdc5-bwhr4" podUID="23af24b5-7173-4169-9169-ea0c14916a53" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 04:05:47.418155 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:47.418111    4496 generic.go:356] "Generic (PLEG): container finished" podID="23af24b5-7173-4169-9169-ea0c14916a53" containerID="e7459e9ca917861b5598286d1018fedc02b0f2c4f3ecfc704b0250722453072a" exitCode=0
Nov 02 04:05:47.418155 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:47.418143    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-5ac93f-6f67ddfdc5-bwhr4" event={"ID":"23af24b5-7173-4169-9169-ea0c14916a53","Type":"ContainerDied","Data":"e7459e9ca917861b5598286d1018fedc02b0f2c4f3ecfc704b0250722453072a"}
Nov 02 04:05:47.512553 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:47.512486    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-5ac93f-6f67ddfdc5-bwhr4"
Nov 02 04:05:47.533268 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:47.533241    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23af24b5-7173-4169-9169-ea0c14916a53-kube-api-access-l6sfw\" (UniqueName: \"kubernetes.io/projected/23af24b5-7173-4169-9169-ea0c14916a53-kube-api-access-l6sfw\") pod \"23af24b5-7173-4169-9169-ea0c14916a53\" (UID: \"23af24b5-7173-4169-9169-ea0c14916a53\") "
Nov 02 04:05:47.533414 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:47.533396    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23af24b5-7173-4169-9169-ea0c14916a53-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23af24b5-7173-4169-9169-ea0c14916a53-plugins-config-volume\") pod \"23af24b5-7173-4169-9169-ea0c14916a53\" (UID: \"23af24b5-7173-4169-9169-ea0c14916a53\") "
Nov 02 04:05:47.534105 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:47.534061    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23af24b5-7173-4169-9169-ea0c14916a53-plugins-config-volume" pod "23af24b5-7173-4169-9169-ea0c14916a53" (UID: "23af24b5-7173-4169-9169-ea0c14916a53"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 04:05:47.538470 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:47.538428    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23af24b5-7173-4169-9169-ea0c14916a53-kube-api-access-l6sfw" pod "23af24b5-7173-4169-9169-ea0c14916a53" (UID: "23af24b5-7173-4169-9169-ea0c14916a53"). InnerVolumeSpecName "kube-api-access-l6sfw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 04:05:47.603663 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:47.603641    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5ac93f-6578c5846c-lxsjl"
Nov 02 04:05:47.635206 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:47.635164    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/deeeaa8e-ec07-4416-8f1c-c032ffe0e55e-kube-api-access-plghg\" (UniqueName: \"kubernetes.io/projected/deeeaa8e-ec07-4416-8f1c-c032ffe0e55e-kube-api-access-plghg\") pod \"deeeaa8e-ec07-4416-8f1c-c032ffe0e55e\" (UID: \"deeeaa8e-ec07-4416-8f1c-c032ffe0e55e\") "
Nov 02 04:05:47.635389 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:47.635332    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/deeeaa8e-ec07-4416-8f1c-c032ffe0e55e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/deeeaa8e-ec07-4416-8f1c-c032ffe0e55e-plugins-config-volume\") pod \"deeeaa8e-ec07-4416-8f1c-c032ffe0e55e\" (UID: \"deeeaa8e-ec07-4416-8f1c-c032ffe0e55e\") "
Nov 02 04:05:47.635691 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:47.635665    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6sfw\" (UniqueName: \"kubernetes.io/projected/23af24b5-7173-4169-9169-ea0c14916a53-kube-api-access-l6sfw\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:05:47.635691 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:47.635692    4496 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23af24b5-7173-4169-9169-ea0c14916a53-plugins-config-volume\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:05:47.635890 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:47.635706    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/deeeaa8e-ec07-4416-8f1c-c032ffe0e55e-plugins-config-volume" pod "deeeaa8e-ec07-4416-8f1c-c032ffe0e55e" (UID: "deeeaa8e-ec07-4416-8f1c-c032ffe0e55e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 04:05:47.638475 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:47.638452    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/deeeaa8e-ec07-4416-8f1c-c032ffe0e55e-kube-api-access-plghg" pod "deeeaa8e-ec07-4416-8f1c-c032ffe0e55e" (UID: "deeeaa8e-ec07-4416-8f1c-c032ffe0e55e"). InnerVolumeSpecName "kube-api-access-plghg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 04:05:47.737065 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:47.737004    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plghg\" (UniqueName: \"kubernetes.io/projected/deeeaa8e-ec07-4416-8f1c-c032ffe0e55e-kube-api-access-plghg\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:05:47.737065 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:47.737065    4496 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/deeeaa8e-ec07-4416-8f1c-c032ffe0e55e-plugins-config-volume\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:05:48.426600 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:48.426566    4496 generic.go:356] "Generic (PLEG): container finished" podID="deeeaa8e-ec07-4416-8f1c-c032ffe0e55e" containerID="397524e61b69d74734d85c3de412784475eb5cebeb98767cf3543f18707240a3" exitCode=0
Nov 02 04:05:48.427128 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:48.426612    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5ac93f-6578c5846c-lxsjl"
Nov 02 04:05:48.427128 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:48.426648    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5ac93f-6578c5846c-lxsjl" event={"ID":"deeeaa8e-ec07-4416-8f1c-c032ffe0e55e","Type":"ContainerDied","Data":"397524e61b69d74734d85c3de412784475eb5cebeb98767cf3543f18707240a3"}
Nov 02 04:05:48.427128 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:48.426722    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5ac93f-6578c5846c-lxsjl" event={"ID":"deeeaa8e-ec07-4416-8f1c-c032ffe0e55e","Type":"ContainerDied","Data":"029c865b7b1557d0aae99a23803127373a70291ebf0ae76905c86ff209bb65bd"}
Nov 02 04:05:48.427128 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:48.426754    4496 scope.go:122] "RemoveContainer" containerID="397524e61b69d74734d85c3de412784475eb5cebeb98767cf3543f18707240a3"
Nov 02 04:05:48.428392 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:48.428370    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-5ac93f-6f67ddfdc5-bwhr4"
Nov 02 04:05:48.428546 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:48.428413    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-5ac93f-6f67ddfdc5-bwhr4" event={"ID":"23af24b5-7173-4169-9169-ea0c14916a53","Type":"ContainerDied","Data":"1ac49e0a51903955094983a5189b5a9a387b7a6f2c40166a77f850991c3de47a"}
Nov 02 04:05:48.438777 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:48.438756    4496 scope.go:122] "RemoveContainer" containerID="397524e61b69d74734d85c3de412784475eb5cebeb98767cf3543f18707240a3"
Nov 02 04:05:48.439115 ip-10-0-133-163 kubenswrapper[4496]: E1102 04:05:48.439091    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"397524e61b69d74734d85c3de412784475eb5cebeb98767cf3543f18707240a3\": container with ID starting with 397524e61b69d74734d85c3de412784475eb5cebeb98767cf3543f18707240a3 not found: ID does not exist" containerID="397524e61b69d74734d85c3de412784475eb5cebeb98767cf3543f18707240a3"
Nov 02 04:05:48.439212 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:48.439123    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"397524e61b69d74734d85c3de412784475eb5cebeb98767cf3543f18707240a3"} err="failed to get container status \"397524e61b69d74734d85c3de412784475eb5cebeb98767cf3543f18707240a3\": rpc error: code = NotFound desc = could not find container \"397524e61b69d74734d85c3de412784475eb5cebeb98767cf3543f18707240a3\": container with ID starting with 397524e61b69d74734d85c3de412784475eb5cebeb98767cf3543f18707240a3 not found: ID does not exist"
Nov 02 04:05:48.439212 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:48.439142    4496 scope.go:122] "RemoveContainer" containerID="e7459e9ca917861b5598286d1018fedc02b0f2c4f3ecfc704b0250722453072a"
Nov 02 04:05:48.613332 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:48.613256    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-5ac93f-6578c5846c-lxsjl"]
Nov 02 04:05:48.660718 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:48.660685    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-5ac93f-6578c5846c-lxsjl"]
Nov 02 04:05:48.769029 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:48.768991    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-5ac93f-6f67ddfdc5-bwhr4"]
Nov 02 04:05:48.774896 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:48.774865    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-5ac93f-6f67ddfdc5-bwhr4"]
Nov 02 04:05:49.574429 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:49.574397    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23af24b5-7173-4169-9169-ea0c14916a53" path="/var/lib/kubelet/pods/23af24b5-7173-4169-9169-ea0c14916a53/volumes"
Nov 02 04:05:49.574893 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:05:49.574878    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="deeeaa8e-ec07-4416-8f1c-c032ffe0e55e" path="/var/lib/kubelet/pods/deeeaa8e-ec07-4416-8f1c-c032ffe0e55e/volumes"
Nov 02 04:06:00.196749 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:00.196708    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-wld4d"]
Nov 02 04:06:00.197240 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:00.197226    4496 cpu_manager.go:414] "containerMap: removing container" podUID="0fabbfdc-768a-49d7-a62d-2bf8680a5cc8" containerName="payload-processing"
Nov 02 04:06:00.197294 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:00.197241    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fabbfdc-768a-49d7-a62d-2bf8680a5cc8" containerName="payload-processing"
Nov 02 04:06:00.197294 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:00.197253    4496 cpu_manager.go:414] "containerMap: removing container" podUID="36c1912c-2ab1-40c8-a09d-2425ab08ec8d" containerName="main"
Nov 02 04:06:00.197294 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:00.197258    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36c1912c-2ab1-40c8-a09d-2425ab08ec8d" containerName="main"
Nov 02 04:06:00.197294 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:00.197269    4496 cpu_manager.go:414] "containerMap: removing container" podUID="5b6aecfa-3e3b-4fb3-a80c-35b77de60ce2" containerName="main"
Nov 02 04:06:00.197294 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:00.197275    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b6aecfa-3e3b-4fb3-a80c-35b77de60ce2" containerName="main"
Nov 02 04:06:00.197294 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:00.197282    4496 cpu_manager.go:414] "containerMap: removing container" podUID="85c3ba82-04d0-48f7-9da2-849ee385edc7" containerName="cleanup"
Nov 02 04:06:00.197294 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:00.197287    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85c3ba82-04d0-48f7-9da2-849ee385edc7" containerName="cleanup"
Nov 02 04:06:00.197294 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:00.197297    4496 cpu_manager.go:414] "containerMap: removing container" podUID="51249edc-f28f-4f57-94cd-eca441b6c5c6" containerName="maas-api"
Nov 02 04:06:00.197586 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:00.197302    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51249edc-f28f-4f57-94cd-eca441b6c5c6" containerName="maas-api"
Nov 02 04:06:00.197586 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:00.197310    4496 cpu_manager.go:414] "containerMap: removing container" podUID="4fd6a8f7-0b10-4009-a6db-de0a01aca2b8" containerName="main"
Nov 02 04:06:00.197586 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:00.197315    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fd6a8f7-0b10-4009-a6db-de0a01aca2b8" containerName="main"
Nov 02 04:06:00.197586 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:00.197324    4496 cpu_manager.go:414] "containerMap: removing container" podUID="57fc7cac-886f-4b70-86ee-6f97a711531a" containerName="payload-pre-processing"
Nov 02 04:06:00.197586 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:00.197329    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57fc7cac-886f-4b70-86ee-6f97a711531a" containerName="payload-pre-processing"
Nov 02 04:06:00.197586 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:00.197339    4496 cpu_manager.go:414] "containerMap: removing container" podUID="23af24b5-7173-4169-9169-ea0c14916a53" containerName="payload-processing"
Nov 02 04:06:00.197586 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:00.197344    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23af24b5-7173-4169-9169-ea0c14916a53" containerName="payload-processing"
Nov 02 04:06:00.197586 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:00.197360    4496 cpu_manager.go:414] "containerMap: removing container" podUID="ba5a7a35-1ea6-48e9-8817-dd7a7f6f08fc" containerName="payload-processing"
Nov 02 04:06:00.197586 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:00.197365    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba5a7a35-1ea6-48e9-8817-dd7a7f6f08fc" containerName="payload-processing"
Nov 02 04:06:00.197586 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:00.197374    4496 cpu_manager.go:414] "containerMap: removing container" podUID="deeeaa8e-ec07-4416-8f1c-c032ffe0e55e" containerName="payload-pre-processing"
Nov 02 04:06:00.197586 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:00.197379    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="deeeaa8e-ec07-4416-8f1c-c032ffe0e55e" containerName="payload-pre-processing"
Nov 02 04:06:00.197586 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:00.197387    4496 cpu_manager.go:414] "containerMap: removing container" podUID="75a1c221-38e8-47cf-a8b5-305396cee703" containerName="main"
Nov 02 04:06:00.197586 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:00.197391    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75a1c221-38e8-47cf-a8b5-305396cee703" containerName="main"
Nov 02 04:06:00.197586 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:00.197398    4496 cpu_manager.go:414] "containerMap: removing container" podUID="ce91e365-c823-4791-900a-c3bb3ccd4d36" containerName="payload-pre-processing"
Nov 02 04:06:00.197586 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:00.197403    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce91e365-c823-4791-900a-c3bb3ccd4d36" containerName="payload-pre-processing"
Nov 02 04:06:00.197586 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:00.197470    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fabbfdc-768a-49d7-a62d-2bf8680a5cc8" containerName="payload-processing"
Nov 02 04:06:00.197586 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:00.197477    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="36c1912c-2ab1-40c8-a09d-2425ab08ec8d" containerName="main"
Nov 02 04:06:00.197586 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:00.197487    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b6aecfa-3e3b-4fb3-a80c-35b77de60ce2" containerName="main"
Nov 02 04:06:00.197586 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:00.197493    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="51249edc-f28f-4f57-94cd-eca441b6c5c6" containerName="maas-api"
Nov 02 04:06:00.197586 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:00.197500    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fd6a8f7-0b10-4009-a6db-de0a01aca2b8" containerName="main"
Nov 02 04:06:00.197586 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:00.197507    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="57fc7cac-886f-4b70-86ee-6f97a711531a" containerName="payload-pre-processing"
Nov 02 04:06:00.197586 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:00.197514    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="23af24b5-7173-4169-9169-ea0c14916a53" containerName="payload-processing"
Nov 02 04:06:00.197586 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:00.197522    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba5a7a35-1ea6-48e9-8817-dd7a7f6f08fc" containerName="payload-processing"
Nov 02 04:06:00.197586 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:00.197527    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="deeeaa8e-ec07-4416-8f1c-c032ffe0e55e" containerName="payload-pre-processing"
Nov 02 04:06:00.197586 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:00.197533    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="75a1c221-38e8-47cf-a8b5-305396cee703" containerName="main"
Nov 02 04:06:00.197586 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:00.197538    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce91e365-c823-4791-900a-c3bb3ccd4d36" containerName="payload-pre-processing"
Nov 02 04:06:00.197586 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:00.197551    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="85c3ba82-04d0-48f7-9da2-849ee385edc7" containerName="cleanup"
Nov 02 04:06:00.200642 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:00.200626    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-wld4d"
Nov 02 04:06:00.266071 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:00.266011    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90e07af3-2747-455c-9496-1cda9c7c6eb9-istio-token\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-wld4d\" (UID: \"90e07af3-2747-455c-9496-1cda9c7c6eb9\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-wld4d"
Nov 02 04:06:00.266262 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:00.266081    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90e07af3-2747-455c-9496-1cda9c7c6eb9-workload-certs\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-wld4d\" (UID: \"90e07af3-2747-455c-9496-1cda9c7c6eb9\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-wld4d"
Nov 02 04:06:00.266262 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:00.266145    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90e07af3-2747-455c-9496-1cda9c7c6eb9-istio-podinfo\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-wld4d\" (UID: \"90e07af3-2747-455c-9496-1cda9c7c6eb9\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-wld4d"
Nov 02 04:06:00.266262 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:00.266209    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90e07af3-2747-455c-9496-1cda9c7c6eb9-istio-data\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-wld4d\" (UID: \"90e07af3-2747-455c-9496-1cda9c7c6eb9\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-wld4d"
Nov 02 04:06:00.266390 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:00.266352    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90e07af3-2747-455c-9496-1cda9c7c6eb9-istio-envoy\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-wld4d\" (UID: \"90e07af3-2747-455c-9496-1cda9c7c6eb9\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-wld4d"
Nov 02 04:06:00.266426 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:00.266402    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90e07af3-2747-455c-9496-1cda9c7c6eb9-credential-socket\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-wld4d\" (UID: \"90e07af3-2747-455c-9496-1cda9c7c6eb9\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-wld4d"
Nov 02 04:06:00.266426 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:00.266421    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90e07af3-2747-455c-9496-1cda9c7c6eb9-istiod-ca-cert\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-wld4d\" (UID: \"90e07af3-2747-455c-9496-1cda9c7c6eb9\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-wld4d"
Nov 02 04:06:00.266575 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:00.266440    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbb55\" (UniqueName: \"kubernetes.io/projected/90e07af3-2747-455c-9496-1cda9c7c6eb9-kube-api-access-bbb55\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-wld4d\" (UID: \"90e07af3-2747-455c-9496-1cda9c7c6eb9\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-wld4d"
Nov 02 04:06:00.266575 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:00.266499    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90e07af3-2747-455c-9496-1cda9c7c6eb9-workload-socket\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-wld4d\" (UID: \"90e07af3-2747-455c-9496-1cda9c7c6eb9\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-wld4d"
Nov 02 04:06:00.293717 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:00.293682    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-wld4d"]
Nov 02 04:06:00.341205 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:00.341167    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-398828-openshift-default-dockercfg-dqq8t\""
Nov 02 04:06:00.367611 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:00.367578    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90e07af3-2747-455c-9496-1cda9c7c6eb9-workload-socket\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-wld4d\" (UID: \"90e07af3-2747-455c-9496-1cda9c7c6eb9\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-wld4d"
Nov 02 04:06:00.367777 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:00.367636    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90e07af3-2747-455c-9496-1cda9c7c6eb9-istio-token\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-wld4d\" (UID: \"90e07af3-2747-455c-9496-1cda9c7c6eb9\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-wld4d"
Nov 02 04:06:00.367777 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:00.367668    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90e07af3-2747-455c-9496-1cda9c7c6eb9-workload-certs\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-wld4d\" (UID: \"90e07af3-2747-455c-9496-1cda9c7c6eb9\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-wld4d"
Nov 02 04:06:00.367777 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:00.367690    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90e07af3-2747-455c-9496-1cda9c7c6eb9-istio-podinfo\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-wld4d\" (UID: \"90e07af3-2747-455c-9496-1cda9c7c6eb9\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-wld4d"
Nov 02 04:06:00.367777 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:00.367710    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90e07af3-2747-455c-9496-1cda9c7c6eb9-istio-data\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-wld4d\" (UID: \"90e07af3-2747-455c-9496-1cda9c7c6eb9\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-wld4d"
Nov 02 04:06:00.367997 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:00.367811    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90e07af3-2747-455c-9496-1cda9c7c6eb9-istio-envoy\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-wld4d\" (UID: \"90e07af3-2747-455c-9496-1cda9c7c6eb9\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-wld4d"
Nov 02 04:06:00.367997 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:00.367837    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90e07af3-2747-455c-9496-1cda9c7c6eb9-credential-socket\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-wld4d\" (UID: \"90e07af3-2747-455c-9496-1cda9c7c6eb9\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-wld4d"
Nov 02 04:06:00.367997 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:00.367864    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90e07af3-2747-455c-9496-1cda9c7c6eb9-istiod-ca-cert\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-wld4d\" (UID: \"90e07af3-2747-455c-9496-1cda9c7c6eb9\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-wld4d"
Nov 02 04:06:00.367997 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:00.367895    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbb55\" (UniqueName: \"kubernetes.io/projected/90e07af3-2747-455c-9496-1cda9c7c6eb9-kube-api-access-bbb55\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-wld4d\" (UID: \"90e07af3-2747-455c-9496-1cda9c7c6eb9\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-wld4d"
Nov 02 04:06:00.368363 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:00.368086    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90e07af3-2747-455c-9496-1cda9c7c6eb9-workload-socket\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-wld4d\" (UID: \"90e07af3-2747-455c-9496-1cda9c7c6eb9\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-wld4d"
Nov 02 04:06:00.368363 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:00.368114    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90e07af3-2747-455c-9496-1cda9c7c6eb9-workload-certs\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-wld4d\" (UID: \"90e07af3-2747-455c-9496-1cda9c7c6eb9\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-wld4d"
Nov 02 04:06:00.368363 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:00.368186    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90e07af3-2747-455c-9496-1cda9c7c6eb9-istio-data\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-wld4d\" (UID: \"90e07af3-2747-455c-9496-1cda9c7c6eb9\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-wld4d"
Nov 02 04:06:00.368527 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:00.368464    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90e07af3-2747-455c-9496-1cda9c7c6eb9-credential-socket\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-wld4d\" (UID: \"90e07af3-2747-455c-9496-1cda9c7c6eb9\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-wld4d"
Nov 02 04:06:00.368806 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:00.368783    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90e07af3-2747-455c-9496-1cda9c7c6eb9-istiod-ca-cert\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-wld4d\" (UID: \"90e07af3-2747-455c-9496-1cda9c7c6eb9\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-wld4d"
Nov 02 04:06:00.370155 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:00.370131    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90e07af3-2747-455c-9496-1cda9c7c6eb9-istio-envoy\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-wld4d\" (UID: \"90e07af3-2747-455c-9496-1cda9c7c6eb9\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-wld4d"
Nov 02 04:06:00.370627 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:00.370606    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90e07af3-2747-455c-9496-1cda9c7c6eb9-istio-podinfo\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-wld4d\" (UID: \"90e07af3-2747-455c-9496-1cda9c7c6eb9\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-wld4d"
Nov 02 04:06:00.425957 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:00.425930    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbb55\" (UniqueName: \"kubernetes.io/projected/90e07af3-2747-455c-9496-1cda9c7c6eb9-kube-api-access-bbb55\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-wld4d\" (UID: \"90e07af3-2747-455c-9496-1cda9c7c6eb9\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-wld4d"
Nov 02 04:06:00.428520 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:00.428500    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90e07af3-2747-455c-9496-1cda9c7c6eb9-istio-token\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-wld4d\" (UID: \"90e07af3-2747-455c-9496-1cda9c7c6eb9\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-wld4d"
Nov 02 04:06:00.514339 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:00.514313    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-wld4d"
Nov 02 04:06:00.946407 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:00.946376    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-wld4d"]
Nov 02 04:06:00.947524 ip-10-0-133-163 kubenswrapper[4496]: W1102 04:06:00.947496    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90e07af3_2747_455c_9496_1cda9c7c6eb9.slice/crio-d38fde0e60a9fb75835dcd3b4cc8469067ad7f8a1c26ff3f02e71d25f2e01545 WatchSource:0}: Error finding container d38fde0e60a9fb75835dcd3b4cc8469067ad7f8a1c26ff3f02e71d25f2e01545: Status 404 returned error can't find the container with id d38fde0e60a9fb75835dcd3b4cc8469067ad7f8a1c26ff3f02e71d25f2e01545
Nov 02 04:06:00.949986 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:00.949956    4496 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 04:06:00.950076 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:00.950035    4496 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 04:06:00.950076 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:00.950072    4496 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 04:06:01.057262 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:01.057223    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-wld4d"]
Nov 02 04:06:01.514164 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:01.514128    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-wld4d" event={"ID":"90e07af3-2747-455c-9496-1cda9c7c6eb9","Type":"ContainerStarted","Data":"d172ee06e39e35634785e2ae488571e0a80a9543a4d28d301a90eec04fa6d880"}
Nov 02 04:06:01.514625 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:01.514188    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-wld4d" event={"ID":"90e07af3-2747-455c-9496-1cda9c7c6eb9","Type":"ContainerStarted","Data":"d38fde0e60a9fb75835dcd3b4cc8469067ad7f8a1c26ff3f02e71d25f2e01545"}
Nov 02 04:06:01.514625 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:01.514341    4496 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-wld4d" secret="" err="secret \"e2e-worker-w4-398828-openshift-default-dockercfg-dqq8t\" not found"
Nov 02 04:06:01.514625 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:01.514392    4496 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-wld4d"
Nov 02 04:06:01.635367 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:01.635293    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-wld4d" podStartSLOduration=1.6352755 podStartE2EDuration="1.6352755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 04:06:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 04:06:01.63356928 +0000 UTC m=+4702.212176722" watchObservedRunningTime="2026-11-02 04:06:01.6352755 +0000 UTC m=+4702.213882882"
Nov 02 04:06:02.516817 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:02.516778    4496 patch_prober.go:29] interesting pod/e2e-worker-w4-398828-openshift-default-5647988965-wld4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.27:15021/healthz/ready\": dial tcp 10.134.1.27:15021: connect: connection refused" start-of-body=
Nov 02 04:06:02.517220 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:02.516841    4496 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-wld4d" podUID="90e07af3-2747-455c-9496-1cda9c7c6eb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.27:15021/healthz/ready\": dial tcp 10.134.1.27:15021: connect: connection refused"
Nov 02 04:06:02.520481 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:02.520431    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-wld4d" podUID="90e07af3-2747-455c-9496-1cda9c7c6eb9" containerName="istio-proxy" containerID="cri-o://d172ee06e39e35634785e2ae488571e0a80a9543a4d28d301a90eec04fa6d880" gracePeriod=30
Nov 02 04:06:03.526943 ip-10-0-133-163 kubenswrapper[4496]: E1102 04:06:03.526908    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 04:06:07.801207 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:07.801180    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-wld4d"
Nov 02 04:06:07.955918 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:07.955885    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90e07af3-2747-455c-9496-1cda9c7c6eb9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90e07af3-2747-455c-9496-1cda9c7c6eb9-workload-socket\") pod \"90e07af3-2747-455c-9496-1cda9c7c6eb9\" (UID: \"90e07af3-2747-455c-9496-1cda9c7c6eb9\") "
Nov 02 04:06:07.956137 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:07.955937    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90e07af3-2747-455c-9496-1cda9c7c6eb9-kube-api-access-bbb55\" (UniqueName: \"kubernetes.io/projected/90e07af3-2747-455c-9496-1cda9c7c6eb9-kube-api-access-bbb55\") pod \"90e07af3-2747-455c-9496-1cda9c7c6eb9\" (UID: \"90e07af3-2747-455c-9496-1cda9c7c6eb9\") "
Nov 02 04:06:07.956137 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:07.955986    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/90e07af3-2747-455c-9496-1cda9c7c6eb9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90e07af3-2747-455c-9496-1cda9c7c6eb9-istio-podinfo\") pod \"90e07af3-2747-455c-9496-1cda9c7c6eb9\" (UID: \"90e07af3-2747-455c-9496-1cda9c7c6eb9\") "
Nov 02 04:06:07.956137 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:07.956049    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90e07af3-2747-455c-9496-1cda9c7c6eb9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90e07af3-2747-455c-9496-1cda9c7c6eb9-istio-envoy\") pod \"90e07af3-2747-455c-9496-1cda9c7c6eb9\" (UID: \"90e07af3-2747-455c-9496-1cda9c7c6eb9\") "
Nov 02 04:06:07.956137 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:07.956081    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90e07af3-2747-455c-9496-1cda9c7c6eb9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90e07af3-2747-455c-9496-1cda9c7c6eb9-workload-certs\") pod \"90e07af3-2747-455c-9496-1cda9c7c6eb9\" (UID: \"90e07af3-2747-455c-9496-1cda9c7c6eb9\") "
Nov 02 04:06:07.956137 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:07.956121    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90e07af3-2747-455c-9496-1cda9c7c6eb9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90e07af3-2747-455c-9496-1cda9c7c6eb9-istio-data\") pod \"90e07af3-2747-455c-9496-1cda9c7c6eb9\" (UID: \"90e07af3-2747-455c-9496-1cda9c7c6eb9\") "
Nov 02 04:06:07.956418 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:07.956185    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90e07af3-2747-455c-9496-1cda9c7c6eb9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90e07af3-2747-455c-9496-1cda9c7c6eb9-credential-socket\") pod \"90e07af3-2747-455c-9496-1cda9c7c6eb9\" (UID: \"90e07af3-2747-455c-9496-1cda9c7c6eb9\") "
Nov 02 04:06:07.956418 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:07.956229    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90e07af3-2747-455c-9496-1cda9c7c6eb9-istio-token\" (UniqueName: \"kubernetes.io/projected/90e07af3-2747-455c-9496-1cda9c7c6eb9-istio-token\") pod \"90e07af3-2747-455c-9496-1cda9c7c6eb9\" (UID: \"90e07af3-2747-455c-9496-1cda9c7c6eb9\") "
Nov 02 04:06:07.956418 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:07.956380    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90e07af3-2747-455c-9496-1cda9c7c6eb9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90e07af3-2747-455c-9496-1cda9c7c6eb9-istiod-ca-cert\") pod \"90e07af3-2747-455c-9496-1cda9c7c6eb9\" (UID: \"90e07af3-2747-455c-9496-1cda9c7c6eb9\") "
Nov 02 04:06:07.956804 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:07.956223    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90e07af3-2747-455c-9496-1cda9c7c6eb9-workload-socket" pod "90e07af3-2747-455c-9496-1cda9c7c6eb9" (UID: "90e07af3-2747-455c-9496-1cda9c7c6eb9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 04:06:07.956804 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:07.956366    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90e07af3-2747-455c-9496-1cda9c7c6eb9-workload-certs" pod "90e07af3-2747-455c-9496-1cda9c7c6eb9" (UID: "90e07af3-2747-455c-9496-1cda9c7c6eb9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 04:06:07.956804 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:07.956581    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90e07af3-2747-455c-9496-1cda9c7c6eb9-credential-socket" pod "90e07af3-2747-455c-9496-1cda9c7c6eb9" (UID: "90e07af3-2747-455c-9496-1cda9c7c6eb9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 04:06:07.956804 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:07.956726    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90e07af3-2747-455c-9496-1cda9c7c6eb9-istio-data" pod "90e07af3-2747-455c-9496-1cda9c7c6eb9" (UID: "90e07af3-2747-455c-9496-1cda9c7c6eb9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 04:06:07.957174 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:07.957148    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90e07af3-2747-455c-9496-1cda9c7c6eb9-istiod-ca-cert" pod "90e07af3-2747-455c-9496-1cda9c7c6eb9" (UID: "90e07af3-2747-455c-9496-1cda9c7c6eb9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 04:06:07.958516 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:07.958494    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/90e07af3-2747-455c-9496-1cda9c7c6eb9-istio-podinfo" pod "90e07af3-2747-455c-9496-1cda9c7c6eb9" (UID: "90e07af3-2747-455c-9496-1cda9c7c6eb9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 04:06:07.958959 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:07.958939    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90e07af3-2747-455c-9496-1cda9c7c6eb9-istio-token" pod "90e07af3-2747-455c-9496-1cda9c7c6eb9" (UID: "90e07af3-2747-455c-9496-1cda9c7c6eb9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 04:06:07.959263 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:07.959245    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90e07af3-2747-455c-9496-1cda9c7c6eb9-istio-envoy" pod "90e07af3-2747-455c-9496-1cda9c7c6eb9" (UID: "90e07af3-2747-455c-9496-1cda9c7c6eb9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 04:06:07.960545 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:07.960525    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90e07af3-2747-455c-9496-1cda9c7c6eb9-kube-api-access-bbb55" pod "90e07af3-2747-455c-9496-1cda9c7c6eb9" (UID: "90e07af3-2747-455c-9496-1cda9c7c6eb9"). InnerVolumeSpecName "kube-api-access-bbb55". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 04:06:08.058136 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:08.058037    4496 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90e07af3-2747-455c-9496-1cda9c7c6eb9-istio-token\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:06:08.058136 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:08.058079    4496 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90e07af3-2747-455c-9496-1cda9c7c6eb9-istiod-ca-cert\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:06:08.058136 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:08.058096    4496 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90e07af3-2747-455c-9496-1cda9c7c6eb9-workload-socket\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:06:08.058136 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:08.058112    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbb55\" (UniqueName: \"kubernetes.io/projected/90e07af3-2747-455c-9496-1cda9c7c6eb9-kube-api-access-bbb55\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:06:08.058136 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:08.058128    4496 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90e07af3-2747-455c-9496-1cda9c7c6eb9-istio-podinfo\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:06:08.058431 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:08.058142    4496 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90e07af3-2747-455c-9496-1cda9c7c6eb9-istio-envoy\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:06:08.058431 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:08.058158    4496 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90e07af3-2747-455c-9496-1cda9c7c6eb9-workload-certs\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:06:08.058431 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:08.058174    4496 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90e07af3-2747-455c-9496-1cda9c7c6eb9-istio-data\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:06:08.058431 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:08.058187    4496 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90e07af3-2747-455c-9496-1cda9c7c6eb9-credential-socket\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:06:08.396232 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:08.392230    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-12c92e-b7bfdfb6d-bfbnt"]
Nov 02 04:06:08.396232 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:08.393460    4496 cpu_manager.go:414] "containerMap: removing container" podUID="90e07af3-2747-455c-9496-1cda9c7c6eb9" containerName="istio-proxy"
Nov 02 04:06:08.396232 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:08.393484    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90e07af3-2747-455c-9496-1cda9c7c6eb9" containerName="istio-proxy"
Nov 02 04:06:08.396232 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:08.393836    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="90e07af3-2747-455c-9496-1cda9c7c6eb9" containerName="istio-proxy"
Nov 02 04:06:08.398600 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:08.398574    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-12c92e-b7bfdfb6d-bfbnt"
Nov 02 04:06:08.451375 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:08.451337    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-12c92e-b7bfdfb6d-bfbnt"]
Nov 02 04:06:08.451797 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:08.451775    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-12c92e-serving-cert\""
Nov 02 04:06:08.562553 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:08.562521    4496 generic.go:356] "Generic (PLEG): container finished" podID="90e07af3-2747-455c-9496-1cda9c7c6eb9" containerID="d172ee06e39e35634785e2ae488571e0a80a9543a4d28d301a90eec04fa6d880" exitCode=0
Nov 02 04:06:08.562759 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:08.562591    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-wld4d"
Nov 02 04:06:08.562759 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:08.562609    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-wld4d" event={"ID":"90e07af3-2747-455c-9496-1cda9c7c6eb9","Type":"ContainerDied","Data":"d172ee06e39e35634785e2ae488571e0a80a9543a4d28d301a90eec04fa6d880"}
Nov 02 04:06:08.562759 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:08.562667    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-wld4d" event={"ID":"90e07af3-2747-455c-9496-1cda9c7c6eb9","Type":"ContainerDied","Data":"d38fde0e60a9fb75835dcd3b4cc8469067ad7f8a1c26ff3f02e71d25f2e01545"}
Nov 02 04:06:08.562759 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:08.562697    4496 scope.go:122] "RemoveContainer" containerID="d172ee06e39e35634785e2ae488571e0a80a9543a4d28d301a90eec04fa6d880"
Nov 02 04:06:08.563286 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:08.563261    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8342203-5e7b-43e7-94cb-06db35124720-metrics-certs\") pod \"maas-api-e2e-extmod-b-12c92e-b7bfdfb6d-bfbnt\" (UID: \"c8342203-5e7b-43e7-94cb-06db35124720\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-12c92e-b7bfdfb6d-bfbnt"
Nov 02 04:06:08.563431 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:08.563307    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ksqd\" (UniqueName: \"kubernetes.io/projected/c8342203-5e7b-43e7-94cb-06db35124720-kube-api-access-9ksqd\") pod \"maas-api-e2e-extmod-b-12c92e-b7bfdfb6d-bfbnt\" (UID: \"c8342203-5e7b-43e7-94cb-06db35124720\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-12c92e-b7bfdfb6d-bfbnt"
Nov 02 04:06:08.563506 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:08.563471    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8342203-5e7b-43e7-94cb-06db35124720-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-12c92e-b7bfdfb6d-bfbnt\" (UID: \"c8342203-5e7b-43e7-94cb-06db35124720\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-12c92e-b7bfdfb6d-bfbnt"
Nov 02 04:06:08.563569 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:08.563546    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8342203-5e7b-43e7-94cb-06db35124720-maas-api-tls\") pod \"maas-api-e2e-extmod-b-12c92e-b7bfdfb6d-bfbnt\" (UID: \"c8342203-5e7b-43e7-94cb-06db35124720\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-12c92e-b7bfdfb6d-bfbnt"
Nov 02 04:06:08.573768 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:08.573742    4496 scope.go:122] "RemoveContainer" containerID="d172ee06e39e35634785e2ae488571e0a80a9543a4d28d301a90eec04fa6d880"
Nov 02 04:06:08.574167 ip-10-0-133-163 kubenswrapper[4496]: E1102 04:06:08.574145    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d172ee06e39e35634785e2ae488571e0a80a9543a4d28d301a90eec04fa6d880\": container with ID starting with d172ee06e39e35634785e2ae488571e0a80a9543a4d28d301a90eec04fa6d880 not found: ID does not exist" containerID="d172ee06e39e35634785e2ae488571e0a80a9543a4d28d301a90eec04fa6d880"
Nov 02 04:06:08.574253 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:08.574175    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d172ee06e39e35634785e2ae488571e0a80a9543a4d28d301a90eec04fa6d880"} err="failed to get container status \"d172ee06e39e35634785e2ae488571e0a80a9543a4d28d301a90eec04fa6d880\": rpc error: code = NotFound desc = could not find container \"d172ee06e39e35634785e2ae488571e0a80a9543a4d28d301a90eec04fa6d880\": container with ID starting with d172ee06e39e35634785e2ae488571e0a80a9543a4d28d301a90eec04fa6d880 not found: ID does not exist"
Nov 02 04:06:08.664417 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:08.664331    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8342203-5e7b-43e7-94cb-06db35124720-metrics-certs\") pod \"maas-api-e2e-extmod-b-12c92e-b7bfdfb6d-bfbnt\" (UID: \"c8342203-5e7b-43e7-94cb-06db35124720\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-12c92e-b7bfdfb6d-bfbnt"
Nov 02 04:06:08.664417 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:08.664372    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ksqd\" (UniqueName: \"kubernetes.io/projected/c8342203-5e7b-43e7-94cb-06db35124720-kube-api-access-9ksqd\") pod \"maas-api-e2e-extmod-b-12c92e-b7bfdfb6d-bfbnt\" (UID: \"c8342203-5e7b-43e7-94cb-06db35124720\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-12c92e-b7bfdfb6d-bfbnt"
Nov 02 04:06:08.664674 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:08.664433    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8342203-5e7b-43e7-94cb-06db35124720-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-12c92e-b7bfdfb6d-bfbnt\" (UID: \"c8342203-5e7b-43e7-94cb-06db35124720\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-12c92e-b7bfdfb6d-bfbnt"
Nov 02 04:06:08.664674 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:08.664474    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8342203-5e7b-43e7-94cb-06db35124720-maas-api-tls\") pod \"maas-api-e2e-extmod-b-12c92e-b7bfdfb6d-bfbnt\" (UID: \"c8342203-5e7b-43e7-94cb-06db35124720\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-12c92e-b7bfdfb6d-bfbnt"
Nov 02 04:06:08.664674 ip-10-0-133-163 kubenswrapper[4496]: E1102 04:06:08.664595    4496 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-b-12c92e-serving-cert: secret "maas-api-e2e-extmod-b-12c92e-serving-cert" not found
Nov 02 04:06:08.664674 ip-10-0-133-163 kubenswrapper[4496]: E1102 04:06:08.664665    4496 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8342203-5e7b-43e7-94cb-06db35124720-maas-api-tls podName:c8342203-5e7b-43e7-94cb-06db35124720 nodeName:}" failed. No retries permitted until 2026-11-02 04:06:09.164647638 +0000 UTC m=+4709.743255014 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c8342203-5e7b-43e7-94cb-06db35124720-maas-api-tls") pod "maas-api-e2e-extmod-b-12c92e-b7bfdfb6d-bfbnt" (UID: "c8342203-5e7b-43e7-94cb-06db35124720") : secret "maas-api-e2e-extmod-b-12c92e-serving-cert" not found
Nov 02 04:06:08.665383 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:08.665356    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8342203-5e7b-43e7-94cb-06db35124720-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-12c92e-b7bfdfb6d-bfbnt\" (UID: \"c8342203-5e7b-43e7-94cb-06db35124720\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-12c92e-b7bfdfb6d-bfbnt"
Nov 02 04:06:08.667270 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:08.667233    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8342203-5e7b-43e7-94cb-06db35124720-metrics-certs\") pod \"maas-api-e2e-extmod-b-12c92e-b7bfdfb6d-bfbnt\" (UID: \"c8342203-5e7b-43e7-94cb-06db35124720\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-12c92e-b7bfdfb6d-bfbnt"
Nov 02 04:06:08.722843 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:08.722814    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ksqd\" (UniqueName: \"kubernetes.io/projected/c8342203-5e7b-43e7-94cb-06db35124720-kube-api-access-9ksqd\") pod \"maas-api-e2e-extmod-b-12c92e-b7bfdfb6d-bfbnt\" (UID: \"c8342203-5e7b-43e7-94cb-06db35124720\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-12c92e-b7bfdfb6d-bfbnt"
Nov 02 04:06:08.733065 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:08.733035    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-wld4d"]
Nov 02 04:06:08.736704 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:08.736679    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-wld4d"]
Nov 02 04:06:09.169894 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:09.169858    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8342203-5e7b-43e7-94cb-06db35124720-maas-api-tls\") pod \"maas-api-e2e-extmod-b-12c92e-b7bfdfb6d-bfbnt\" (UID: \"c8342203-5e7b-43e7-94cb-06db35124720\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-12c92e-b7bfdfb6d-bfbnt"
Nov 02 04:06:09.172497 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:09.172471    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8342203-5e7b-43e7-94cb-06db35124720-maas-api-tls\") pod \"maas-api-e2e-extmod-b-12c92e-b7bfdfb6d-bfbnt\" (UID: \"c8342203-5e7b-43e7-94cb-06db35124720\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-12c92e-b7bfdfb6d-bfbnt"
Nov 02 04:06:09.314851 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:09.314814    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-12c92e-b7bfdfb6d-bfbnt"
Nov 02 04:06:09.572790 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:09.572758    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90e07af3-2747-455c-9496-1cda9c7c6eb9" path="/var/lib/kubelet/pods/90e07af3-2747-455c-9496-1cda9c7c6eb9/volumes"
Nov 02 04:06:09.691869 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:09.691839    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-12c92e-b7bfdfb6d-bfbnt"]
Nov 02 04:06:09.693242 ip-10-0-133-163 kubenswrapper[4496]: W1102 04:06:09.693212    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8342203_5e7b_43e7_94cb_06db35124720.slice/crio-88efa3dba739a6263cc5d97bd48d67aacd96968cc374dbfcbb0005460933597c WatchSource:0}: Error finding container 88efa3dba739a6263cc5d97bd48d67aacd96968cc374dbfcbb0005460933597c: Status 404 returned error can't find the container with id 88efa3dba739a6263cc5d97bd48d67aacd96968cc374dbfcbb0005460933597c
Nov 02 04:06:10.577064 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:10.577028    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-12c92e-b7bfdfb6d-bfbnt" event={"ID":"c8342203-5e7b-43e7-94cb-06db35124720","Type":"ContainerStarted","Data":"88efa3dba739a6263cc5d97bd48d67aacd96968cc374dbfcbb0005460933597c"}
Nov 02 04:06:11.583902 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:11.583868    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-12c92e-b7bfdfb6d-bfbnt" event={"ID":"c8342203-5e7b-43e7-94cb-06db35124720","Type":"ContainerStarted","Data":"abb25b096c0cfb56210d33fed2c26bd3e917bb435a629e6ccd3e71a29088cf9e"}
Nov 02 04:06:11.584334 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:11.584025    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-12c92e-b7bfdfb6d-bfbnt"
Nov 02 04:06:11.700636 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:11.700582    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-12c92e-b7bfdfb6d-bfbnt" podStartSLOduration=2.7343717229999998 podStartE2EDuration="3.700567592s" totalImagesPullingTime="966.195869ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 04:06:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 04:06:11.699792917 +0000 UTC m=+4712.278400303" watchObservedRunningTime="2026-11-02 04:06:11.700567592 +0000 UTC m=+4712.279174972"
Nov 02 04:06:17.595661 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:06:17.595631    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-12c92e-b7bfdfb6d-bfbnt"
Nov 02 04:06:33.526881 ip-10-0-133-163 kubenswrapper[4496]: E1102 04:06:33.526803    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 04:07:03.527129 ip-10-0-133-163 kubenswrapper[4496]: E1102 04:07:03.527089    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 04:07:33.526065 ip-10-0-133-163 kubenswrapper[4496]: E1102 04:07:33.526004    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 04:07:40.533954 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:07:40.533923    4496 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hww8q_5c797b08-a314-44ed-8026-8a0ce1f6c36f/ovn-acl-logging/0.log"
Nov 02 04:07:40.551628 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:07:40.551610    4496 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hww8q_5c797b08-a314-44ed-8026-8a0ce1f6c36f/ovn-acl-logging/0.log"
Nov 02 04:07:50.702785 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:07:50.702751    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-3f24d0-5d7d5dd778-xgccr"]
Nov 02 04:07:50.703252 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:07:50.702982    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3f24d0-5d7d5dd778-xgccr" podUID="7c022b24-eb29-42be-a94f-7c1df4216688" containerName="maas-api" containerID="cri-o://28f4d798bd970d538655a5c90b1e5bd4dea9c17529e569a71ca618c748e62ed3" gracePeriod=30
Nov 02 04:07:50.950214 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:07:50.950175    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3f24d0-5d7d5dd778-xgccr"
Nov 02 04:07:51.010332 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:07:51.010306    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c022b24-eb29-42be-a94f-7c1df4216688-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c022b24-eb29-42be-a94f-7c1df4216688-openshift-service-ca\") pod \"7c022b24-eb29-42be-a94f-7c1df4216688\" (UID: \"7c022b24-eb29-42be-a94f-7c1df4216688\") "
Nov 02 04:07:51.010498 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:07:51.010366    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c022b24-eb29-42be-a94f-7c1df4216688-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c022b24-eb29-42be-a94f-7c1df4216688-metrics-certs\") pod \"7c022b24-eb29-42be-a94f-7c1df4216688\" (UID: \"7c022b24-eb29-42be-a94f-7c1df4216688\") "
Nov 02 04:07:51.010498 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:07:51.010460    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c022b24-eb29-42be-a94f-7c1df4216688-kube-api-access-dndjp\" (UniqueName: \"kubernetes.io/projected/7c022b24-eb29-42be-a94f-7c1df4216688-kube-api-access-dndjp\") pod \"7c022b24-eb29-42be-a94f-7c1df4216688\" (UID: \"7c022b24-eb29-42be-a94f-7c1df4216688\") "
Nov 02 04:07:51.010632 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:07:51.010529    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c022b24-eb29-42be-a94f-7c1df4216688-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c022b24-eb29-42be-a94f-7c1df4216688-maas-api-tls\") pod \"7c022b24-eb29-42be-a94f-7c1df4216688\" (UID: \"7c022b24-eb29-42be-a94f-7c1df4216688\") "
Nov 02 04:07:51.010632 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:07:51.010615    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c022b24-eb29-42be-a94f-7c1df4216688-openshift-service-ca" pod "7c022b24-eb29-42be-a94f-7c1df4216688" (UID: "7c022b24-eb29-42be-a94f-7c1df4216688"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 04:07:51.010857 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:07:51.010834    4496 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c022b24-eb29-42be-a94f-7c1df4216688-openshift-service-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:07:51.012551 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:07:51.012520    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c022b24-eb29-42be-a94f-7c1df4216688-kube-api-access-dndjp" pod "7c022b24-eb29-42be-a94f-7c1df4216688" (UID: "7c022b24-eb29-42be-a94f-7c1df4216688"). InnerVolumeSpecName "kube-api-access-dndjp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 04:07:51.012655 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:07:51.012572    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c022b24-eb29-42be-a94f-7c1df4216688-maas-api-tls" pod "7c022b24-eb29-42be-a94f-7c1df4216688" (UID: "7c022b24-eb29-42be-a94f-7c1df4216688"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 04:07:51.012655 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:07:51.012598    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c022b24-eb29-42be-a94f-7c1df4216688-metrics-certs" pod "7c022b24-eb29-42be-a94f-7c1df4216688" (UID: "7c022b24-eb29-42be-a94f-7c1df4216688"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 04:07:51.111414 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:07:51.111380    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c022b24-eb29-42be-a94f-7c1df4216688-maas-api-tls\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:07:51.111414 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:07:51.111408    4496 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c022b24-eb29-42be-a94f-7c1df4216688-metrics-certs\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:07:51.111414 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:07:51.111418    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dndjp\" (UniqueName: \"kubernetes.io/projected/7c022b24-eb29-42be-a94f-7c1df4216688-kube-api-access-dndjp\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:07:51.172597 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:07:51.172566    4496 generic.go:356] "Generic (PLEG): container finished" podID="7c022b24-eb29-42be-a94f-7c1df4216688" containerID="28f4d798bd970d538655a5c90b1e5bd4dea9c17529e569a71ca618c748e62ed3" exitCode=0
Nov 02 04:07:51.172762 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:07:51.172609    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3f24d0-5d7d5dd778-xgccr"
Nov 02 04:07:51.172762 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:07:51.172652    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3f24d0-5d7d5dd778-xgccr" event={"ID":"7c022b24-eb29-42be-a94f-7c1df4216688","Type":"ContainerDied","Data":"28f4d798bd970d538655a5c90b1e5bd4dea9c17529e569a71ca618c748e62ed3"}
Nov 02 04:07:51.172762 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:07:51.172702    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3f24d0-5d7d5dd778-xgccr" event={"ID":"7c022b24-eb29-42be-a94f-7c1df4216688","Type":"ContainerDied","Data":"a5505046f68bcef6a8ff35b5817ec10b18efb277325c0a57a53ae7fbf5a1b848"}
Nov 02 04:07:51.172762 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:07:51.172725    4496 scope.go:122] "RemoveContainer" containerID="28f4d798bd970d538655a5c90b1e5bd4dea9c17529e569a71ca618c748e62ed3"
Nov 02 04:07:51.183215 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:07:51.183196    4496 scope.go:122] "RemoveContainer" containerID="28f4d798bd970d538655a5c90b1e5bd4dea9c17529e569a71ca618c748e62ed3"
Nov 02 04:07:51.183480 ip-10-0-133-163 kubenswrapper[4496]: E1102 04:07:51.183463    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28f4d798bd970d538655a5c90b1e5bd4dea9c17529e569a71ca618c748e62ed3\": container with ID starting with 28f4d798bd970d538655a5c90b1e5bd4dea9c17529e569a71ca618c748e62ed3 not found: ID does not exist" containerID="28f4d798bd970d538655a5c90b1e5bd4dea9c17529e569a71ca618c748e62ed3"
Nov 02 04:07:51.183535 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:07:51.183487    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28f4d798bd970d538655a5c90b1e5bd4dea9c17529e569a71ca618c748e62ed3"} err="failed to get container status \"28f4d798bd970d538655a5c90b1e5bd4dea9c17529e569a71ca618c748e62ed3\": rpc error: code = NotFound desc = could not find container \"28f4d798bd970d538655a5c90b1e5bd4dea9c17529e569a71ca618c748e62ed3\": container with ID starting with 28f4d798bd970d538655a5c90b1e5bd4dea9c17529e569a71ca618c748e62ed3 not found: ID does not exist"
Nov 02 04:07:51.339467 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:07:51.339393    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-3f24d0-5d7d5dd778-xgccr"]
Nov 02 04:07:51.342454 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:07:51.342431    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-3f24d0-5d7d5dd778-xgccr"]
Nov 02 04:07:51.572301 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:07:51.572268    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c022b24-eb29-42be-a94f-7c1df4216688" path="/var/lib/kubelet/pods/7c022b24-eb29-42be-a94f-7c1df4216688/volumes"
Nov 02 04:08:03.527469 ip-10-0-133-163 kubenswrapper[4496]: E1102 04:08:03.527428    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 04:08:26.082168 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:08:26.082131    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-12c92e-b7bfdfb6d-bfbnt"]
Nov 02 04:08:26.082707 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:08:26.082362    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-12c92e-b7bfdfb6d-bfbnt" podUID="c8342203-5e7b-43e7-94cb-06db35124720" containerName="maas-api" containerID="cri-o://abb25b096c0cfb56210d33fed2c26bd3e917bb435a629e6ccd3e71a29088cf9e" gracePeriod=30
Nov 02 04:08:26.341252 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:08:26.341193    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-12c92e-b7bfdfb6d-bfbnt"
Nov 02 04:08:26.388103 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:08:26.388074    4496 generic.go:356] "Generic (PLEG): container finished" podID="c8342203-5e7b-43e7-94cb-06db35124720" containerID="abb25b096c0cfb56210d33fed2c26bd3e917bb435a629e6ccd3e71a29088cf9e" exitCode=0
Nov 02 04:08:26.388242 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:08:26.388117    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-12c92e-b7bfdfb6d-bfbnt"
Nov 02 04:08:26.388242 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:08:26.388138    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-12c92e-b7bfdfb6d-bfbnt" event={"ID":"c8342203-5e7b-43e7-94cb-06db35124720","Type":"ContainerDied","Data":"abb25b096c0cfb56210d33fed2c26bd3e917bb435a629e6ccd3e71a29088cf9e"}
Nov 02 04:08:26.388242 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:08:26.388184    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-12c92e-b7bfdfb6d-bfbnt" event={"ID":"c8342203-5e7b-43e7-94cb-06db35124720","Type":"ContainerDied","Data":"88efa3dba739a6263cc5d97bd48d67aacd96968cc374dbfcbb0005460933597c"}
Nov 02 04:08:26.388242 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:08:26.388205    4496 scope.go:122] "RemoveContainer" containerID="abb25b096c0cfb56210d33fed2c26bd3e917bb435a629e6ccd3e71a29088cf9e"
Nov 02 04:08:26.397425 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:08:26.397409    4496 scope.go:122] "RemoveContainer" containerID="abb25b096c0cfb56210d33fed2c26bd3e917bb435a629e6ccd3e71a29088cf9e"
Nov 02 04:08:26.397696 ip-10-0-133-163 kubenswrapper[4496]: E1102 04:08:26.397674    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abb25b096c0cfb56210d33fed2c26bd3e917bb435a629e6ccd3e71a29088cf9e\": container with ID starting with abb25b096c0cfb56210d33fed2c26bd3e917bb435a629e6ccd3e71a29088cf9e not found: ID does not exist" containerID="abb25b096c0cfb56210d33fed2c26bd3e917bb435a629e6ccd3e71a29088cf9e"
Nov 02 04:08:26.397741 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:08:26.397704    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abb25b096c0cfb56210d33fed2c26bd3e917bb435a629e6ccd3e71a29088cf9e"} err="failed to get container status \"abb25b096c0cfb56210d33fed2c26bd3e917bb435a629e6ccd3e71a29088cf9e\": rpc error: code = NotFound desc = could not find container \"abb25b096c0cfb56210d33fed2c26bd3e917bb435a629e6ccd3e71a29088cf9e\": container with ID starting with abb25b096c0cfb56210d33fed2c26bd3e917bb435a629e6ccd3e71a29088cf9e not found: ID does not exist"
Nov 02 04:08:26.456621 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:08:26.456425    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8342203-5e7b-43e7-94cb-06db35124720-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8342203-5e7b-43e7-94cb-06db35124720-metrics-certs\") pod \"c8342203-5e7b-43e7-94cb-06db35124720\" (UID: \"c8342203-5e7b-43e7-94cb-06db35124720\") "
Nov 02 04:08:26.456621 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:08:26.456520    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8342203-5e7b-43e7-94cb-06db35124720-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8342203-5e7b-43e7-94cb-06db35124720-openshift-service-ca\") pod \"c8342203-5e7b-43e7-94cb-06db35124720\" (UID: \"c8342203-5e7b-43e7-94cb-06db35124720\") "
Nov 02 04:08:26.456621 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:08:26.456581    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8342203-5e7b-43e7-94cb-06db35124720-kube-api-access-9ksqd\" (UniqueName: \"kubernetes.io/projected/c8342203-5e7b-43e7-94cb-06db35124720-kube-api-access-9ksqd\") pod \"c8342203-5e7b-43e7-94cb-06db35124720\" (UID: \"c8342203-5e7b-43e7-94cb-06db35124720\") "
Nov 02 04:08:26.456621 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:08:26.456622    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8342203-5e7b-43e7-94cb-06db35124720-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8342203-5e7b-43e7-94cb-06db35124720-maas-api-tls\") pod \"c8342203-5e7b-43e7-94cb-06db35124720\" (UID: \"c8342203-5e7b-43e7-94cb-06db35124720\") "
Nov 02 04:08:26.457287 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:08:26.457246    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8342203-5e7b-43e7-94cb-06db35124720-openshift-service-ca" pod "c8342203-5e7b-43e7-94cb-06db35124720" (UID: "c8342203-5e7b-43e7-94cb-06db35124720"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 04:08:26.459458 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:08:26.459405    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8342203-5e7b-43e7-94cb-06db35124720-maas-api-tls" pod "c8342203-5e7b-43e7-94cb-06db35124720" (UID: "c8342203-5e7b-43e7-94cb-06db35124720"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 04:08:26.459613 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:08:26.459580    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8342203-5e7b-43e7-94cb-06db35124720-metrics-certs" pod "c8342203-5e7b-43e7-94cb-06db35124720" (UID: "c8342203-5e7b-43e7-94cb-06db35124720"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 04:08:26.460213 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:08:26.460186    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8342203-5e7b-43e7-94cb-06db35124720-kube-api-access-9ksqd" pod "c8342203-5e7b-43e7-94cb-06db35124720" (UID: "c8342203-5e7b-43e7-94cb-06db35124720"). InnerVolumeSpecName "kube-api-access-9ksqd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 04:08:26.557671 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:08:26.557634    4496 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8342203-5e7b-43e7-94cb-06db35124720-openshift-service-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:08:26.557671 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:08:26.557663    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ksqd\" (UniqueName: \"kubernetes.io/projected/c8342203-5e7b-43e7-94cb-06db35124720-kube-api-access-9ksqd\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:08:26.557884 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:08:26.557681    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8342203-5e7b-43e7-94cb-06db35124720-maas-api-tls\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:08:26.557884 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:08:26.557696    4496 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8342203-5e7b-43e7-94cb-06db35124720-metrics-certs\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:08:26.809818 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:08:26.809782    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-12c92e-b7bfdfb6d-bfbnt"]
Nov 02 04:08:26.813330 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:08:26.813304    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-12c92e-b7bfdfb6d-bfbnt"]
Nov 02 04:08:27.573459 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:08:27.573424    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8342203-5e7b-43e7-94cb-06db35124720" path="/var/lib/kubelet/pods/c8342203-5e7b-43e7-94cb-06db35124720/volumes"
Nov 02 04:08:33.526148 ip-10-0-133-163 kubenswrapper[4496]: E1102 04:08:33.526111    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 04:08:46.761340 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:08:46.761305    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-aad578-6d6755f65f-5ljdx"]
Nov 02 04:08:46.868437 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:08:46.868188    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-aad578-c96769d99-2zkf4"]
Nov 02 04:08:51.761737 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:08:51.761683    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-aad578-6d6755f65f-5ljdx" podUID="19637a1f-28d0-4615-abec-aa4ee3f20a4b" containerName="payload-processing" containerID="cri-o://8cce88161dfcf9cf9bab09a3be36318db8741093357ab0a731e175956f5f39a1" gracePeriod=25
Nov 02 04:08:51.869465 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:08:51.869430    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-aad578-c96769d99-2zkf4" podUID="54092503-5ffa-4b0c-9e72-520195aa8966" containerName="payload-pre-processing" containerID="cri-o://1554d5bd975260e24cc742381dc525ea338f48388df3dc550110a015c1394ad7" gracePeriod=25
Nov 02 04:08:51.978417 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:08:51.978379    4496 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-aad578-c96769d99-2zkf4" podUID="54092503-5ffa-4b0c-9e72-520195aa8966" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 04:08:51.978751 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:08:51.978421    4496 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-aad578-6d6755f65f-5ljdx" podUID="19637a1f-28d0-4615-abec-aa4ee3f20a4b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 04:08:56.717732 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:08:56.717699    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-aad578-65f6d8585d-k7ds9"]
Nov 02 04:08:56.718179 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:08:56.717908    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-aad578-65f6d8585d-k7ds9" podUID="f4fc73f5-ff8b-4099-a240-7dce74b1fd14" containerName="maas-api" containerID="cri-o://efc9fc54d1491b2bcb1b9aa365d3277b6ee9cd150e68c3196fad1558cb90303f" gracePeriod=30
Nov 02 04:08:56.975895 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:08:56.975871    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-aad578-65f6d8585d-k7ds9"
Nov 02 04:08:57.052734 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:08:57.052692    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4fc73f5-ff8b-4099-a240-7dce74b1fd14-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4fc73f5-ff8b-4099-a240-7dce74b1fd14-metrics-certs\") pod \"f4fc73f5-ff8b-4099-a240-7dce74b1fd14\" (UID: \"f4fc73f5-ff8b-4099-a240-7dce74b1fd14\") "
Nov 02 04:08:57.052734 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:08:57.052745    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4fc73f5-ff8b-4099-a240-7dce74b1fd14-kube-api-access-lzzcm\" (UniqueName: \"kubernetes.io/projected/f4fc73f5-ff8b-4099-a240-7dce74b1fd14-kube-api-access-lzzcm\") pod \"f4fc73f5-ff8b-4099-a240-7dce74b1fd14\" (UID: \"f4fc73f5-ff8b-4099-a240-7dce74b1fd14\") "
Nov 02 04:08:57.052947 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:08:57.052857    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4fc73f5-ff8b-4099-a240-7dce74b1fd14-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4fc73f5-ff8b-4099-a240-7dce74b1fd14-maas-api-tls\") pod \"f4fc73f5-ff8b-4099-a240-7dce74b1fd14\" (UID: \"f4fc73f5-ff8b-4099-a240-7dce74b1fd14\") "
Nov 02 04:08:57.052947 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:08:57.052938    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4fc73f5-ff8b-4099-a240-7dce74b1fd14-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4fc73f5-ff8b-4099-a240-7dce74b1fd14-openshift-service-ca\") pod \"f4fc73f5-ff8b-4099-a240-7dce74b1fd14\" (UID: \"f4fc73f5-ff8b-4099-a240-7dce74b1fd14\") "
Nov 02 04:08:57.053417 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:08:57.053388    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4fc73f5-ff8b-4099-a240-7dce74b1fd14-openshift-service-ca" pod "f4fc73f5-ff8b-4099-a240-7dce74b1fd14" (UID: "f4fc73f5-ff8b-4099-a240-7dce74b1fd14"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 04:08:57.054885 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:08:57.054856    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4fc73f5-ff8b-4099-a240-7dce74b1fd14-maas-api-tls" pod "f4fc73f5-ff8b-4099-a240-7dce74b1fd14" (UID: "f4fc73f5-ff8b-4099-a240-7dce74b1fd14"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 04:08:57.054993 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:08:57.054896    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4fc73f5-ff8b-4099-a240-7dce74b1fd14-metrics-certs" pod "f4fc73f5-ff8b-4099-a240-7dce74b1fd14" (UID: "f4fc73f5-ff8b-4099-a240-7dce74b1fd14"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 04:08:57.054993 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:08:57.054945    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4fc73f5-ff8b-4099-a240-7dce74b1fd14-kube-api-access-lzzcm" pod "f4fc73f5-ff8b-4099-a240-7dce74b1fd14" (UID: "f4fc73f5-ff8b-4099-a240-7dce74b1fd14"). InnerVolumeSpecName "kube-api-access-lzzcm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 04:08:57.154419 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:08:57.154386    4496 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4fc73f5-ff8b-4099-a240-7dce74b1fd14-metrics-certs\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:08:57.154419 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:08:57.154416    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzzcm\" (UniqueName: \"kubernetes.io/projected/f4fc73f5-ff8b-4099-a240-7dce74b1fd14-kube-api-access-lzzcm\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:08:57.154419 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:08:57.154426    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4fc73f5-ff8b-4099-a240-7dce74b1fd14-maas-api-tls\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:08:57.154708 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:08:57.154435    4496 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4fc73f5-ff8b-4099-a240-7dce74b1fd14-openshift-service-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:08:57.587895 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:08:57.587862    4496 generic.go:356] "Generic (PLEG): container finished" podID="f4fc73f5-ff8b-4099-a240-7dce74b1fd14" containerID="efc9fc54d1491b2bcb1b9aa365d3277b6ee9cd150e68c3196fad1558cb90303f" exitCode=0
Nov 02 04:08:57.588106 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:08:57.587920    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-aad578-65f6d8585d-k7ds9"
Nov 02 04:08:57.588106 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:08:57.587942    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-aad578-65f6d8585d-k7ds9" event={"ID":"f4fc73f5-ff8b-4099-a240-7dce74b1fd14","Type":"ContainerDied","Data":"efc9fc54d1491b2bcb1b9aa365d3277b6ee9cd150e68c3196fad1558cb90303f"}
Nov 02 04:08:57.588106 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:08:57.588069    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-aad578-65f6d8585d-k7ds9" event={"ID":"f4fc73f5-ff8b-4099-a240-7dce74b1fd14","Type":"ContainerDied","Data":"247345f42e91ed41de8d0adcbf20dd190586b8fe1311a9ba04444f7b989bc313"}
Nov 02 04:08:57.588106 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:08:57.588103    4496 scope.go:122] "RemoveContainer" containerID="efc9fc54d1491b2bcb1b9aa365d3277b6ee9cd150e68c3196fad1558cb90303f"
Nov 02 04:08:57.597662 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:08:57.597644    4496 scope.go:122] "RemoveContainer" containerID="efc9fc54d1491b2bcb1b9aa365d3277b6ee9cd150e68c3196fad1558cb90303f"
Nov 02 04:08:57.597931 ip-10-0-133-163 kubenswrapper[4496]: E1102 04:08:57.597912    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efc9fc54d1491b2bcb1b9aa365d3277b6ee9cd150e68c3196fad1558cb90303f\": container with ID starting with efc9fc54d1491b2bcb1b9aa365d3277b6ee9cd150e68c3196fad1558cb90303f not found: ID does not exist" containerID="efc9fc54d1491b2bcb1b9aa365d3277b6ee9cd150e68c3196fad1558cb90303f"
Nov 02 04:08:57.597991 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:08:57.597938    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efc9fc54d1491b2bcb1b9aa365d3277b6ee9cd150e68c3196fad1558cb90303f"} err="failed to get container status \"efc9fc54d1491b2bcb1b9aa365d3277b6ee9cd150e68c3196fad1558cb90303f\": rpc error: code = NotFound desc = could not find container \"efc9fc54d1491b2bcb1b9aa365d3277b6ee9cd150e68c3196fad1558cb90303f\": container with ID starting with efc9fc54d1491b2bcb1b9aa365d3277b6ee9cd150e68c3196fad1558cb90303f not found: ID does not exist"
Nov 02 04:08:57.754402 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:08:57.754360    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-aad578-65f6d8585d-k7ds9"]
Nov 02 04:08:57.805140 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:08:57.805110    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-aad578-65f6d8585d-k7ds9"]
Nov 02 04:08:59.572153 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:08:59.572117    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4fc73f5-ff8b-4099-a240-7dce74b1fd14" path="/var/lib/kubelet/pods/f4fc73f5-ff8b-4099-a240-7dce74b1fd14/volumes"
Nov 02 04:09:01.978637 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:01.978597    4496 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-aad578-6d6755f65f-5ljdx" podUID="19637a1f-28d0-4615-abec-aa4ee3f20a4b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 04:09:01.979184 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:01.978597    4496 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-aad578-c96769d99-2zkf4" podUID="54092503-5ffa-4b0c-9e72-520195aa8966" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 04:09:03.526780 ip-10-0-133-163 kubenswrapper[4496]: E1102 04:09:03.526743    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 04:09:05.414912 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:05.414874    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-aad578-openshift-default-688f6b55cc-qxqmg"]
Nov 02 04:09:05.415470 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:05.415125    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-aad578-openshift-default-688f6b55cc-qxqmg" podUID="1e754d50-d985-4886-9184-102561eb17a8" containerName="istio-proxy" containerID="cri-o://6d1b07d6fcdd1c10233c620c5745a5c7a4b62b807ed9ffcc48a5c3bad31c2beb" gracePeriod=30
Nov 02 04:09:10.664431 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:10.664400    4496 generic.go:356] "Generic (PLEG): container finished" podID="1e754d50-d985-4886-9184-102561eb17a8" containerID="6d1b07d6fcdd1c10233c620c5745a5c7a4b62b807ed9ffcc48a5c3bad31c2beb" exitCode=0
Nov 02 04:09:10.664877 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:10.664469    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-aad578-openshift-default-688f6b55cc-qxqmg" event={"ID":"1e754d50-d985-4886-9184-102561eb17a8","Type":"ContainerDied","Data":"6d1b07d6fcdd1c10233c620c5745a5c7a4b62b807ed9ffcc48a5c3bad31c2beb"}
Nov 02 04:09:10.664877 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:10.664526    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-aad578-openshift-default-688f6b55cc-qxqmg" event={"ID":"1e754d50-d985-4886-9184-102561eb17a8","Type":"ContainerDied","Data":"6e3d85ec18d018c0a5ea3b0254aa9f77c456a7b3838bc834ab776dd0ee18bd75"}
Nov 02 04:09:10.664877 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:10.664544    4496 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6e3d85ec18d018c0a5ea3b0254aa9f77c456a7b3838bc834ab776dd0ee18bd75"
Nov 02 04:09:10.664877 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:10.664557    4496 scope.go:122] "RemoveContainer" containerID="2c8eac79e4c5e23da4285d294e4910ece44f2aca0053a9a741c669b4c5e04a6b"
Nov 02 04:09:10.677333 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:10.677313    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-aad578-openshift-default-688f6b55cc-qxqmg"
Nov 02 04:09:10.785794 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:10.785746    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e754d50-d985-4886-9184-102561eb17a8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e754d50-d985-4886-9184-102561eb17a8-workload-certs\") pod \"1e754d50-d985-4886-9184-102561eb17a8\" (UID: \"1e754d50-d985-4886-9184-102561eb17a8\") "
Nov 02 04:09:10.785995 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:10.785815    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e754d50-d985-4886-9184-102561eb17a8-istio-token\" (UniqueName: \"kubernetes.io/projected/1e754d50-d985-4886-9184-102561eb17a8-istio-token\") pod \"1e754d50-d985-4886-9184-102561eb17a8\" (UID: \"1e754d50-d985-4886-9184-102561eb17a8\") "
Nov 02 04:09:10.785995 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:10.785871    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1e754d50-d985-4886-9184-102561eb17a8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e754d50-d985-4886-9184-102561eb17a8-istio-podinfo\") pod \"1e754d50-d985-4886-9184-102561eb17a8\" (UID: \"1e754d50-d985-4886-9184-102561eb17a8\") "
Nov 02 04:09:10.785995 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:10.785951    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e754d50-d985-4886-9184-102561eb17a8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e754d50-d985-4886-9184-102561eb17a8-istio-data\") pod \"1e754d50-d985-4886-9184-102561eb17a8\" (UID: \"1e754d50-d985-4886-9184-102561eb17a8\") "
Nov 02 04:09:10.786199 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:10.786168    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e754d50-d985-4886-9184-102561eb17a8-workload-certs" pod "1e754d50-d985-4886-9184-102561eb17a8" (UID: "1e754d50-d985-4886-9184-102561eb17a8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 04:09:10.786258 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:10.786193    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e754d50-d985-4886-9184-102561eb17a8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e754d50-d985-4886-9184-102561eb17a8-credential-socket\") pod \"1e754d50-d985-4886-9184-102561eb17a8\" (UID: \"1e754d50-d985-4886-9184-102561eb17a8\") "
Nov 02 04:09:10.786312 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:10.786269    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e754d50-d985-4886-9184-102561eb17a8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e754d50-d985-4886-9184-102561eb17a8-istiod-ca-cert\") pod \"1e754d50-d985-4886-9184-102561eb17a8\" (UID: \"1e754d50-d985-4886-9184-102561eb17a8\") "
Nov 02 04:09:10.786369 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:10.786329    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e754d50-d985-4886-9184-102561eb17a8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e754d50-d985-4886-9184-102561eb17a8-istio-envoy\") pod \"1e754d50-d985-4886-9184-102561eb17a8\" (UID: \"1e754d50-d985-4886-9184-102561eb17a8\") "
Nov 02 04:09:10.786428 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:10.786408    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e754d50-d985-4886-9184-102561eb17a8-kube-api-access-tdjwg\" (UniqueName: \"kubernetes.io/projected/1e754d50-d985-4886-9184-102561eb17a8-kube-api-access-tdjwg\") pod \"1e754d50-d985-4886-9184-102561eb17a8\" (UID: \"1e754d50-d985-4886-9184-102561eb17a8\") "
Nov 02 04:09:10.786504 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:10.786432    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e754d50-d985-4886-9184-102561eb17a8-istio-data" pod "1e754d50-d985-4886-9184-102561eb17a8" (UID: "1e754d50-d985-4886-9184-102561eb17a8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 04:09:10.786504 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:10.786454    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e754d50-d985-4886-9184-102561eb17a8-credential-socket" pod "1e754d50-d985-4886-9184-102561eb17a8" (UID: "1e754d50-d985-4886-9184-102561eb17a8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 04:09:10.786504 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:10.786470    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e754d50-d985-4886-9184-102561eb17a8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e754d50-d985-4886-9184-102561eb17a8-workload-socket\") pod \"1e754d50-d985-4886-9184-102561eb17a8\" (UID: \"1e754d50-d985-4886-9184-102561eb17a8\") "
Nov 02 04:09:10.786758 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:10.786739    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e754d50-d985-4886-9184-102561eb17a8-istiod-ca-cert" pod "1e754d50-d985-4886-9184-102561eb17a8" (UID: "1e754d50-d985-4886-9184-102561eb17a8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 04:09:10.786758 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:10.786746    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e754d50-d985-4886-9184-102561eb17a8-workload-socket" pod "1e754d50-d985-4886-9184-102561eb17a8" (UID: "1e754d50-d985-4886-9184-102561eb17a8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 04:09:10.786922 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:10.786827    4496 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e754d50-d985-4886-9184-102561eb17a8-istio-data\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:09:10.786922 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:10.786846    4496 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e754d50-d985-4886-9184-102561eb17a8-credential-socket\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:09:10.786922 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:10.786861    4496 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e754d50-d985-4886-9184-102561eb17a8-istiod-ca-cert\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:09:10.786922 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:10.786874    4496 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e754d50-d985-4886-9184-102561eb17a8-workload-socket\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:09:10.786922 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:10.786888    4496 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e754d50-d985-4886-9184-102561eb17a8-workload-certs\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:09:10.788394 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:10.788368    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1e754d50-d985-4886-9184-102561eb17a8-istio-podinfo" pod "1e754d50-d985-4886-9184-102561eb17a8" (UID: "1e754d50-d985-4886-9184-102561eb17a8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 04:09:10.788538 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:10.788515    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e754d50-d985-4886-9184-102561eb17a8-istio-envoy" pod "1e754d50-d985-4886-9184-102561eb17a8" (UID: "1e754d50-d985-4886-9184-102561eb17a8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 04:09:10.788618 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:10.788602    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e754d50-d985-4886-9184-102561eb17a8-istio-token" pod "1e754d50-d985-4886-9184-102561eb17a8" (UID: "1e754d50-d985-4886-9184-102561eb17a8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 04:09:10.788710 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:10.788698    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e754d50-d985-4886-9184-102561eb17a8-kube-api-access-tdjwg" pod "1e754d50-d985-4886-9184-102561eb17a8" (UID: "1e754d50-d985-4886-9184-102561eb17a8"). InnerVolumeSpecName "kube-api-access-tdjwg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 04:09:10.887735 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:10.887698    4496 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e754d50-d985-4886-9184-102561eb17a8-istio-envoy\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:09:10.887735 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:10.887730    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdjwg\" (UniqueName: \"kubernetes.io/projected/1e754d50-d985-4886-9184-102561eb17a8-kube-api-access-tdjwg\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:09:10.887735 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:10.887741    4496 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e754d50-d985-4886-9184-102561eb17a8-istio-token\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:09:10.887952 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:10.887749    4496 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e754d50-d985-4886-9184-102561eb17a8-istio-podinfo\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:09:11.671559 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:11.671531    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-aad578-openshift-default-688f6b55cc-qxqmg"
Nov 02 04:09:11.956410 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:11.956378    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-aad578-openshift-default-688f6b55cc-qxqmg"]
Nov 02 04:09:11.959495 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:11.959456    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-aad578-openshift-default-688f6b55cc-qxqmg"]
Nov 02 04:09:11.978186 ip-10-0-133-163 kubenswrapper[4496]: W1102 04:09:11.978159    4496 logging.go:55] [core] [Channel #18579 SubChannel #18581]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.1.5:50052", ServerName: "10.134.1.5:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.1.5:50052: connect: connection refused"
Nov 02 04:09:12.017791 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:12.017762    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-aad578-6d6755f65f-5ljdx"
Nov 02 04:09:12.097285 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:12.097250    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19637a1f-28d0-4615-abec-aa4ee3f20a4b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19637a1f-28d0-4615-abec-aa4ee3f20a4b-plugins-config-volume\") pod \"19637a1f-28d0-4615-abec-aa4ee3f20a4b\" (UID: \"19637a1f-28d0-4615-abec-aa4ee3f20a4b\") "
Nov 02 04:09:12.097473 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:12.097293    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19637a1f-28d0-4615-abec-aa4ee3f20a4b-kube-api-access-86t5f\" (UniqueName: \"kubernetes.io/projected/19637a1f-28d0-4615-abec-aa4ee3f20a4b-kube-api-access-86t5f\") pod \"19637a1f-28d0-4615-abec-aa4ee3f20a4b\" (UID: \"19637a1f-28d0-4615-abec-aa4ee3f20a4b\") "
Nov 02 04:09:12.097765 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:12.097733    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19637a1f-28d0-4615-abec-aa4ee3f20a4b-plugins-config-volume" pod "19637a1f-28d0-4615-abec-aa4ee3f20a4b" (UID: "19637a1f-28d0-4615-abec-aa4ee3f20a4b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 04:09:12.099479 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:12.099449    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19637a1f-28d0-4615-abec-aa4ee3f20a4b-kube-api-access-86t5f" pod "19637a1f-28d0-4615-abec-aa4ee3f20a4b" (UID: "19637a1f-28d0-4615-abec-aa4ee3f20a4b"). InnerVolumeSpecName "kube-api-access-86t5f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 04:09:12.116723 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:12.116703    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-aad578-c96769d99-2zkf4"
Nov 02 04:09:12.198557 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:12.198479    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54092503-5ffa-4b0c-9e72-520195aa8966-kube-api-access-294zk\" (UniqueName: \"kubernetes.io/projected/54092503-5ffa-4b0c-9e72-520195aa8966-kube-api-access-294zk\") pod \"54092503-5ffa-4b0c-9e72-520195aa8966\" (UID: \"54092503-5ffa-4b0c-9e72-520195aa8966\") "
Nov 02 04:09:12.198557 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:12.198535    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54092503-5ffa-4b0c-9e72-520195aa8966-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54092503-5ffa-4b0c-9e72-520195aa8966-plugins-config-volume\") pod \"54092503-5ffa-4b0c-9e72-520195aa8966\" (UID: \"54092503-5ffa-4b0c-9e72-520195aa8966\") "
Nov 02 04:09:12.198896 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:12.198863    4496 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19637a1f-28d0-4615-abec-aa4ee3f20a4b-plugins-config-volume\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:09:12.198896 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:12.198891    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86t5f\" (UniqueName: \"kubernetes.io/projected/19637a1f-28d0-4615-abec-aa4ee3f20a4b-kube-api-access-86t5f\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:09:12.199107 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:12.198949    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54092503-5ffa-4b0c-9e72-520195aa8966-plugins-config-volume" pod "54092503-5ffa-4b0c-9e72-520195aa8966" (UID: "54092503-5ffa-4b0c-9e72-520195aa8966"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 04:09:12.200600 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:12.200582    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54092503-5ffa-4b0c-9e72-520195aa8966-kube-api-access-294zk" pod "54092503-5ffa-4b0c-9e72-520195aa8966" (UID: "54092503-5ffa-4b0c-9e72-520195aa8966"). InnerVolumeSpecName "kube-api-access-294zk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 04:09:12.300359 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:12.300326    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-294zk\" (UniqueName: \"kubernetes.io/projected/54092503-5ffa-4b0c-9e72-520195aa8966-kube-api-access-294zk\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:09:12.300359 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:12.300356    4496 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54092503-5ffa-4b0c-9e72-520195aa8966-plugins-config-volume\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:09:12.679590 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:12.679553    4496 generic.go:356] "Generic (PLEG): container finished" podID="19637a1f-28d0-4615-abec-aa4ee3f20a4b" containerID="8cce88161dfcf9cf9bab09a3be36318db8741093357ab0a731e175956f5f39a1" exitCode=0
Nov 02 04:09:12.680074 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:12.679621    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-aad578-6d6755f65f-5ljdx"
Nov 02 04:09:12.680074 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:12.679642    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-aad578-6d6755f65f-5ljdx" event={"ID":"19637a1f-28d0-4615-abec-aa4ee3f20a4b","Type":"ContainerDied","Data":"8cce88161dfcf9cf9bab09a3be36318db8741093357ab0a731e175956f5f39a1"}
Nov 02 04:09:12.680074 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:12.679717    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-aad578-6d6755f65f-5ljdx" event={"ID":"19637a1f-28d0-4615-abec-aa4ee3f20a4b","Type":"ContainerDied","Data":"c49de4294529c19d0959c8a23810010d1dbbcbd9c1915917148d7e430b51a901"}
Nov 02 04:09:12.680074 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:12.679753    4496 scope.go:122] "RemoveContainer" containerID="8cce88161dfcf9cf9bab09a3be36318db8741093357ab0a731e175956f5f39a1"
Nov 02 04:09:12.681243 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:12.681217    4496 generic.go:356] "Generic (PLEG): container finished" podID="54092503-5ffa-4b0c-9e72-520195aa8966" containerID="1554d5bd975260e24cc742381dc525ea338f48388df3dc550110a015c1394ad7" exitCode=0
Nov 02 04:09:12.681359 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:12.681267    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-aad578-c96769d99-2zkf4" event={"ID":"54092503-5ffa-4b0c-9e72-520195aa8966","Type":"ContainerDied","Data":"1554d5bd975260e24cc742381dc525ea338f48388df3dc550110a015c1394ad7"}
Nov 02 04:09:12.681359 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:12.681276    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-aad578-c96769d99-2zkf4"
Nov 02 04:09:12.681359 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:12.681307    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-aad578-c96769d99-2zkf4" event={"ID":"54092503-5ffa-4b0c-9e72-520195aa8966","Type":"ContainerDied","Data":"a21b49e9d7019139c4cb31774e5f6fe246ad69e98677ae048e0d1cb0562331b1"}
Nov 02 04:09:12.690070 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:12.690050    4496 scope.go:122] "RemoveContainer" containerID="8cce88161dfcf9cf9bab09a3be36318db8741093357ab0a731e175956f5f39a1"
Nov 02 04:09:12.690438 ip-10-0-133-163 kubenswrapper[4496]: E1102 04:09:12.690405    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cce88161dfcf9cf9bab09a3be36318db8741093357ab0a731e175956f5f39a1\": container with ID starting with 8cce88161dfcf9cf9bab09a3be36318db8741093357ab0a731e175956f5f39a1 not found: ID does not exist" containerID="8cce88161dfcf9cf9bab09a3be36318db8741093357ab0a731e175956f5f39a1"
Nov 02 04:09:12.690513 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:12.690449    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cce88161dfcf9cf9bab09a3be36318db8741093357ab0a731e175956f5f39a1"} err="failed to get container status \"8cce88161dfcf9cf9bab09a3be36318db8741093357ab0a731e175956f5f39a1\": rpc error: code = NotFound desc = could not find container \"8cce88161dfcf9cf9bab09a3be36318db8741093357ab0a731e175956f5f39a1\": container with ID starting with 8cce88161dfcf9cf9bab09a3be36318db8741093357ab0a731e175956f5f39a1 not found: ID does not exist"
Nov 02 04:09:12.690513 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:12.690473    4496 scope.go:122] "RemoveContainer" containerID="1554d5bd975260e24cc742381dc525ea338f48388df3dc550110a015c1394ad7"
Nov 02 04:09:12.699007 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:12.698990    4496 scope.go:122] "RemoveContainer" containerID="1554d5bd975260e24cc742381dc525ea338f48388df3dc550110a015c1394ad7"
Nov 02 04:09:12.699294 ip-10-0-133-163 kubenswrapper[4496]: E1102 04:09:12.699274    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1554d5bd975260e24cc742381dc525ea338f48388df3dc550110a015c1394ad7\": container with ID starting with 1554d5bd975260e24cc742381dc525ea338f48388df3dc550110a015c1394ad7 not found: ID does not exist" containerID="1554d5bd975260e24cc742381dc525ea338f48388df3dc550110a015c1394ad7"
Nov 02 04:09:12.699356 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:12.699301    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1554d5bd975260e24cc742381dc525ea338f48388df3dc550110a015c1394ad7"} err="failed to get container status \"1554d5bd975260e24cc742381dc525ea338f48388df3dc550110a015c1394ad7\": rpc error: code = NotFound desc = could not find container \"1554d5bd975260e24cc742381dc525ea338f48388df3dc550110a015c1394ad7\": container with ID starting with 1554d5bd975260e24cc742381dc525ea338f48388df3dc550110a015c1394ad7 not found: ID does not exist"
Nov 02 04:09:12.845862 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:12.845829    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-aad578-6d6755f65f-5ljdx"]
Nov 02 04:09:12.847984 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:12.847963    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-aad578-6d6755f65f-5ljdx"]
Nov 02 04:09:12.958176 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:12.958089    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-aad578-c96769d99-2zkf4"]
Nov 02 04:09:12.963930 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:12.963892    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-aad578-c96769d99-2zkf4"]
Nov 02 04:09:12.978760 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:12.978722    4496 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-aad578-6d6755f65f-5ljdx" podUID="19637a1f-28d0-4615-abec-aa4ee3f20a4b" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.1.6:50052\" within 1s: context deadline exceeded"
Nov 02 04:09:12.978896 ip-10-0-133-163 kubenswrapper[4496]: W1102 04:09:12.978780    4496 logging.go:55] [core] [Channel #18578 SubChannel #18580]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.1.6:50052", ServerName: "10.134.1.6:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.1.6:50052: operation was canceled"
Nov 02 04:09:12.978896 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:12.978817    4496 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-aad578-c96769d99-2zkf4" podUID="54092503-5ffa-4b0c-9e72-520195aa8966" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.1.5:50052\" within 1s: context deadline exceeded"
Nov 02 04:09:12.979052 ip-10-0-133-163 kubenswrapper[4496]: W1102 04:09:12.978923    4496 logging.go:55] [core] [Channel #18579 SubChannel #18581]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.1.5:50052", ServerName: "10.134.1.5:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.1.5:50052: operation was canceled"
Nov 02 04:09:13.575218 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:13.575181    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19637a1f-28d0-4615-abec-aa4ee3f20a4b" path="/var/lib/kubelet/pods/19637a1f-28d0-4615-abec-aa4ee3f20a4b/volumes"
Nov 02 04:09:13.575847 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:13.575827    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e754d50-d985-4886-9184-102561eb17a8" path="/var/lib/kubelet/pods/1e754d50-d985-4886-9184-102561eb17a8/volumes"
Nov 02 04:09:13.576379 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:13.576359    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54092503-5ffa-4b0c-9e72-520195aa8966" path="/var/lib/kubelet/pods/54092503-5ffa-4b0c-9e72-520195aa8966/volumes"
Nov 02 04:09:17.916619 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:17.916583    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3d14ee-6b76657b9-rs68l"]
Nov 02 04:09:17.917118 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:17.916797    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3d14ee-6b76657b9-rs68l" podUID="0591e186-a26e-46a9-b1a6-482335af9889" containerName="maas-api" containerID="cri-o://eed8eb0799e9200e460377b31b8de68ff1854cc5effd9adaec99197e99043473" gracePeriod=30
Nov 02 04:09:18.182781 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:18.182714    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3d14ee-6b76657b9-rs68l"
Nov 02 04:09:18.255409 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:18.255376    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0591e186-a26e-46a9-b1a6-482335af9889-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0591e186-a26e-46a9-b1a6-482335af9889-openshift-service-ca\") pod \"0591e186-a26e-46a9-b1a6-482335af9889\" (UID: \"0591e186-a26e-46a9-b1a6-482335af9889\") "
Nov 02 04:09:18.255575 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:18.255427    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0591e186-a26e-46a9-b1a6-482335af9889-kube-api-access-4zrb8\" (UniqueName: \"kubernetes.io/projected/0591e186-a26e-46a9-b1a6-482335af9889-kube-api-access-4zrb8\") pod \"0591e186-a26e-46a9-b1a6-482335af9889\" (UID: \"0591e186-a26e-46a9-b1a6-482335af9889\") "
Nov 02 04:09:18.255575 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:18.255485    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0591e186-a26e-46a9-b1a6-482335af9889-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0591e186-a26e-46a9-b1a6-482335af9889-metrics-certs\") pod \"0591e186-a26e-46a9-b1a6-482335af9889\" (UID: \"0591e186-a26e-46a9-b1a6-482335af9889\") "
Nov 02 04:09:18.255575 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:18.255528    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0591e186-a26e-46a9-b1a6-482335af9889-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0591e186-a26e-46a9-b1a6-482335af9889-maas-api-tls\") pod \"0591e186-a26e-46a9-b1a6-482335af9889\" (UID: \"0591e186-a26e-46a9-b1a6-482335af9889\") "
Nov 02 04:09:18.255822 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:18.255799    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0591e186-a26e-46a9-b1a6-482335af9889-openshift-service-ca" pod "0591e186-a26e-46a9-b1a6-482335af9889" (UID: "0591e186-a26e-46a9-b1a6-482335af9889"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 04:09:18.255939 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:18.255915    4496 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0591e186-a26e-46a9-b1a6-482335af9889-openshift-service-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:09:18.257547 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:18.257513    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0591e186-a26e-46a9-b1a6-482335af9889-kube-api-access-4zrb8" pod "0591e186-a26e-46a9-b1a6-482335af9889" (UID: "0591e186-a26e-46a9-b1a6-482335af9889"). InnerVolumeSpecName "kube-api-access-4zrb8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 04:09:18.257667 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:18.257625    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0591e186-a26e-46a9-b1a6-482335af9889-metrics-certs" pod "0591e186-a26e-46a9-b1a6-482335af9889" (UID: "0591e186-a26e-46a9-b1a6-482335af9889"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 04:09:18.257732 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:18.257667    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0591e186-a26e-46a9-b1a6-482335af9889-maas-api-tls" pod "0591e186-a26e-46a9-b1a6-482335af9889" (UID: "0591e186-a26e-46a9-b1a6-482335af9889"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 04:09:18.357321 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:18.357287    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zrb8\" (UniqueName: \"kubernetes.io/projected/0591e186-a26e-46a9-b1a6-482335af9889-kube-api-access-4zrb8\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:09:18.357321 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:18.357317    4496 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0591e186-a26e-46a9-b1a6-482335af9889-metrics-certs\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:09:18.357321 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:18.357327    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0591e186-a26e-46a9-b1a6-482335af9889-maas-api-tls\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:09:18.719189 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:18.719155    4496 generic.go:356] "Generic (PLEG): container finished" podID="0591e186-a26e-46a9-b1a6-482335af9889" containerID="eed8eb0799e9200e460377b31b8de68ff1854cc5effd9adaec99197e99043473" exitCode=0
Nov 02 04:09:18.719385 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:18.719205    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3d14ee-6b76657b9-rs68l"
Nov 02 04:09:18.719385 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:18.719238    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3d14ee-6b76657b9-rs68l" event={"ID":"0591e186-a26e-46a9-b1a6-482335af9889","Type":"ContainerDied","Data":"eed8eb0799e9200e460377b31b8de68ff1854cc5effd9adaec99197e99043473"}
Nov 02 04:09:18.719385 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:18.719293    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3d14ee-6b76657b9-rs68l" event={"ID":"0591e186-a26e-46a9-b1a6-482335af9889","Type":"ContainerDied","Data":"3993f8f5f241a9e48a1d11c2b13856c0b2251af2e89f6bacfe916fc2ede43e81"}
Nov 02 04:09:18.719385 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:18.719317    4496 scope.go:122] "RemoveContainer" containerID="eed8eb0799e9200e460377b31b8de68ff1854cc5effd9adaec99197e99043473"
Nov 02 04:09:18.729782 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:18.729764    4496 scope.go:122] "RemoveContainer" containerID="eed8eb0799e9200e460377b31b8de68ff1854cc5effd9adaec99197e99043473"
Nov 02 04:09:18.730103 ip-10-0-133-163 kubenswrapper[4496]: E1102 04:09:18.730075    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eed8eb0799e9200e460377b31b8de68ff1854cc5effd9adaec99197e99043473\": container with ID starting with eed8eb0799e9200e460377b31b8de68ff1854cc5effd9adaec99197e99043473 not found: ID does not exist" containerID="eed8eb0799e9200e460377b31b8de68ff1854cc5effd9adaec99197e99043473"
Nov 02 04:09:18.730203 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:18.730105    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eed8eb0799e9200e460377b31b8de68ff1854cc5effd9adaec99197e99043473"} err="failed to get container status \"eed8eb0799e9200e460377b31b8de68ff1854cc5effd9adaec99197e99043473\": rpc error: code = NotFound desc = could not find container \"eed8eb0799e9200e460377b31b8de68ff1854cc5effd9adaec99197e99043473\": container with ID starting with eed8eb0799e9200e460377b31b8de68ff1854cc5effd9adaec99197e99043473 not found: ID does not exist"
Nov 02 04:09:18.885770 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:18.885690    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3d14ee-6b76657b9-rs68l"]
Nov 02 04:09:18.936173 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:18.936141    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3d14ee-6b76657b9-rs68l"]
Nov 02 04:09:19.572078 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:19.572046    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0591e186-a26e-46a9-b1a6-482335af9889" path="/var/lib/kubelet/pods/0591e186-a26e-46a9-b1a6-482335af9889/volumes"
Nov 02 04:09:30.226773 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:30.226692    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-858236-c9zmt"]
Nov 02 04:09:30.227369 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:30.227348    4496 cpu_manager.go:414] "containerMap: removing container" podUID="0591e186-a26e-46a9-b1a6-482335af9889" containerName="maas-api"
Nov 02 04:09:30.227461 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:30.227371    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0591e186-a26e-46a9-b1a6-482335af9889" containerName="maas-api"
Nov 02 04:09:30.227461 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:30.227400    4496 cpu_manager.go:414] "containerMap: removing container" podUID="54092503-5ffa-4b0c-9e72-520195aa8966" containerName="payload-pre-processing"
Nov 02 04:09:30.227461 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:30.227409    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54092503-5ffa-4b0c-9e72-520195aa8966" containerName="payload-pre-processing"
Nov 02 04:09:30.227461 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:30.227427    4496 cpu_manager.go:414] "containerMap: removing container" podUID="c8342203-5e7b-43e7-94cb-06db35124720" containerName="maas-api"
Nov 02 04:09:30.227461 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:30.227435    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8342203-5e7b-43e7-94cb-06db35124720" containerName="maas-api"
Nov 02 04:09:30.227461 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:30.227448    4496 cpu_manager.go:414] "containerMap: removing container" podUID="19637a1f-28d0-4615-abec-aa4ee3f20a4b" containerName="payload-processing"
Nov 02 04:09:30.227461 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:30.227456    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19637a1f-28d0-4615-abec-aa4ee3f20a4b" containerName="payload-processing"
Nov 02 04:09:30.227836 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:30.227472    4496 cpu_manager.go:414] "containerMap: removing container" podUID="f4fc73f5-ff8b-4099-a240-7dce74b1fd14" containerName="maas-api"
Nov 02 04:09:30.227836 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:30.227480    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4fc73f5-ff8b-4099-a240-7dce74b1fd14" containerName="maas-api"
Nov 02 04:09:30.227836 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:30.227492    4496 cpu_manager.go:414] "containerMap: removing container" podUID="7c022b24-eb29-42be-a94f-7c1df4216688" containerName="maas-api"
Nov 02 04:09:30.227836 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:30.227499    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c022b24-eb29-42be-a94f-7c1df4216688" containerName="maas-api"
Nov 02 04:09:30.227836 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:30.227519    4496 cpu_manager.go:414] "containerMap: removing container" podUID="1e754d50-d985-4886-9184-102561eb17a8" containerName="istio-proxy"
Nov 02 04:09:30.227836 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:30.227527    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e754d50-d985-4886-9184-102561eb17a8" containerName="istio-proxy"
Nov 02 04:09:30.227836 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:30.227539    4496 cpu_manager.go:414] "containerMap: removing container" podUID="1e754d50-d985-4886-9184-102561eb17a8" containerName="istio-proxy"
Nov 02 04:09:30.227836 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:30.227548    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e754d50-d985-4886-9184-102561eb17a8" containerName="istio-proxy"
Nov 02 04:09:30.227836 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:30.227670    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="0591e186-a26e-46a9-b1a6-482335af9889" containerName="maas-api"
Nov 02 04:09:30.227836 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:30.227687    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="54092503-5ffa-4b0c-9e72-520195aa8966" containerName="payload-pre-processing"
Nov 02 04:09:30.227836 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:30.227696    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8342203-5e7b-43e7-94cb-06db35124720" containerName="maas-api"
Nov 02 04:09:30.227836 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:30.227706    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="19637a1f-28d0-4615-abec-aa4ee3f20a4b" containerName="payload-processing"
Nov 02 04:09:30.227836 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:30.227715    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4fc73f5-ff8b-4099-a240-7dce74b1fd14" containerName="maas-api"
Nov 02 04:09:30.227836 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:30.227723    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c022b24-eb29-42be-a94f-7c1df4216688" containerName="maas-api"
Nov 02 04:09:30.227836 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:30.227732    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e754d50-d985-4886-9184-102561eb17a8" containerName="istio-proxy"
Nov 02 04:09:30.227836 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:30.227741    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e754d50-d985-4886-9184-102561eb17a8" containerName="istio-proxy"
Nov 02 04:09:30.231165 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:30.231144    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-858236-c9zmt"
Nov 02 04:09:30.287964 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:30.287925    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-858236-c9zmt"]
Nov 02 04:09:30.335224 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:30.335192    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 04:09:30.335371 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:30.335246    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9z5bk\""
Nov 02 04:09:30.376150 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:30.376119    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9f8e30e-8655-426c-9ebe-4911c950e579-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-858236-c9zmt\" (UID: \"a9f8e30e-8655-426c-9ebe-4911c950e579\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-858236-c9zmt"
Nov 02 04:09:30.376339 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:30.376312    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9f8e30e-8655-426c-9ebe-4911c950e579-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-858236-c9zmt\" (UID: \"a9f8e30e-8655-426c-9ebe-4911c950e579\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-858236-c9zmt"
Nov 02 04:09:30.376406 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:30.376360    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jj8fm\" (UniqueName: \"kubernetes.io/projected/a9f8e30e-8655-426c-9ebe-4911c950e579-kube-api-access-jj8fm\") pod \"maas-api-revoke-keys-e2e-worker-w3-858236-c9zmt\" (UID: \"a9f8e30e-8655-426c-9ebe-4911c950e579\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-858236-c9zmt"
Nov 02 04:09:30.386622 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:30.386587    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 04:09:30.386622 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:30.386613    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 04:09:30.477241 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:30.477167    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9f8e30e-8655-426c-9ebe-4911c950e579-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-858236-c9zmt\" (UID: \"a9f8e30e-8655-426c-9ebe-4911c950e579\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-858236-c9zmt"
Nov 02 04:09:30.477241 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:30.477204    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jj8fm\" (UniqueName: \"kubernetes.io/projected/a9f8e30e-8655-426c-9ebe-4911c950e579-kube-api-access-jj8fm\") pod \"maas-api-revoke-keys-e2e-worker-w3-858236-c9zmt\" (UID: \"a9f8e30e-8655-426c-9ebe-4911c950e579\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-858236-c9zmt"
Nov 02 04:09:30.477451 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:30.477249    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9f8e30e-8655-426c-9ebe-4911c950e579-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-858236-c9zmt\" (UID: \"a9f8e30e-8655-426c-9ebe-4911c950e579\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-858236-c9zmt"
Nov 02 04:09:30.477982 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:30.477963    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9f8e30e-8655-426c-9ebe-4911c950e579-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-858236-c9zmt\" (UID: \"a9f8e30e-8655-426c-9ebe-4911c950e579\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-858236-c9zmt"
Nov 02 04:09:30.479507 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:30.479487    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9f8e30e-8655-426c-9ebe-4911c950e579-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-858236-c9zmt\" (UID: \"a9f8e30e-8655-426c-9ebe-4911c950e579\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-858236-c9zmt"
Nov 02 04:09:30.534599 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:30.534576    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jj8fm\" (UniqueName: \"kubernetes.io/projected/a9f8e30e-8655-426c-9ebe-4911c950e579-kube-api-access-jj8fm\") pod \"maas-api-revoke-keys-e2e-worker-w3-858236-c9zmt\" (UID: \"a9f8e30e-8655-426c-9ebe-4911c950e579\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-858236-c9zmt"
Nov 02 04:09:30.543724 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:30.543701    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-858236-c9zmt"
Nov 02 04:09:30.924855 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:30.924829    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-858236-c9zmt"]
Nov 02 04:09:30.926828 ip-10-0-133-163 kubenswrapper[4496]: W1102 04:09:30.926796    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9f8e30e_8655_426c_9ebe_4911c950e579.slice/crio-57fd66444356f088a3c819cdfb44fffd5a98adb3f7d14c3b64187e5b89f0a43a WatchSource:0}: Error finding container 57fd66444356f088a3c819cdfb44fffd5a98adb3f7d14c3b64187e5b89f0a43a: Status 404 returned error can't find the container with id 57fd66444356f088a3c819cdfb44fffd5a98adb3f7d14c3b64187e5b89f0a43a
Nov 02 04:09:31.797092 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:31.797054    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-858236-c9zmt" event={"ID":"a9f8e30e-8655-426c-9ebe-4911c950e579","Type":"ContainerStarted","Data":"8a91dc2fbc2735dce85a83878433d611df7bfc99eacb45f8ad2bbaaf4049ea8d"}
Nov 02 04:09:31.797506 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:31.797123    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-858236-c9zmt" event={"ID":"a9f8e30e-8655-426c-9ebe-4911c950e579","Type":"ContainerStarted","Data":"57fd66444356f088a3c819cdfb44fffd5a98adb3f7d14c3b64187e5b89f0a43a"}
Nov 02 04:09:31.913148 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:31.913094    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-858236-c9zmt" podStartSLOduration=1.913079175 podStartE2EDuration="1.913079175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 04:09:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 04:09:31.911600012 +0000 UTC m=+4912.490207395" watchObservedRunningTime="2026-11-02 04:09:31.913079175 +0000 UTC m=+4912.491686554"
Nov 02 04:09:32.804792 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:32.804746    4496 generic.go:356] "Generic (PLEG): container finished" podID="a9f8e30e-8655-426c-9ebe-4911c950e579" containerID="8a91dc2fbc2735dce85a83878433d611df7bfc99eacb45f8ad2bbaaf4049ea8d" exitCode=0
Nov 02 04:09:32.805286 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:32.804817    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-858236-c9zmt" event={"ID":"a9f8e30e-8655-426c-9ebe-4911c950e579","Type":"ContainerDied","Data":"8a91dc2fbc2735dce85a83878433d611df7bfc99eacb45f8ad2bbaaf4049ea8d"}
Nov 02 04:09:33.526488 ip-10-0-133-163 kubenswrapper[4496]: E1102 04:09:33.526451    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 04:09:33.936522 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:33.936494    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-858236-c9zmt"
Nov 02 04:09:34.009038 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:34.008993    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9f8e30e-8655-426c-9ebe-4911c950e579-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9f8e30e-8655-426c-9ebe-4911c950e579-maas-api-xks-ca\") pod \"a9f8e30e-8655-426c-9ebe-4911c950e579\" (UID: \"a9f8e30e-8655-426c-9ebe-4911c950e579\") "
Nov 02 04:09:34.009171 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:34.009117    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9f8e30e-8655-426c-9ebe-4911c950e579-kube-api-access-jj8fm\" (UniqueName: \"kubernetes.io/projected/a9f8e30e-8655-426c-9ebe-4911c950e579-kube-api-access-jj8fm\") pod \"a9f8e30e-8655-426c-9ebe-4911c950e579\" (UID: \"a9f8e30e-8655-426c-9ebe-4911c950e579\") "
Nov 02 04:09:34.009171 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:34.009140    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9f8e30e-8655-426c-9ebe-4911c950e579-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9f8e30e-8655-426c-9ebe-4911c950e579-maas-api-openshift-ca\") pod \"a9f8e30e-8655-426c-9ebe-4911c950e579\" (UID: \"a9f8e30e-8655-426c-9ebe-4911c950e579\") "
Nov 02 04:09:34.009538 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:34.009511    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9f8e30e-8655-426c-9ebe-4911c950e579-maas-api-openshift-ca" pod "a9f8e30e-8655-426c-9ebe-4911c950e579" (UID: "a9f8e30e-8655-426c-9ebe-4911c950e579"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 04:09:34.010955 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:34.010935    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9f8e30e-8655-426c-9ebe-4911c950e579-maas-api-xks-ca" pod "a9f8e30e-8655-426c-9ebe-4911c950e579" (UID: "a9f8e30e-8655-426c-9ebe-4911c950e579"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 04:09:34.011151 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:34.011131    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9f8e30e-8655-426c-9ebe-4911c950e579-kube-api-access-jj8fm" pod "a9f8e30e-8655-426c-9ebe-4911c950e579" (UID: "a9f8e30e-8655-426c-9ebe-4911c950e579"). InnerVolumeSpecName "kube-api-access-jj8fm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 04:09:34.110123 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:34.110044    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9f8e30e-8655-426c-9ebe-4911c950e579-maas-api-xks-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:09:34.110123 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:34.110073    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jj8fm\" (UniqueName: \"kubernetes.io/projected/a9f8e30e-8655-426c-9ebe-4911c950e579-kube-api-access-jj8fm\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:09:34.110123 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:34.110084    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9f8e30e-8655-426c-9ebe-4911c950e579-maas-api-openshift-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:09:34.665703 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:34.665656    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-858236/distinct-w3-858236-kserve-df6dbd64b-qbk9g"]
Nov 02 04:09:34.818362 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:34.818333    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-858236-c9zmt"
Nov 02 04:09:34.818545 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:34.818335    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-858236-c9zmt" event={"ID":"a9f8e30e-8655-426c-9ebe-4911c950e579","Type":"ContainerDied","Data":"57fd66444356f088a3c819cdfb44fffd5a98adb3f7d14c3b64187e5b89f0a43a"}
Nov 02 04:09:34.818545 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:34.818462    4496 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="57fd66444356f088a3c819cdfb44fffd5a98adb3f7d14c3b64187e5b89f0a43a"
Nov 02 04:09:41.707755 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:41.707718    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-858236-646789f8d5-hzm6p"]
Nov 02 04:09:41.814384 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:41.814353    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-858236-5c5788679f-7r9bv"]
Nov 02 04:09:42.872163 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:42.872131    4496 scope.go:122] "RemoveContainer" containerID="6d1b07d6fcdd1c10233c620c5745a5c7a4b62b807ed9ffcc48a5c3bad31c2beb"
Nov 02 04:09:46.708251 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:46.708207    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-858236-646789f8d5-hzm6p" podUID="09a61fb0-97bf-4daa-8bb0-d1d07bc858fd" containerName="payload-processing" containerID="cri-o://0bf2270fc4c4642fbff347fc13b7015364e2c92d06807d7f19943e35bb55d5e7" gracePeriod=25
Nov 02 04:09:46.814800 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:46.814761    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-858236-5c5788679f-7r9bv" podUID="4ea5b3ef-2d8a-49a5-be96-3a9b3dc766be" containerName="payload-pre-processing" containerID="cri-o://ad12f9d3f2e8c81ea9b3d40c001151ff6a4cce7393b642cb0132494d121ca366" gracePeriod=25
Nov 02 04:09:48.566813 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:48.566770    4496 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-858236-5c5788679f-7r9bv" podUID="4ea5b3ef-2d8a-49a5-be96-3a9b3dc766be" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 04:09:49.000260 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:49.000231    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29b0784c-4d28j"]
Nov 02 04:09:49.000690 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:49.000678    4496 cpu_manager.go:414] "containerMap: removing container" podUID="a9f8e30e-8655-426c-9ebe-4911c950e579" containerName="revoke-keys"
Nov 02 04:09:49.000742 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:49.000692    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9f8e30e-8655-426c-9ebe-4911c950e579" containerName="revoke-keys"
Nov 02 04:09:49.000779 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:49.000754    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9f8e30e-8655-426c-9ebe-4911c950e579" containerName="revoke-keys"
Nov 02 04:09:49.005092 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:49.005074    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29b0784c-4d28j"
Nov 02 04:09:49.060163 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:49.060126    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ec2533a-c319-4438-9b79-44bc348b3005-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29b0784c-4d28j\" (UID: \"3ec2533a-c319-4438-9b79-44bc348b3005\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29b0784c-4d28j"
Nov 02 04:09:49.060313 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:49.060221    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ec2533a-c319-4438-9b79-44bc348b3005-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29b0784c-4d28j\" (UID: \"3ec2533a-c319-4438-9b79-44bc348b3005\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29b0784c-4d28j"
Nov 02 04:09:49.060313 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:49.060252    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpfbh\" (UniqueName: \"kubernetes.io/projected/3ec2533a-c319-4438-9b79-44bc348b3005-kube-api-access-mpfbh\") pod \"maas-api-revoke-sub-29b0784c-4d28j\" (UID: \"3ec2533a-c319-4438-9b79-44bc348b3005\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29b0784c-4d28j"
Nov 02 04:09:49.060792 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:49.060767    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29b0784c-4d28j"]
Nov 02 04:09:49.108146 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:49.108107    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 04:09:49.108296 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:49.108147    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 04:09:49.108296 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:49.108213    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 04:09:49.108296 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:49.108226    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9z5bk\""
Nov 02 04:09:49.161336 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:49.161306    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ec2533a-c319-4438-9b79-44bc348b3005-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29b0784c-4d28j\" (UID: \"3ec2533a-c319-4438-9b79-44bc348b3005\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29b0784c-4d28j"
Nov 02 04:09:49.161493 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:49.161345    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpfbh\" (UniqueName: \"kubernetes.io/projected/3ec2533a-c319-4438-9b79-44bc348b3005-kube-api-access-mpfbh\") pod \"maas-api-revoke-sub-29b0784c-4d28j\" (UID: \"3ec2533a-c319-4438-9b79-44bc348b3005\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29b0784c-4d28j"
Nov 02 04:09:49.161493 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:49.161456    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ec2533a-c319-4438-9b79-44bc348b3005-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29b0784c-4d28j\" (UID: \"3ec2533a-c319-4438-9b79-44bc348b3005\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29b0784c-4d28j"
Nov 02 04:09:49.162007 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:49.161986    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ec2533a-c319-4438-9b79-44bc348b3005-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29b0784c-4d28j\" (UID: \"3ec2533a-c319-4438-9b79-44bc348b3005\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29b0784c-4d28j"
Nov 02 04:09:49.163783 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:49.163761    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ec2533a-c319-4438-9b79-44bc348b3005-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29b0784c-4d28j\" (UID: \"3ec2533a-c319-4438-9b79-44bc348b3005\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29b0784c-4d28j"
Nov 02 04:09:49.218559 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:49.218536    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpfbh\" (UniqueName: \"kubernetes.io/projected/3ec2533a-c319-4438-9b79-44bc348b3005-kube-api-access-mpfbh\") pod \"maas-api-revoke-sub-29b0784c-4d28j\" (UID: \"3ec2533a-c319-4438-9b79-44bc348b3005\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29b0784c-4d28j"
Nov 02 04:09:49.317701 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:49.317610    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29b0784c-4d28j"
Nov 02 04:09:49.576583 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:49.576500    4496 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-858236-646789f8d5-hzm6p" podUID="09a61fb0-97bf-4daa-8bb0-d1d07bc858fd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 04:09:49.686939 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:49.686553    4496 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-858236/distinct-w3-858236-kserve-df6dbd64b-qbk9g" podUID="719bb9e5-ea72-4782-bb17-034b909f874d" containerName="main" containerID="cri-o://d8a5a9e207fa197916278177f145dd27a6a162de125fc82e25b9d35d237a634d" gracePeriod=45
Nov 02 04:09:49.690641 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:49.690611    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29b0784c-4d28j"]
Nov 02 04:09:49.692093 ip-10-0-133-163 kubenswrapper[4496]: W1102 04:09:49.692064    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ec2533a_c319_4438_9b79_44bc348b3005.slice/crio-bcabac0a57fd6c84f210c92edf238be214478224cde0bdd42812c308fcc899d0 WatchSource:0}: Error finding container bcabac0a57fd6c84f210c92edf238be214478224cde0bdd42812c308fcc899d0: Status 404 returned error can't find the container with id bcabac0a57fd6c84f210c92edf238be214478224cde0bdd42812c308fcc899d0
Nov 02 04:09:49.909619 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:49.909586    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29b0784c-4d28j" event={"ID":"3ec2533a-c319-4438-9b79-44bc348b3005","Type":"ContainerStarted","Data":"bcabac0a57fd6c84f210c92edf238be214478224cde0bdd42812c308fcc899d0"}
Nov 02 04:09:49.910972 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:49.910950    4496 generic.go:356] "Generic (PLEG): container finished" podID="719bb9e5-ea72-4782-bb17-034b909f874d" containerID="d8a5a9e207fa197916278177f145dd27a6a162de125fc82e25b9d35d237a634d" exitCode=0
Nov 02 04:09:49.911135 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:49.910980    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-858236/distinct-w3-858236-kserve-df6dbd64b-qbk9g" event={"ID":"719bb9e5-ea72-4782-bb17-034b909f874d","Type":"ContainerDied","Data":"d8a5a9e207fa197916278177f145dd27a6a162de125fc82e25b9d35d237a634d"}
Nov 02 04:09:49.915468 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:49.915447    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-858236/distinct-w3-858236-kserve-df6dbd64b-qbk9g"
Nov 02 04:09:49.976721 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:49.975567    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/719bb9e5-ea72-4782-bb17-034b909f874d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/719bb9e5-ea72-4782-bb17-034b909f874d-model-cache\") pod \"719bb9e5-ea72-4782-bb17-034b909f874d\" (UID: \"719bb9e5-ea72-4782-bb17-034b909f874d\") "
Nov 02 04:09:49.976721 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:49.975641    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/719bb9e5-ea72-4782-bb17-034b909f874d-tls-certs\" (UniqueName: \"kubernetes.io/secret/719bb9e5-ea72-4782-bb17-034b909f874d-tls-certs\") pod \"719bb9e5-ea72-4782-bb17-034b909f874d\" (UID: \"719bb9e5-ea72-4782-bb17-034b909f874d\") "
Nov 02 04:09:49.976721 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:49.975682    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/719bb9e5-ea72-4782-bb17-034b909f874d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/719bb9e5-ea72-4782-bb17-034b909f874d-tmp-dir\") pod \"719bb9e5-ea72-4782-bb17-034b909f874d\" (UID: \"719bb9e5-ea72-4782-bb17-034b909f874d\") "
Nov 02 04:09:49.976721 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:49.975736    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/719bb9e5-ea72-4782-bb17-034b909f874d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/719bb9e5-ea72-4782-bb17-034b909f874d-dshm\") pod \"719bb9e5-ea72-4782-bb17-034b909f874d\" (UID: \"719bb9e5-ea72-4782-bb17-034b909f874d\") "
Nov 02 04:09:49.976721 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:49.975765    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/719bb9e5-ea72-4782-bb17-034b909f874d-home\" (UniqueName: \"kubernetes.io/empty-dir/719bb9e5-ea72-4782-bb17-034b909f874d-home\") pod \"719bb9e5-ea72-4782-bb17-034b909f874d\" (UID: \"719bb9e5-ea72-4782-bb17-034b909f874d\") "
Nov 02 04:09:49.976721 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:49.975808    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/719bb9e5-ea72-4782-bb17-034b909f874d-kube-api-access-66dpg\" (UniqueName: \"kubernetes.io/projected/719bb9e5-ea72-4782-bb17-034b909f874d-kube-api-access-66dpg\") pod \"719bb9e5-ea72-4782-bb17-034b909f874d\" (UID: \"719bb9e5-ea72-4782-bb17-034b909f874d\") "
Nov 02 04:09:49.978593 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:49.978551    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/719bb9e5-ea72-4782-bb17-034b909f874d-tmp-dir" pod "719bb9e5-ea72-4782-bb17-034b909f874d" (UID: "719bb9e5-ea72-4782-bb17-034b909f874d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 04:09:49.978711 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:49.978642    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/719bb9e5-ea72-4782-bb17-034b909f874d-model-cache" pod "719bb9e5-ea72-4782-bb17-034b909f874d" (UID: "719bb9e5-ea72-4782-bb17-034b909f874d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 04:09:49.978711 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:49.978682    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/719bb9e5-ea72-4782-bb17-034b909f874d-home" pod "719bb9e5-ea72-4782-bb17-034b909f874d" (UID: "719bb9e5-ea72-4782-bb17-034b909f874d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 04:09:49.980833 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:49.980810    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/719bb9e5-ea72-4782-bb17-034b909f874d-tls-certs" pod "719bb9e5-ea72-4782-bb17-034b909f874d" (UID: "719bb9e5-ea72-4782-bb17-034b909f874d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 04:09:49.980929 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:49.980813    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/719bb9e5-ea72-4782-bb17-034b909f874d-dshm" pod "719bb9e5-ea72-4782-bb17-034b909f874d" (UID: "719bb9e5-ea72-4782-bb17-034b909f874d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 04:09:49.980929 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:49.980868    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/719bb9e5-ea72-4782-bb17-034b909f874d-kube-api-access-66dpg" pod "719bb9e5-ea72-4782-bb17-034b909f874d" (UID: "719bb9e5-ea72-4782-bb17-034b909f874d"). InnerVolumeSpecName "kube-api-access-66dpg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 04:09:50.077555 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:50.077517    4496 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/719bb9e5-ea72-4782-bb17-034b909f874d-model-cache\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:09:50.077555 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:50.077555    4496 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/719bb9e5-ea72-4782-bb17-034b909f874d-tls-certs\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:09:50.077826 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:50.077586    4496 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/719bb9e5-ea72-4782-bb17-034b909f874d-tmp-dir\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:09:50.077826 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:50.077594    4496 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/719bb9e5-ea72-4782-bb17-034b909f874d-dshm\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:09:50.077826 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:50.077603    4496 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/719bb9e5-ea72-4782-bb17-034b909f874d-home\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:09:50.077826 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:50.077612    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66dpg\" (UniqueName: \"kubernetes.io/projected/719bb9e5-ea72-4782-bb17-034b909f874d-kube-api-access-66dpg\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:09:50.917667 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:50.917566    4496 generic.go:356] "Generic (PLEG): container finished" podID="3ec2533a-c319-4438-9b79-44bc348b3005" containerID="99d11b52461d6a66ba12d9960689bee2fc9e6a6c26e7b9b3e258e51ce401b34b" exitCode=0
Nov 02 04:09:50.918096 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:50.917662    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29b0784c-4d28j" event={"ID":"3ec2533a-c319-4438-9b79-44bc348b3005","Type":"ContainerDied","Data":"99d11b52461d6a66ba12d9960689bee2fc9e6a6c26e7b9b3e258e51ce401b34b"}
Nov 02 04:09:50.919101 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:50.919084    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-858236/distinct-w3-858236-kserve-df6dbd64b-qbk9g"
Nov 02 04:09:50.919187 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:50.919116    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-858236/distinct-w3-858236-kserve-df6dbd64b-qbk9g" event={"ID":"719bb9e5-ea72-4782-bb17-034b909f874d","Type":"ContainerDied","Data":"e4390ffb363b5104dfae4ca0666e21c1e89fd5ef5d82caa3fff7bf9d990a84e5"}
Nov 02 04:09:50.919187 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:50.919168    4496 scope.go:122] "RemoveContainer" containerID="d8a5a9e207fa197916278177f145dd27a6a162de125fc82e25b9d35d237a634d"
Nov 02 04:09:51.146625 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:51.146586    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-858236/distinct-w3-858236-kserve-df6dbd64b-qbk9g"]
Nov 02 04:09:51.149926 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:51.149899    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-858236/distinct-w3-858236-kserve-df6dbd64b-qbk9g"]
Nov 02 04:09:51.572980 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:51.572942    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="719bb9e5-ea72-4782-bb17-034b909f874d" path="/var/lib/kubelet/pods/719bb9e5-ea72-4782-bb17-034b909f874d/volumes"
Nov 02 04:09:52.058752 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:52.058726    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29b0784c-4d28j"
Nov 02 04:09:52.197584 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:52.197503    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ec2533a-c319-4438-9b79-44bc348b3005-kube-api-access-mpfbh\" (UniqueName: \"kubernetes.io/projected/3ec2533a-c319-4438-9b79-44bc348b3005-kube-api-access-mpfbh\") pod \"3ec2533a-c319-4438-9b79-44bc348b3005\" (UID: \"3ec2533a-c319-4438-9b79-44bc348b3005\") "
Nov 02 04:09:52.197584 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:52.197556    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ec2533a-c319-4438-9b79-44bc348b3005-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ec2533a-c319-4438-9b79-44bc348b3005-maas-api-openshift-ca\") pod \"3ec2533a-c319-4438-9b79-44bc348b3005\" (UID: \"3ec2533a-c319-4438-9b79-44bc348b3005\") "
Nov 02 04:09:52.197823 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:52.197670    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ec2533a-c319-4438-9b79-44bc348b3005-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ec2533a-c319-4438-9b79-44bc348b3005-maas-api-xks-ca\") pod \"3ec2533a-c319-4438-9b79-44bc348b3005\" (UID: \"3ec2533a-c319-4438-9b79-44bc348b3005\") "
Nov 02 04:09:52.197952 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:52.197927    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ec2533a-c319-4438-9b79-44bc348b3005-maas-api-openshift-ca" pod "3ec2533a-c319-4438-9b79-44bc348b3005" (UID: "3ec2533a-c319-4438-9b79-44bc348b3005"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 04:09:52.199697 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:52.199670    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ec2533a-c319-4438-9b79-44bc348b3005-maas-api-xks-ca" pod "3ec2533a-c319-4438-9b79-44bc348b3005" (UID: "3ec2533a-c319-4438-9b79-44bc348b3005"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 04:09:52.199779 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:52.199717    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ec2533a-c319-4438-9b79-44bc348b3005-kube-api-access-mpfbh" pod "3ec2533a-c319-4438-9b79-44bc348b3005" (UID: "3ec2533a-c319-4438-9b79-44bc348b3005"). InnerVolumeSpecName "kube-api-access-mpfbh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 04:09:52.298342 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:52.298305    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ec2533a-c319-4438-9b79-44bc348b3005-maas-api-xks-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:09:52.298342 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:52.298334    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpfbh\" (UniqueName: \"kubernetes.io/projected/3ec2533a-c319-4438-9b79-44bc348b3005-kube-api-access-mpfbh\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:09:52.298342 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:52.298346    4496 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ec2533a-c319-4438-9b79-44bc348b3005-maas-api-openshift-ca\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:09:52.933158 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:52.933122    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29b0784c-4d28j"
Nov 02 04:09:52.933363 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:52.933157    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29b0784c-4d28j" event={"ID":"3ec2533a-c319-4438-9b79-44bc348b3005","Type":"ContainerDied","Data":"bcabac0a57fd6c84f210c92edf238be214478224cde0bdd42812c308fcc899d0"}
Nov 02 04:09:52.933363 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:52.933206    4496 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bcabac0a57fd6c84f210c92edf238be214478224cde0bdd42812c308fcc899d0"
Nov 02 04:09:58.567110 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:58.567073    4496 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-858236-5c5788679f-7r9bv" podUID="4ea5b3ef-2d8a-49a5-be96-3a9b3dc766be" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 04:09:59.576688 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:09:59.576651    4496 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-858236-646789f8d5-hzm6p" podUID="09a61fb0-97bf-4daa-8bb0-d1d07bc858fd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 04:10:03.525647 ip-10-0-133-163 kubenswrapper[4496]: E1102 04:10:03.525609    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 04:10:06.454465 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:10:06.454432    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-858236-c9zmt"]
Nov 02 04:10:06.505326 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:10:06.505293    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-858236-c9zmt"]
Nov 02 04:10:06.966191 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:10:06.966120    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-858236-646789f8d5-hzm6p"
Nov 02 04:10:07.018597 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:10:07.018490    4496 generic.go:356] "Generic (PLEG): container finished" podID="4ea5b3ef-2d8a-49a5-be96-3a9b3dc766be" containerID="ad12f9d3f2e8c81ea9b3d40c001151ff6a4cce7393b642cb0132494d121ca366" exitCode=0
Nov 02 04:10:07.018597 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:10:07.018528    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-858236-5c5788679f-7r9bv" event={"ID":"4ea5b3ef-2d8a-49a5-be96-3a9b3dc766be","Type":"ContainerDied","Data":"ad12f9d3f2e8c81ea9b3d40c001151ff6a4cce7393b642cb0132494d121ca366"}
Nov 02 04:10:07.020231 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:10:07.020205    4496 generic.go:356] "Generic (PLEG): container finished" podID="09a61fb0-97bf-4daa-8bb0-d1d07bc858fd" containerID="0bf2270fc4c4642fbff347fc13b7015364e2c92d06807d7f19943e35bb55d5e7" exitCode=0
Nov 02 04:10:07.020394 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:10:07.020304    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-858236-646789f8d5-hzm6p" event={"ID":"09a61fb0-97bf-4daa-8bb0-d1d07bc858fd","Type":"ContainerDied","Data":"0bf2270fc4c4642fbff347fc13b7015364e2c92d06807d7f19943e35bb55d5e7"}
Nov 02 04:10:07.020394 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:10:07.020344    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-858236-646789f8d5-hzm6p" event={"ID":"09a61fb0-97bf-4daa-8bb0-d1d07bc858fd","Type":"ContainerDied","Data":"66f2043bafc891cc17c45d53254c65d12b63612108f8af0dee4537077995fe6c"}
Nov 02 04:10:07.020394 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:10:07.020373    4496 scope.go:122] "RemoveContainer" containerID="0bf2270fc4c4642fbff347fc13b7015364e2c92d06807d7f19943e35bb55d5e7"
Nov 02 04:10:07.020590 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:10:07.020520    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-858236-646789f8d5-hzm6p"
Nov 02 04:10:07.030635 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:10:07.030615    4496 scope.go:122] "RemoveContainer" containerID="0bf2270fc4c4642fbff347fc13b7015364e2c92d06807d7f19943e35bb55d5e7"
Nov 02 04:10:07.030983 ip-10-0-133-163 kubenswrapper[4496]: E1102 04:10:07.030963    4496 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bf2270fc4c4642fbff347fc13b7015364e2c92d06807d7f19943e35bb55d5e7\": container with ID starting with 0bf2270fc4c4642fbff347fc13b7015364e2c92d06807d7f19943e35bb55d5e7 not found: ID does not exist" containerID="0bf2270fc4c4642fbff347fc13b7015364e2c92d06807d7f19943e35bb55d5e7"
Nov 02 04:10:07.031111 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:10:07.030990    4496 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bf2270fc4c4642fbff347fc13b7015364e2c92d06807d7f19943e35bb55d5e7"} err="failed to get container status \"0bf2270fc4c4642fbff347fc13b7015364e2c92d06807d7f19943e35bb55d5e7\": rpc error: code = NotFound desc = could not find container \"0bf2270fc4c4642fbff347fc13b7015364e2c92d06807d7f19943e35bb55d5e7\": container with ID starting with 0bf2270fc4c4642fbff347fc13b7015364e2c92d06807d7f19943e35bb55d5e7 not found: ID does not exist"
Nov 02 04:10:07.051383 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:10:07.051362    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09a61fb0-97bf-4daa-8bb0-d1d07bc858fd-kube-api-access-tcm29\" (UniqueName: \"kubernetes.io/projected/09a61fb0-97bf-4daa-8bb0-d1d07bc858fd-kube-api-access-tcm29\") pod \"09a61fb0-97bf-4daa-8bb0-d1d07bc858fd\" (UID: \"09a61fb0-97bf-4daa-8bb0-d1d07bc858fd\") "
Nov 02 04:10:07.051500 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:10:07.051400    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09a61fb0-97bf-4daa-8bb0-d1d07bc858fd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09a61fb0-97bf-4daa-8bb0-d1d07bc858fd-plugins-config-volume\") pod \"09a61fb0-97bf-4daa-8bb0-d1d07bc858fd\" (UID: \"09a61fb0-97bf-4daa-8bb0-d1d07bc858fd\") "
Nov 02 04:10:07.051839 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:10:07.051818    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09a61fb0-97bf-4daa-8bb0-d1d07bc858fd-plugins-config-volume" pod "09a61fb0-97bf-4daa-8bb0-d1d07bc858fd" (UID: "09a61fb0-97bf-4daa-8bb0-d1d07bc858fd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 04:10:07.053583 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:10:07.053562    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09a61fb0-97bf-4daa-8bb0-d1d07bc858fd-kube-api-access-tcm29" pod "09a61fb0-97bf-4daa-8bb0-d1d07bc858fd" (UID: "09a61fb0-97bf-4daa-8bb0-d1d07bc858fd"). InnerVolumeSpecName "kube-api-access-tcm29". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 04:10:07.072483 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:10:07.072463    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-858236-5c5788679f-7r9bv"
Nov 02 04:10:07.152182 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:10:07.152141    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ea5b3ef-2d8a-49a5-be96-3a9b3dc766be-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ea5b3ef-2d8a-49a5-be96-3a9b3dc766be-plugins-config-volume\") pod \"4ea5b3ef-2d8a-49a5-be96-3a9b3dc766be\" (UID: \"4ea5b3ef-2d8a-49a5-be96-3a9b3dc766be\") "
Nov 02 04:10:07.152389 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:10:07.152201    4496 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ea5b3ef-2d8a-49a5-be96-3a9b3dc766be-kube-api-access-ng9qj\" (UniqueName: \"kubernetes.io/projected/4ea5b3ef-2d8a-49a5-be96-3a9b3dc766be-kube-api-access-ng9qj\") pod \"4ea5b3ef-2d8a-49a5-be96-3a9b3dc766be\" (UID: \"4ea5b3ef-2d8a-49a5-be96-3a9b3dc766be\") "
Nov 02 04:10:07.152517 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:10:07.152497    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcm29\" (UniqueName: \"kubernetes.io/projected/09a61fb0-97bf-4daa-8bb0-d1d07bc858fd-kube-api-access-tcm29\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:10:07.152586 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:10:07.152524    4496 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09a61fb0-97bf-4daa-8bb0-d1d07bc858fd-plugins-config-volume\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:10:07.152586 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:10:07.152528    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ea5b3ef-2d8a-49a5-be96-3a9b3dc766be-plugins-config-volume" pod "4ea5b3ef-2d8a-49a5-be96-3a9b3dc766be" (UID: "4ea5b3ef-2d8a-49a5-be96-3a9b3dc766be"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 04:10:07.154401 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:10:07.154375    4496 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ea5b3ef-2d8a-49a5-be96-3a9b3dc766be-kube-api-access-ng9qj" pod "4ea5b3ef-2d8a-49a5-be96-3a9b3dc766be" (UID: "4ea5b3ef-2d8a-49a5-be96-3a9b3dc766be"). InnerVolumeSpecName "kube-api-access-ng9qj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 04:10:07.253351 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:10:07.253316    4496 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ea5b3ef-2d8a-49a5-be96-3a9b3dc766be-plugins-config-volume\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:10:07.253351 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:10:07.253353    4496 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ng9qj\" (UniqueName: \"kubernetes.io/projected/4ea5b3ef-2d8a-49a5-be96-3a9b3dc766be-kube-api-access-ng9qj\") on node \"ip-10-0-133-163.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 04:10:07.442445 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:10:07.442413    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-858236-646789f8d5-hzm6p"]
Nov 02 04:10:07.446241 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:10:07.446216    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-858236-646789f8d5-hzm6p"]
Nov 02 04:10:07.572471 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:10:07.572389    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09a61fb0-97bf-4daa-8bb0-d1d07bc858fd" path="/var/lib/kubelet/pods/09a61fb0-97bf-4daa-8bb0-d1d07bc858fd/volumes"
Nov 02 04:10:07.572898 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:10:07.572791    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9f8e30e-8655-426c-9ebe-4911c950e579" path="/var/lib/kubelet/pods/a9f8e30e-8655-426c-9ebe-4911c950e579/volumes"
Nov 02 04:10:08.027161 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:10:08.027127    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-858236-5c5788679f-7r9bv" event={"ID":"4ea5b3ef-2d8a-49a5-be96-3a9b3dc766be","Type":"ContainerDied","Data":"b9de93ba64afadeca6c42aabe8c44aac6fe1f8b4b726cf4984b49065f11f60e8"}
Nov 02 04:10:08.027335 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:10:08.027215    4496 scope.go:122] "RemoveContainer" containerID="ad12f9d3f2e8c81ea9b3d40c001151ff6a4cce7393b642cb0132494d121ca366"
Nov 02 04:10:08.027335 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:10:08.027138    4496 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-858236-5c5788679f-7r9bv"
Nov 02 04:10:08.193852 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:10:08.193809    4496 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-858236-5c5788679f-7r9bv"]
Nov 02 04:10:08.197408 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:10:08.197377    4496 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-858236-5c5788679f-7r9bv"]
Nov 02 04:10:09.571812 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:10:09.571776    4496 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ea5b3ef-2d8a-49a5-be96-3a9b3dc766be" path="/var/lib/kubelet/pods/4ea5b3ef-2d8a-49a5-be96-3a9b3dc766be/volumes"
Nov 02 04:10:14.449960 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:10:14.449931    4496 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5ccd9d7cb9-hg9ft_fbfd5bbe-5d78-41b3-90b2-9e0565607d40/authorino/0.log"
Nov 02 04:10:14.708882 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:10:14.708804    4496 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-tlh72_475f7114-488a-487c-98f7-316ef5b2a6dd/manager/0.log"
Nov 02 04:10:14.936217 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:10:14.936187    4496 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-xgxnp_a8b89697-14b6-4df5-a8d0-4c195599a4b4/manager/0.log"
Nov 02 04:10:16.380064 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:10:16.380005    4496 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-fxd5p_31b4528f-969b-41a0-b61d-4d4d882e2e8c/manager/0.log"
Nov 02 04:10:17.752763 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:10:17.752730    4496 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-7fcb66b96b-dwcmz_9dd13471-dd39-4838-be2a-dd99db00133b/manager/0.log"
Nov 02 04:10:19.677842 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:10:19.677805    4496 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-fj5pj_ced04bab-431b-4207-a07b-647649f4419e/proxy/0.log"
Nov 02 04:10:20.441599 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:10:20.441561    4496 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5ccd9d7cb9-hg9ft_fbfd5bbe-5d78-41b3-90b2-9e0565607d40/authorino/0.log"
Nov 02 04:10:30.188799 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:10:30.188767    4496 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-7fcb66b96b-dwcmz_9dd13471-dd39-4838-be2a-dd99db00133b/manager/0.log"
Nov 02 04:10:32.385806 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:10:32.385772    4496 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-fj5pj_ced04bab-431b-4207-a07b-647649f4419e/proxy/0.log"
Nov 02 04:10:33.468074 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:10:33.468047    4496 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-29b0784c-4d28j_3ec2533a-c319-4438-9b79-44bc348b3005/revoke-keys/0.log"
Nov 02 04:10:33.532146 ip-10-0-133-163 kubenswrapper[4496]: E1102 04:10:33.532112    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 04:10:35.714246 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:10:35.714211    4496 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5ccd9d7cb9-hg9ft_fbfd5bbe-5d78-41b3-90b2-9e0565607d40/authorino/0.log"
Nov 02 04:10:36.106468 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:10:36.106427    4496 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-tlh72_475f7114-488a-487c-98f7-316ef5b2a6dd/manager/0.log"
Nov 02 04:10:36.320385 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:10:36.320353    4496 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-xgxnp_a8b89697-14b6-4df5-a8d0-4c195599a4b4/manager/0.log"
Nov 02 04:10:37.467635 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:10:37.467603    4496 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-fxd5p_31b4528f-969b-41a0-b61d-4d4d882e2e8c/manager/0.log"
Nov 02 04:10:38.772215 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:10:38.772177    4496 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-59cf4dc6bfb8rqj_099ee094-c411-47c4-a050-c63bcea57bf7/istio-proxy/0.log"
Nov 02 04:10:39.907734 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:10:39.907699    4496 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-856b6dffcf-8jm8s_2f73e718-b58c-4073-8b3a-73cba88870e3/istio-proxy/0.log"
Nov 02 04:10:40.123428 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:10:40.123391    4496 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-xmc9j_99a7ac28-5904-4047-acc6-571f7aa78e35/payload-pre-processing/0.log"
Nov 02 04:10:40.325316 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:10:40.325292    4496 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-gzc2f_bd8a1b95-491e-44ea-988a-e4437c386a1a/payload-processing/0.log"
Nov 02 04:10:41.537579 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:10:41.537546    4496 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-bm2wq_d81db86e-2a08-4c67-add9-b7353fb0372d/main/0.log"
Nov 02 04:10:41.814107 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:10:41.813998    4496 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-vnxgf_bebc74da-01c1-4547-ab5d-90a0a8ec1ad7/main/0.log"
Nov 02 04:10:42.067202 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:10:42.067122    4496 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-qlqqk_3c94d962-9ac1-4c2c-a6bb-8bb264d26f3e/main/0.log"
Nov 02 04:10:42.986128 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:10:42.986099    4496 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-xlbmw_2baeda96-f36e-49cb-bb8a-3389c9e8d61d/main/0.log"
Nov 02 04:10:52.467941 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:10:52.467857    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-52wqt/must-gather-ps5v2"]
Nov 02 04:10:52.468407 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:10:52.468363    4496 cpu_manager.go:414] "containerMap: removing container" podUID="4ea5b3ef-2d8a-49a5-be96-3a9b3dc766be" containerName="payload-pre-processing"
Nov 02 04:10:52.468407 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:10:52.468376    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ea5b3ef-2d8a-49a5-be96-3a9b3dc766be" containerName="payload-pre-processing"
Nov 02 04:10:52.468407 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:10:52.468400    4496 cpu_manager.go:414] "containerMap: removing container" podUID="09a61fb0-97bf-4daa-8bb0-d1d07bc858fd" containerName="payload-processing"
Nov 02 04:10:52.468407 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:10:52.468408    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09a61fb0-97bf-4daa-8bb0-d1d07bc858fd" containerName="payload-processing"
Nov 02 04:10:52.468585 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:10:52.468430    4496 cpu_manager.go:414] "containerMap: removing container" podUID="719bb9e5-ea72-4782-bb17-034b909f874d" containerName="main"
Nov 02 04:10:52.468585 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:10:52.468435    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="719bb9e5-ea72-4782-bb17-034b909f874d" containerName="main"
Nov 02 04:10:52.468585 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:10:52.468444    4496 cpu_manager.go:414] "containerMap: removing container" podUID="3ec2533a-c319-4438-9b79-44bc348b3005" containerName="revoke-keys"
Nov 02 04:10:52.468585 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:10:52.468448    4496 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ec2533a-c319-4438-9b79-44bc348b3005" containerName="revoke-keys"
Nov 02 04:10:52.468585 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:10:52.468526    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="719bb9e5-ea72-4782-bb17-034b909f874d" containerName="main"
Nov 02 04:10:52.468585 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:10:52.468534    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ec2533a-c319-4438-9b79-44bc348b3005" containerName="revoke-keys"
Nov 02 04:10:52.468585 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:10:52.468542    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ea5b3ef-2d8a-49a5-be96-3a9b3dc766be" containerName="payload-pre-processing"
Nov 02 04:10:52.468585 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:10:52.468553    4496 memory_manager.go:365] "RemoveStaleState removing state" podUID="09a61fb0-97bf-4daa-8bb0-d1d07bc858fd" containerName="payload-processing"
Nov 02 04:10:52.473470 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:10:52.473449    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-52wqt/must-gather-ps5v2"
Nov 02 04:10:52.526305 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:10:52.526276    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-52wqt\"/\"openshift-service-ca.crt\""
Nov 02 04:10:52.564602 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:10:52.564570    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f547fe86-b797-45c5-9d07-631f1a2be8d5-must-gather-output\") pod \"must-gather-ps5v2\" (UID: \"f547fe86-b797-45c5-9d07-631f1a2be8d5\") " pod="openshift-must-gather-52wqt/must-gather-ps5v2"
Nov 02 04:10:52.564792 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:10:52.564651    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hql9x\" (UniqueName: \"kubernetes.io/projected/f547fe86-b797-45c5-9d07-631f1a2be8d5-kube-api-access-hql9x\") pod \"must-gather-ps5v2\" (UID: \"f547fe86-b797-45c5-9d07-631f1a2be8d5\") " pod="openshift-must-gather-52wqt/must-gather-ps5v2"
Nov 02 04:10:52.577011 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:10:52.576985    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-52wqt\"/\"default-dockercfg-7b25c\""
Nov 02 04:10:52.577186 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:10:52.577068    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-52wqt\"/\"kube-root-ca.crt\""
Nov 02 04:10:52.579486 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:10:52.579466    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-52wqt/must-gather-ps5v2"]
Nov 02 04:10:52.665268 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:10:52.665233    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hql9x\" (UniqueName: \"kubernetes.io/projected/f547fe86-b797-45c5-9d07-631f1a2be8d5-kube-api-access-hql9x\") pod \"must-gather-ps5v2\" (UID: \"f547fe86-b797-45c5-9d07-631f1a2be8d5\") " pod="openshift-must-gather-52wqt/must-gather-ps5v2"
Nov 02 04:10:52.665456 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:10:52.665345    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f547fe86-b797-45c5-9d07-631f1a2be8d5-must-gather-output\") pod \"must-gather-ps5v2\" (UID: \"f547fe86-b797-45c5-9d07-631f1a2be8d5\") " pod="openshift-must-gather-52wqt/must-gather-ps5v2"
Nov 02 04:10:52.665746 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:10:52.665720    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f547fe86-b797-45c5-9d07-631f1a2be8d5-must-gather-output\") pod \"must-gather-ps5v2\" (UID: \"f547fe86-b797-45c5-9d07-631f1a2be8d5\") " pod="openshift-must-gather-52wqt/must-gather-ps5v2"
Nov 02 04:10:52.723742 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:10:52.723665    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hql9x\" (UniqueName: \"kubernetes.io/projected/f547fe86-b797-45c5-9d07-631f1a2be8d5-kube-api-access-hql9x\") pod \"must-gather-ps5v2\" (UID: \"f547fe86-b797-45c5-9d07-631f1a2be8d5\") " pod="openshift-must-gather-52wqt/must-gather-ps5v2"
Nov 02 04:10:52.788559 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:10:52.788522    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-52wqt/must-gather-ps5v2"
Nov 02 04:10:53.170560 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:10:53.170526    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-52wqt/must-gather-ps5v2"]
Nov 02 04:10:53.171857 ip-10-0-133-163 kubenswrapper[4496]: W1102 04:10:53.171829    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf547fe86_b797_45c5_9d07_631f1a2be8d5.slice/crio-79d89e479e7ef9b7edc6967e9b8d715ba2e6769d7e10ff956bd16b89744c8982 WatchSource:0}: Error finding container 79d89e479e7ef9b7edc6967e9b8d715ba2e6769d7e10ff956bd16b89744c8982: Status 404 returned error can't find the container with id 79d89e479e7ef9b7edc6967e9b8d715ba2e6769d7e10ff956bd16b89744c8982
Nov 02 04:10:53.173930 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:10:53.173908    4496 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 04:10:53.293376 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:10:53.293337    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-52wqt/must-gather-ps5v2" event={"ID":"f547fe86-b797-45c5-9d07-631f1a2be8d5","Type":"ContainerStarted","Data":"79d89e479e7ef9b7edc6967e9b8d715ba2e6769d7e10ff956bd16b89744c8982"}
Nov 02 04:11:01.347985 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:11:01.347945    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-52wqt/must-gather-ps5v2" event={"ID":"f547fe86-b797-45c5-9d07-631f1a2be8d5","Type":"ContainerStarted","Data":"03981f5ed5b67dc15e4a30673f7c46c1055220848d5c500c73d6da424050b769"}
Nov 02 04:11:01.348399 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:11:01.347996    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-52wqt/must-gather-ps5v2" event={"ID":"f547fe86-b797-45c5-9d07-631f1a2be8d5","Type":"ContainerStarted","Data":"25d1c2bfd33dfba6868a9ab7a7635ce851a0423d7f1d8ebdd1bfeb3a5b75d0b4"}
Nov 02 04:11:01.486189 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:11:01.486133    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-52wqt/must-gather-ps5v2" podStartSLOduration=2.183021332 podStartE2EDuration="9.486120367s" totalImagesPullingTime="7.303099035s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 04:10:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 04:11:01.462209755 +0000 UTC m=+5002.040817136" watchObservedRunningTime="2026-11-02 04:11:01.486120367 +0000 UTC m=+5002.064727748"
Nov 02 04:11:01.517122 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:11:01.517086    4496 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-dxlld_e400bad6-59c3-4c43-aed0-40c731c967db/global-pull-secret-syncer/0.log"
Nov 02 04:11:03.527920 ip-10-0-133-163 kubenswrapper[4496]: E1102 04:11:03.527888    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 04:11:04.510374 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:11:04.510345    4496 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-7fcb66b96b-dwcmz_9dd13471-dd39-4838-be2a-dd99db00133b/manager/0.log"
Nov 02 04:11:05.462944 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:11:05.462905    4496 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-z8sq6_c078a1ee-3179-45c9-9d3b-4d4a46f0c42f/konnectivity-agent/0.log"
Nov 02 04:11:06.290524 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:11:06.290492    4496 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-163.us-west-1.compute.internal_562e4cfd5885e6864e00b6a01f2722f3/haproxy/0.log"
Nov 02 04:11:06.959616 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:11:06.959580    4496 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-fj5pj_ced04bab-431b-4207-a07b-647649f4419e/proxy/0.log"
Nov 02 04:11:10.003535 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:11:10.003501    4496 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-7fcb66b96b-dwcmz_9dd13471-dd39-4838-be2a-dd99db00133b/manager/0.log"
Nov 02 04:11:11.084433 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:11:11.084399    4496 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5ccd9d7cb9-hg9ft_fbfd5bbe-5d78-41b3-90b2-9e0565607d40/authorino/0.log"
Nov 02 04:11:11.402578 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:11:11.402492    4496 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-tlh72_475f7114-488a-487c-98f7-316ef5b2a6dd/manager/0.log"
Nov 02 04:11:11.629567 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:11:11.629521    4496 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-rbmzv/perf-node-gather-daemonset-hjg76"]
Nov 02 04:11:11.636458 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:11:11.636431    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rbmzv/perf-node-gather-daemonset-hjg76"
Nov 02 04:11:11.689486 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:11:11.689402    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rbmzv\"/\"kube-root-ca.crt\""
Nov 02 04:11:11.689486 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:11:11.689404    4496 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rbmzv\"/\"openshift-service-ca.crt\""
Nov 02 04:11:11.740080 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:11:11.740054    4496 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-rbmzv\"/\"default-dockercfg-c7lwr\""
Nov 02 04:11:11.743477 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:11:11.743450    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rbmzv/perf-node-gather-daemonset-hjg76"]
Nov 02 04:11:11.755536 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:11:11.755509    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tq9k4\" (UniqueName: \"kubernetes.io/projected/bd235183-d122-4249-9161-e556565c4500-kube-api-access-tq9k4\") pod \"perf-node-gather-daemonset-hjg76\" (UID: \"bd235183-d122-4249-9161-e556565c4500\") " pod="openshift-must-gather-rbmzv/perf-node-gather-daemonset-hjg76"
Nov 02 04:11:11.755662 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:11:11.755555    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bd235183-d122-4249-9161-e556565c4500-lib-modules\") pod \"perf-node-gather-daemonset-hjg76\" (UID: \"bd235183-d122-4249-9161-e556565c4500\") " pod="openshift-must-gather-rbmzv/perf-node-gather-daemonset-hjg76"
Nov 02 04:11:11.755662 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:11:11.755572    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bd235183-d122-4249-9161-e556565c4500-proc\") pod \"perf-node-gather-daemonset-hjg76\" (UID: \"bd235183-d122-4249-9161-e556565c4500\") " pod="openshift-must-gather-rbmzv/perf-node-gather-daemonset-hjg76"
Nov 02 04:11:11.755751 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:11:11.755702    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bd235183-d122-4249-9161-e556565c4500-podres\") pod \"perf-node-gather-daemonset-hjg76\" (UID: \"bd235183-d122-4249-9161-e556565c4500\") " pod="openshift-must-gather-rbmzv/perf-node-gather-daemonset-hjg76"
Nov 02 04:11:11.755751 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:11:11.755733    4496 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bd235183-d122-4249-9161-e556565c4500-sys\") pod \"perf-node-gather-daemonset-hjg76\" (UID: \"bd235183-d122-4249-9161-e556565c4500\") " pod="openshift-must-gather-rbmzv/perf-node-gather-daemonset-hjg76"
Nov 02 04:11:11.817754 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:11:11.817719    4496 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-xgxnp_a8b89697-14b6-4df5-a8d0-4c195599a4b4/manager/0.log"
Nov 02 04:11:11.856590 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:11:11.856558    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tq9k4\" (UniqueName: \"kubernetes.io/projected/bd235183-d122-4249-9161-e556565c4500-kube-api-access-tq9k4\") pod \"perf-node-gather-daemonset-hjg76\" (UID: \"bd235183-d122-4249-9161-e556565c4500\") " pod="openshift-must-gather-rbmzv/perf-node-gather-daemonset-hjg76"
Nov 02 04:11:11.856739 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:11:11.856604    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bd235183-d122-4249-9161-e556565c4500-lib-modules\") pod \"perf-node-gather-daemonset-hjg76\" (UID: \"bd235183-d122-4249-9161-e556565c4500\") " pod="openshift-must-gather-rbmzv/perf-node-gather-daemonset-hjg76"
Nov 02 04:11:11.856739 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:11:11.856626    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bd235183-d122-4249-9161-e556565c4500-proc\") pod \"perf-node-gather-daemonset-hjg76\" (UID: \"bd235183-d122-4249-9161-e556565c4500\") " pod="openshift-must-gather-rbmzv/perf-node-gather-daemonset-hjg76"
Nov 02 04:11:11.856739 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:11:11.856674    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bd235183-d122-4249-9161-e556565c4500-podres\") pod \"perf-node-gather-daemonset-hjg76\" (UID: \"bd235183-d122-4249-9161-e556565c4500\") " pod="openshift-must-gather-rbmzv/perf-node-gather-daemonset-hjg76"
Nov 02 04:11:11.856739 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:11:11.856708    4496 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bd235183-d122-4249-9161-e556565c4500-sys\") pod \"perf-node-gather-daemonset-hjg76\" (UID: \"bd235183-d122-4249-9161-e556565c4500\") " pod="openshift-must-gather-rbmzv/perf-node-gather-daemonset-hjg76"
Nov 02 04:11:11.856739 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:11:11.856725    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bd235183-d122-4249-9161-e556565c4500-lib-modules\") pod \"perf-node-gather-daemonset-hjg76\" (UID: \"bd235183-d122-4249-9161-e556565c4500\") " pod="openshift-must-gather-rbmzv/perf-node-gather-daemonset-hjg76"
Nov 02 04:11:11.856968 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:11:11.856783    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bd235183-d122-4249-9161-e556565c4500-sys\") pod \"perf-node-gather-daemonset-hjg76\" (UID: \"bd235183-d122-4249-9161-e556565c4500\") " pod="openshift-must-gather-rbmzv/perf-node-gather-daemonset-hjg76"
Nov 02 04:11:11.856968 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:11:11.856785    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bd235183-d122-4249-9161-e556565c4500-proc\") pod \"perf-node-gather-daemonset-hjg76\" (UID: \"bd235183-d122-4249-9161-e556565c4500\") " pod="openshift-must-gather-rbmzv/perf-node-gather-daemonset-hjg76"
Nov 02 04:11:11.856968 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:11:11.856798    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bd235183-d122-4249-9161-e556565c4500-podres\") pod \"perf-node-gather-daemonset-hjg76\" (UID: \"bd235183-d122-4249-9161-e556565c4500\") " pod="openshift-must-gather-rbmzv/perf-node-gather-daemonset-hjg76"
Nov 02 04:11:11.914711 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:11:11.914685    4496 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tq9k4\" (UniqueName: \"kubernetes.io/projected/bd235183-d122-4249-9161-e556565c4500-kube-api-access-tq9k4\") pod \"perf-node-gather-daemonset-hjg76\" (UID: \"bd235183-d122-4249-9161-e556565c4500\") " pod="openshift-must-gather-rbmzv/perf-node-gather-daemonset-hjg76"
Nov 02 04:11:11.950497 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:11:11.950424    4496 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rbmzv/perf-node-gather-daemonset-hjg76"
Nov 02 04:11:12.331290 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:11:12.331265    4496 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rbmzv/perf-node-gather-daemonset-hjg76"]
Nov 02 04:11:12.332364 ip-10-0-133-163 kubenswrapper[4496]: W1102 04:11:12.332337    4496 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podbd235183_d122_4249_9161_e556565c4500.slice/crio-8df5c2cd84a7f8d2a05a60882025b833893200d1fc5565ed5dc02b6dfe06fffd WatchSource:0}: Error finding container 8df5c2cd84a7f8d2a05a60882025b833893200d1fc5565ed5dc02b6dfe06fffd: Status 404 returned error can't find the container with id 8df5c2cd84a7f8d2a05a60882025b833893200d1fc5565ed5dc02b6dfe06fffd
Nov 02 04:11:12.418909 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:11:12.418869    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rbmzv/perf-node-gather-daemonset-hjg76" event={"ID":"bd235183-d122-4249-9161-e556565c4500","Type":"ContainerStarted","Data":"8df5c2cd84a7f8d2a05a60882025b833893200d1fc5565ed5dc02b6dfe06fffd"}
Nov 02 04:11:12.542877 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:11:12.542842    4496 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-fj5pj_ced04bab-431b-4207-a07b-647649f4419e/proxy/0.log"
Nov 02 04:11:13.314579 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:11:13.314541    4496 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-fxd5p_31b4528f-969b-41a0-b61d-4d4d882e2e8c/manager/0.log"
Nov 02 04:11:13.427350 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:11:13.427301    4496 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rbmzv/perf-node-gather-daemonset-hjg76" event={"ID":"bd235183-d122-4249-9161-e556565c4500","Type":"ContainerStarted","Data":"66edc45f7bfb7a3607ca1b8888af3cb72558711df28b22fdb6109be39b6f31b3"}
Nov 02 04:11:13.427788 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:11:13.427536    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-rbmzv/perf-node-gather-daemonset-hjg76"
Nov 02 04:11:13.542919 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:11:13.542864    4496 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-rbmzv/perf-node-gather-daemonset-hjg76" podStartSLOduration=2.5428533140000003 podStartE2EDuration="2.542853314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 04:11:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 04:11:13.541826066 +0000 UTC m=+5014.120433441" watchObservedRunningTime="2026-11-02 04:11:13.542853314 +0000 UTC m=+5014.121460694"
Nov 02 04:11:15.457262 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:11:15.457221    4496 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-7fcb66b96b-dwcmz_9dd13471-dd39-4838-be2a-dd99db00133b/manager/0.log"
Nov 02 04:11:17.853277 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:11:17.853247    4496 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-fj5pj_ced04bab-431b-4207-a07b-647649f4419e/proxy/0.log"
Nov 02 04:11:19.446786 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:11:19.446755    4496 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-rbmzv/perf-node-gather-daemonset-hjg76"
Nov 02 04:11:19.977595 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:11:19.977561    4496 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-sdtnc_9cf4b8c4-e1ad-4880-a501-53d6a5970ac8/kube-state-metrics/0.log"
Nov 02 04:11:20.383243 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:11:20.383146    4496 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-sdtnc_9cf4b8c4-e1ad-4880-a501-53d6a5970ac8/kube-rbac-proxy-main/0.log"
Nov 02 04:11:20.797473 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:11:20.797442    4496 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-sdtnc_9cf4b8c4-e1ad-4880-a501-53d6a5970ac8/kube-rbac-proxy-self/0.log"
Nov 02 04:11:21.156458 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:11:21.156381    4496 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-59cf4dc6bfb8rqj_099ee094-c411-47c4-a050-c63bcea57bf7/istio-proxy/0.log"
Nov 02 04:11:21.215266 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:11:21.215232    4496 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-76dcdd8ff6-59rns_5d932f33-ed28-4494-930b-c87718b95d42/metrics-server/0.log"
Nov 02 04:11:21.647358 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:11:21.647311    4496 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-d7ljw_67a0e3bb-443b-4f8e-aa30-ddedc41efb83/monitoring-plugin/0.log"
Nov 02 04:11:22.550808 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:11:22.550780    4496 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-856b6dffcf-8jm8s_2f73e718-b58c-4073-8b3a-73cba88870e3/istio-proxy/0.log"
Nov 02 04:11:22.759766 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:11:22.759730    4496 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-xmc9j_99a7ac28-5904-4047-acc6-571f7aa78e35/payload-pre-processing/0.log"
Nov 02 04:11:22.968797 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:11:22.968722    4496 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-gzc2f_bd8a1b95-491e-44ea-988a-e4437c386a1a/payload-processing/0.log"
Nov 02 04:11:23.321918 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:11:23.321887    4496 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-d2wl9_55c81f55-e3b2-4006-ad91-c2a843a426ad/node-exporter/0.log"
Nov 02 04:11:23.740506 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:11:23.740477    4496 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-d2wl9_55c81f55-e3b2-4006-ad91-c2a843a426ad/kube-rbac-proxy/0.log"
Nov 02 04:11:24.155734 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:11:24.155662    4496 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-d2wl9_55c81f55-e3b2-4006-ad91-c2a843a426ad/init-textfile/0.log"
Nov 02 04:11:25.855631 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:11:25.854459    4496 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-q2bzg_2d589c8b-5c19-4cee-a17e-c8bc46c6fa35/kube-rbac-proxy-main/0.log"
Nov 02 04:11:26.268396 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:11:26.268364    4496 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-q2bzg_2d589c8b-5c19-4cee-a17e-c8bc46c6fa35/kube-rbac-proxy-self/0.log"
Nov 02 04:11:26.681093 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:11:26.680984    4496 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-q2bzg_2d589c8b-5c19-4cee-a17e-c8bc46c6fa35/openshift-state-metrics/0.log"
Nov 02 04:11:28.665564 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:11:28.665511    4496 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5ccd9d7cb9-hg9ft_fbfd5bbe-5d78-41b3-90b2-9e0565607d40/authorino/0.log"
Nov 02 04:11:28.777115 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:11:28.777083    4496 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-tlh72_475f7114-488a-487c-98f7-316ef5b2a6dd/manager/0.log"
Nov 02 04:11:28.982946 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:11:28.982912    4496 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-xgxnp_a8b89697-14b6-4df5-a8d0-4c195599a4b4/manager/0.log"
Nov 02 04:11:30.013519 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:11:30.013480    4496 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-bk94k_51e4a664-eed1-49a4-bfb8-49112a42b99c/prometheus-operator/0.log"
Nov 02 04:11:30.061468 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:11:30.061436    4496 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-fxd5p_31b4528f-969b-41a0-b61d-4d4d882e2e8c/manager/0.log"
Nov 02 04:11:30.418409 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:11:30.418329    4496 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-bk94k_51e4a664-eed1-49a4-bfb8-49112a42b99c/kube-rbac-proxy/0.log"
Nov 02 04:11:30.832652 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:11:30.832623    4496 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-cv2tt_56e6d06c-317d-43db-90a1-5c758a6d5d24/prometheus-operator-admission-webhook/0.log"
Nov 02 04:11:32.493119 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:11:32.493086    4496 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-fb86b55c4-sq549_6ff7dc8e-86b1-4e4b-b16d-9fbac2dea1cd/thanos-query/0.log"
Nov 02 04:11:32.904672 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:11:32.904589    4496 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-fb86b55c4-sq549_6ff7dc8e-86b1-4e4b-b16d-9fbac2dea1cd/kube-rbac-proxy-web/0.log"
Nov 02 04:11:33.316589 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:11:33.316562    4496 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-fb86b55c4-sq549_6ff7dc8e-86b1-4e4b-b16d-9fbac2dea1cd/kube-rbac-proxy/0.log"
Nov 02 04:11:33.530151 ip-10-0-133-163 kubenswrapper[4496]: E1102 04:11:33.530115    4496 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4zs9" containerName="extractor"
Nov 02 04:11:33.731899 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:11:33.731868    4496 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-fb86b55c4-sq549_6ff7dc8e-86b1-4e4b-b16d-9fbac2dea1cd/prom-label-proxy/0.log"
Nov 02 04:11:34.145119 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:11:34.145042    4496 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-fb86b55c4-sq549_6ff7dc8e-86b1-4e4b-b16d-9fbac2dea1cd/kube-rbac-proxy-rules/0.log"
Nov 02 04:11:34.560471 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:11:34.560427    4496 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-fb86b55c4-sq549_6ff7dc8e-86b1-4e4b-b16d-9fbac2dea1cd/kube-rbac-proxy-metrics/0.log"
Nov 02 04:11:34.740969 ip-10-0-133-163 kubenswrapper[4496]: I1102 04:11:34.740928    4496 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-59cf4dc6bfb8rqj_099ee094-c411-47c4-a050-c63bcea57bf7/istio-proxy/0.log"